Agent Desktop

Agent Desktop is described in ^[src-104] as an open-source desktop-control layer that can read an operating system accessibility tree and expose actions an AI agent can use to operate local applications.

Key facts

• Type: Desktop-control tool / open-source repo
• First seen in wiki: Pat Simmons's GPT Realtime 2 desktop-control walkthrough ^[src-104]
• Role in source: Acts as the "hands" for applications that do not expose an MCP server or API ^[src-104].
• Evidence note: The transcript names the repo as Agent Desktop, but the creator attribution is caption-uncertain and should be verified before being treated as a durable entity fact.

What it does

The source positions Agent Desktop as the fallback layer after structured integrations are exhausted. Browser search can be handled through dedicated browser tools; Obsidian can be handled through MCP/API; but a complex app such as Premiere Pro may need a different path because it exposes controls through the desktop UI rather than an agent-friendly API ^[src-104].

In that setup, the accessibility tree tells the agent what controls exist, while Agent Desktop-style actions let the agent click buttons, move through menus, play, pause, cut, mark in, mark out, or ripple-delete inside the application ^[src-104].

Related

• See also: Voice-Driven Desktop Agents, Agent Security Boundaries, Voice-to-Action Interfaces, GPT Realtime 2, Claude Code Computer Use

Source references

• ^[src-104] Pat Simmons – "GPT Realtime 2 Can Now Run Your Entire Computer (Just Your Voice)" (2026-06-17)