23 Daily — Vercel-labs launched agent-desktop in November 2026, an open-source cross-platform command-line interface (CLI) written in Rust. It enables users to control AI agents for desktop applications via CLI commands, aiming to enhance desktop automation and accessibility. According to the project's description, agent-desktop attracted over 9,236 stars on GitHub Trending, showcasing strong developer interest.
CLI Based Semantic Desktop Application Control
Agent-desktop leverages the OS accessibility tree to read and operate desktop apps, returning JSON snapshots containing references (refs) to every UI element an agent can manipulate, instead of relying on pixel matching. It supports macOS, Windows, and Linux.
On macOS, it reuses a private command host to persist native accessibility objects, preventing silent resolution errors when UI elements change. The host shuts down after 15 minutes of inactivity. Unlike traditional pixel-based methods, agent-desktop allows CLI agents to perform clicks, typing, and window management without shifting focus or moving the mouse pointer.
Comprehensive Command Set Enables Diverse Interactions
Agent-desktop offers over 60 command names, including snapshotting UI elements, click actions, typing text, focusing elements, window management (launch, close-app, resize-window), notification and clipboard operations. Commands return JSON envelopes with data or detailed errors, facilitating automation and integration. Users can list operable apps via "agent-desktop list-apps" and update UI snapshots to maintain accurate references for precise control.
Cross-platform Installation and Permissions
Agent-desktop installation supports npm, Bun, Yarn, and pnpm package managers for x64 and ARM64 architectures. macOS users need to grant Accessibility permission; Windows requires no additional privileges, while Linux needs the graphical session with "at-spi2-core" accessibility services enabled. Examples include controlling the Finder application on macOS.
Editor's take: A Feasible Path for CLI-Driven AI Agent Desktop Automation
By structuring control around accessibility trees and CLI, agent-desktop overcomes the constraints of pixel-based automation, establishing a semantic and cross-platform approach for AI desktop agents. This foundational infrastructure benefits startups focusing on desktop automation, accessibility, and AI applications.
Verification can start by integrating existing AI agents to automate daily desktop tasks while monitoring adaptability to UI changes and error reporting. Developers can leverage its JSON outputs and error codes to build robust retry mechanisms and feedback loops, enhancing agent decision-making processes.