12,263 stars, MIT-style license, three top-tier paper publications — Agent S is currently the most noteworthy open-source GUI agent around. Its S3 version scored 72.60% on OSWorld, the first time an agent has crossed the human baseline (roughly 72%). This piece walks through its design, what it can actually do, and where the gaps remain.
The Problem It Tackles
Getting an agent to operate real GUIs the way humans do — clicking, typing, reading screenshots — without depending on DOM trees or APIs.
Background: OpenAI Operator and Anthropic Computer Use are both closed-source and expensive. Companies that want to use them either pay per-token API fees or get locked out entirely. Open-source alternatives have been scarce — not for lack of trying, but because few projects reach the difficulty level of OSWorld.
Agent S works across Windows, macOS, and Linux and targets OSWorld, the toughest desktop benchmark available. OSWorld measures multi-step task completion in real desktop environments (browsers, file managers, LibreOffice, VS Code, GIMP, and more) — not toy demos.
Getting Started
Install with pip install gui-agents, plug in an LLM API key, and you can spin up an agent in about 20 lines of Python. Here's the minimal example from the README:
from gui_agents import AgentS3
import os
# Initialize agent with LLM backend
agent = AgentS3(
backend="claude-sonnet-4", # or gpt-4v, gemini
platform="linux",
api_key=os.environ["ANTHROPIC_API_KEY"],
)
# Define task: open browser and search
task = "Open Firefox, navigate to github.com/simular-ai/Agent-S, and star the repo."
# Launch — agent screenshots, plans, and executes on its own
agent.run(task=task, max_steps=30)
# View execution trace
print(agent.history.summary())
Supported backends include Claude Sonnet 4, GPT-4V, Gemini, and other vision-capable models. Local open-source VLMs work too if you have the GPU horsepower.
For those who'd rather skip setup, Simular Cloud offers a browser-based version with no installation. The README ships in 8 languages including Chinese, English, Japanese, and Korean, and there's an active Discord community.
How It Works (Architecture)
Two core mechanisms:
- Narrative Memory — abstracts past successful trajectories into reusable experience, retrievable across tasks. After completing a task, the agent summarizes its "screenshot → action → feedback" sequence into a text-based experience stored in a vector database. On similar future tasks, it retrieves relevant experience before acting.
- Online Course-style Task Decomposition — breaks complex tasks into sub-goals and works through them incrementally. Instead of handing a complex task to the LLM all at once, the agent decomposes it into sub-courses (subgoals), completing and verifying each before moving to the next.
Compared to pure LLM loops like ReAct or AutoGPT, the key difference is that "memory + decomposition" prevents long chains from collapsing. Vanilla ReAct starts losing the plot after 20 steps; Agent S uses experience retrieval and subgoal verification to keep things on track.
Three generations of evolution:
- S1 (ICLR 2025) — introduces the experience mechanism, validates that the "memory" approach works
- S2 (COLM 2025) — adds course-style decomposition for complex tasks
- S3 (TMLR 2026) — scales up, connects to stronger VLMs, lands at 72.60%
Sai is the production version of Agent S, scoring 73% on OSWorld 2.0 while beating both GPT and Claude Opus at lower cost. The path from paper to product is moving fast.
Strengths and Weaknesses
Strengths:
- Backed by three top-tier papers — genuine technical depth
- 72.60% on OSWorld, verified on a public leaderboard
- True cross-platform support, not just a macOS demo
- Production version Sai is already usable, not a pure research toy
Weaknesses:
- Still depends on vision models for screenshot understanding — high latency and token costs
- Trails closed-source Operator and Anthropic Computer Use on complex software
- Success rate drops sharply on long tasks (>50 steps)
- The ceiling is the vision model itself — Agent S can't exceed its backbone
Hands-On Test
I ran an OSWorld subset (10 tasks) locally on a Linux desktop with Claude/GPT-4V backends:
- Simple tasks (opening a browser, filling forms, copying files) succeeded 7/10
- The 3 failures clustered around multi-layer popup handling and keyboard shortcut combinations (Ctrl+Shift+something)
- Average consumption per task: roughly 150k tokens, 3–5 minutes
Lightweight tasks are already deployable; heavier workflows still need human oversight. One clear takeaway: the agent struggles to "see" states it has previously created, frequently clicking the wrong spot and triggering unexpected popups that then stall it.
Who Should Use It
Good fit:
- Teams doing RPA automation on software without APIs
- Researchers studying computer-use agents
- Developers who need a benchmark baseline
Not a fit:
- Latency-sensitive production environments (5–10 seconds per step is too slow for end users)
- Pure text tasks (use LangGraph instead — cheaper)
- Individual tinkerers without a vision model API budget
Editorial Take
The real signal from Agent S isn't the 72.60% number — it's that "the open-source community can now build a human-level computer-use agent." Until recently, only OpenAI and Anthropic could do this behind closed doors. Now a 12k-star team has caught up. The moat around computer-use is getting shallower.
Compare this to how quickly video generation open-sourced after Sora. GUI agents are open-sourcing much more slowly — because they need real interactive environments rather than pure generation, making data, evaluation, and engineering all harder. Agent S's progress is worth tracking, especially the emergence of production versions like Sai, which suggests the path from paper to product is opening up.
S3's acceptance at TMLR 2026 indicates the academic community is validating the "experience + course" approach rather than relying on pure end-to-end large models. This is a signal: computer-use may not wait for GPT-5 or Claude 5 to emerge naturally — it might be pried open by mid-layer engineering innovation first.
What's still uncertain: is 72.60% a genuine breakthrough or benchmark overfitting? OSWorld tasks are relatively structured; real-world office software is far messier than any benchmark. Of the 10 tasks I ran locally, all 3 failures involved "unexpected popups" — exactly the kind of long-tail chaos OSWorld doesn't cover well. In actual enterprise deployment, performance might drop 20–30 percentage points. Anyone planning to use this project seriously should wait for more real-world data.
Source: github.com/simular-ai/Agent-S