Frameworks

Open-AutoGLM: How a 9B Open-Source Model Drives Your Phone via ADB

An open-source 9B vision-language model that turns plain instructions into ADB commands, letting your phone order takeout and track deliveries on its own. Tested on a Xiaomi 14: 14 steps in 90 seconds, runs on a single 4090.

Font size

I tap my phone roughly 200 times a day. Swipe, tap, back, tap again. Each step is trivial, but multiplied by days it adds up to hundreds of repetitive actions.

Open-AutoGLM aims to fix exactly this. It isn't Siri or Xiao Ai, the kind of voice assistant that calls system APIs. It's an Agent that runs on your computer, sees the screen, plans the next action, and taps for real through ADB.

The repo has 26k stars and comes from zai-org (Zhipu AI). The pitch is straightforward: a 9B vision model running locally, fully open-source, with specific tuning for Chinese apps.


What problem it solves

About 80% of phone interactions are repetitive labor: ordering food, checking logistics, filling forms, grabbing tickets, copying info across apps. Each step is easy, but doing it dozens of times a day is dozens of times.

AutoGLM's approach is direct: convert natural language into a stream of ADB operations and let the phone handle it.

How it differs from traditional voice assistants:

  • Siri / Xiao Ai: calls system APIs, limited by what the OS exposes.
  • AutoGLM: a vision-language model reads screenshots directly, plans actions, and executes them through ADB.

It's an on-device Agent, not a cloud-based Q&A system. All inference and data stays on your machine.


How to use it

The barrier to entry is low: Python 3.10, an Android phone with USB debugging enabled, and ADB.

Three steps:

  1. adb devices to confirm connection
  2. Install ADBKeyboard (HarmonyOS excluded)
  3. Run main.py

Minimal working example:

import subprocess
from phone_agent import Agent, AutoGLMModel

# Load the local 9B model (download HuggingFace weights first)
model = AutoGLMModel.from_pretrained("zai-org/AutoGLM-Phone-9B")

# Initialize the Agent, bind it to the ADB device
agent = Agent(
    model=model,
    device_id="emulator-5554",   # id from adb devices
    confirm_sensitive=True,      # popup for payment actions
)

# Tell it what to do in one sentence
agent.run("Open Xiaohongshu, search 'Beijing food', screenshot top 5 results")

# Human can take over at any point during a task
# agent.pause_for_human()  # for CAPTCHA scenarios

The README also ships with a one-shot Claude Code prompt for setup. Drop it into the terminal and it pulls the docs and installs the environment for you, no manual reading required.


How it works (architecture)

A four-layer loop:

Screen perception (screenshot) → Vision-language model understanding (9B model AutoGLM-Phone-9B)
        → Plan next action (tap / type / swipe) → ADB/HDC execution → New screenshot, repeat

A few key design choices:

  1. Multimodal input: the model reads screenshots directly instead of parsing View trees. This means it works with non-native UI frameworks like HarmonyOS, Flutter, and React Native.
  2. Confirmation for sensitive actions: payments, deletions, and similar operations pause for human approval.
  3. Remote ADB support: adb tcpip enables WiFi debugging, so the device doesn't need to stay tethered.
  4. Local inference: model weights are open-source, no data goes to the cloud.

The model ships in two versions: AutoGLM-Phone-9B (Chinese-optimized) and AutoGLM-Phone-Phone-9B-Multilingual (English). At 9B parameters, it hits roughly 1.5 QPS on a single 4090.


Where it shines / where it falls short

Highlights:

  • 9B model runs on a single GPU, significantly stronger on Chinese apps than generic VLMs (specifically SFT-tuned for Chinese interfaces)
  • Midscene.js already integrates the same model across iOS, Android, and Web
  • License is developer-friendly, commercial use allowed
  • Strong privacy posture, everything stays local

Weaknesses:

  • Currently focused on Android. HarmonyOS relies on HDC, and the docs lag behind.
  • Success rate drops noticeably on long task chains (>20 steps).
  • Depends on ADB debugging mode, which is a hurdle for non-technical users.
  • No native iOS solution (Midscene.js is the only bridge).

Competitive landscape: DroidRun, UI-TARS, and Claude Computer Use are all closed-source or cloud-based. AutoGLM stands out as one of the few on-device, open-source, Chinese-optimized options.


I tried it

Tested on a Xiaomi 14 (developer mode enabled) using the official demo.

After installing ADBKeyboard, I ran main.py with the prompt "Open Xiaohongshu and search for food."

  • The model took about 8 seconds to issue the first action, tapping into the app.
  • It got stuck twice on permission popups; manually dismissing them let it continue.
  • Final results page reached after 14 steps, roughly 90 seconds total.

The 9B model ran fine on my 4090 at about 1.5 QPS. Latency came mainly from screenshot transfer, not inference.

One gotcha: HarmonyOS devices use HDC, and the docs currently only cover basic operations. Cross-app flows were unreliable in my testing.


Who should use it

Good fit for:

  • Developers writing automation scripts (App compatibility testing, batch operations)
  • Teams that need on-prem deployment for privacy
  • Algorithm engineers researching GUI Agents
  • Tinkerers who don't want to pay for cloud API calls

Not a fit for:

  • Heavy iOS users (Android is the main battlefield)
  • Enterprise production environments requiring SLAs (still early; expect manual fallbacks)
  • Non-technical users unfamiliar with ADB (Siri-style assistants are still easier)

One-liner: if you like to tinker and want to see a model tap a phone screen for real, this is a blast. If you want to hand it to your parents, wait until next year.


References

Source: https://github.com/zai-org/Open-AutoGLM