Case Studies

GitHub's Open-Source AI Agent Surfaces 24 Android CVEs

GitHub released its in-house security-auditing agent as open source, and running it against Android turned up 24 real vulnerabilities — not a demo, but verified CVEs.

Font size

Earlier this month, GitHub published a post titled "How we found 24 Android vulnerabilities using our open source AI security agent."

The short version: GitHub built an AI agent that surfaced 24 real vulnerabilities in Android. The code is fully open source, and anyone can reproduce the results.

The post picked up traction — it hit the front page of Hacker News, and the comment thread was mostly variations of "finally, something that isn't a demo."

Here's a breakdown.

How the 24 Vulnerabilities Were Found

These are not toy examples. This is real Android system-level auditing.

GitHub's security team had a clear target: rather than running a public benchmark, they pointed the agent directly at AOSP (Android Open Source Project) code. Over a few weeks, the agent surfaced 24 vulnerabilities spanning memory safety, permission bypasses, and information leaks.

The workflow has four steps:

  1. Code indexing — the agent breaks the target codebase into searchable units and builds an index.
  2. Pattern reasoning — based on vulnerability models, it reasons about suspicious patterns (out-of-bounds access, use-after-free, missing permission checks).
  3. PoC generation — it generates test inputs or fuzz harnesses for suspicious sites.
  4. Human review — a security researcher verifies the findings and submits fixes.

Step 4 is the key: the agent does not issue CVEs directly. Its job is to narrow the search space and produce evidence.

The Core Idea: Make the LLM Audit Like a Human

Traditional SAST (Static Application Security Testing) tools rely on rules — pattern matching that flags strcpy and calls it a day. They miss a lot, because real vulnerabilities often span functions and modules, with logic buried in data flow.

GitHub's agent flips this: the LLM plays the role of a junior security researcher.

Concretely, the agent is designed to call a chain of tools:

# Simplified agent main loop
while not done:
    thought = llm(f"Current suspicion: {state}, already tried: {history}")
    action = llm(f"Based on the above, what tool next?\nTool list: " +
                 "[code_search, read_file, run_fuzzer, write_poc]")
    observation = execute(action)
    state.update(observation)

The core tools:

  • code_search — a grep-like tool, but with semantic queries
  • read_file — reads source snippets with context window management
  • run_fuzzer — calls libFuzzer / AFL to run test cases
  • crash_triage — dedup and priority sorting for crashes

GitHub didn't go deep on prompt design, but one line stood out: have the agent emulate the reasoning chain of an experienced auditor rather than asking "is there a bug here?" The difference matters — the former builds a threat model first and then checks systematically; the latter drifts into hallucination and reports bugs that aren't there.

Where to Get It and What It Costs to Run

The repository lives under GitHub's Augments project (the "open source AI security agent" referenced in the blog title). The exact link is in GitHub's blog post — not repeating it here to avoid pointing at a stale URL in a few months.

To run it locally, you'll need:

  • GPU — for inference, 16GB VRAM minimum (fits 7B–13B models); or just call the OpenAI / Anthropic API
  • API key — at least one of Claude / GPT
  • Code target — a medium-sized C/C++ project (Android system services run a few hundred thousand lines of C++)
  • Time — reproducing a single finding runs from a few hours to about a day, depending on fuzz time

One caveat: the agent's value sits at the balance point between inference cost and human cost. Scanning a single npm package? Traditional SAST is faster. The agent fits scenarios with large codebases, complex vulnerability patterns, and infrequent human review.

How It Compares to Burp Suite and CodeQL

DimensionCodeQLBurp SuiteAI Agent
Use caseStatic analysis, rule-drivenWeb penetration testingSystem-level auditing, logic bugs
SpeedFast (minutes)InteractiveSlow (hours to days)
False negativesHigh (rules miss things)MediumLow (reasoning covers more ground)
False positivesMediumLowMedium–high (LLM hallucination)
Expertise neededWriting QL queriesPentest experiencePrompting + security background

The agent isn't a replacement — it's a complement. Rule engines can't crack logic flaws; the agent has a shot. The agent can't instantly match known patterns; rule engines do that in seconds.

The interesting framing from GitHub: they didn't pitch it as "AI replacing security researchers." They positioned it as an amplifier — letting one researcher do the work of five.

What Makes This Different

AI-assisted vulnerability discovery isn't new. Google has OSS-Fuzz generating seeds with LLMs, Meta (formerly Facebook) ran CVE-Bench evaluations, and academia has produced a long list of papers.

What this release has that others didn't: open source + real CVEs + a major-vendor stamp. First time all three showed up together.

Two implications I'd put weight on:

Short term (1–2 years): mid-size companies will deploy similar tools for internal auditing. The cost math shifts — hiring a security researcher used to run about $500K a year; an agent plus one person can do more for around $300K. Junior roles at top security consultancies will shrink.

Long term (3–5 years): the open-source ecosystem will produce 10–20 similar "vertical agents" — cryptography auditing, smart contract auditing, binary reverse engineering. As LLM reasoning improves, the marginal cost of finding vulnerabilities will trend toward zero.

One contrarian take: this is good news for open source. Smaller projects that couldn't afford security audits can now run an agent over their codebase and clear out the low-hanging bugs. Big projects like AOSP benefit even more — most of those 24 vulnerabilities likely affect every downstream vendor.

Uncertainty: GitHub hasn't published the full set of vulnerability details or reproduction steps, so I can't independently verify the severity distribution. If more than half are low-severity info leaks, the "disruption" narrative loses force. Worth waiting until the CVE records are fully live on NVD before drawing conclusions.

Sources:

Source: https://github.blog/security/how-we-found-24-android-vulnerabilities-using-our-open-source-ai-security-agent/