Products

Reverse-Engineering ChatGPT Work: What OpenAI's Agent Stack Actually Looks Like

Latent Space reconstructed OpenAI's Agent architecture from the outside, piecing together Memory, Browser Use, Plugins, and four other modules. The result is a map of which pieces are real and which are still vapor.

Font size

Latent Space published Unpacking ChatGPT Work: An external reconstruction of the Agent for a Billion Users, reverse-engineering the parts of ChatGPT Work that OpenAI hasn't bothered to explain.

This isn't an official document. It's community archaeology.

And that's worth talking about. OpenAI has positioned ChatGPT Work as the "Agent for a Billion Users," yet publishes virtually nothing about how it works. To figure out what the product actually is, you have to read UI diffs, watch for new fields in API responses, and infer behavior from user patterns. Latent Space stitched those signals into seven modules: Memory, Proactivity, Scheduling, Browser Use, Plugins, Skills, and Tools.

I have a few takes after reading it.

Why an "external reconstruction" instead of an official one

OpenAI's disclosure strategy is schizophrenic. On the model side (GPT-5 and friends), they'll drop papers and system cards. On the product side, especially ChatGPT Work, they say almost nothing.

What you can find in the help center is all "how to use" content: how to turn on Memory, how to enable Browser Use, how to install a plugin. The architecture-level questions — where Memory lives, what triggers Proactivity, how Scheduling integrates with calendars — are all black boxes.

Latent Space's approach: watch UI changes closely, compare product copy across versions, track new fields in API responses, and assemble those signals into a system diagram.

This kind of reverse-engineering is inherently speculative, and the authors say so. But it exposes something real — where OpenAI actually is on the Agent roadmap versus what they're pitching. Which modules work today. Which ones are placeholder slides.

Of the seven modules, which ones actually work

Split the seven modules by "can you use it today."

Working and useful:

  • Memory: Long-term memory has shipped. My own ChatGPT account has had Memory on for six-plus months. It remembers the language I write in, my role, preferences from past conversations.
  • Browser Use: The Operator direction — letting ChatGPT open a browser and click things itself.
  • Plugins and Skills/Tools: The tool-calling layer, three overlapping systems (more on this below).

Looks like roadmap placeholders:

  • Proactivity: The model proactively takes actions based on memory. Cool concept, no evidence it's actually running.
  • Scheduling: Calendar integration, time-triggered tasks. Demoed, but no shipped behavior to point to.

This distinction matters. OpenAI tends to fold roadmap items into product marketing, so consumers think things work when they're aspirational. Memory and Browser Use are real. Proactivity and Scheduling are "we intend to build this."

Memory isn't new. Memory plus Proactivity is.

Memory on its own isn't novel. ChatGPT has had single-user long-term memory for a while.

What gives "Agent for a Billion Users" any teeth is the combination of Memory and Proactivity:

The model uses long-term memory to proactively trigger actions at the right moment.

Once that loop closes, ChatGPT goes from "you ask, it answers" to "it finds you."

Example. You tell ChatGPT you're going to Tokyo next week. Based on that stored fact, on Monday morning it pings you: "Weather in Tokyo looks rough — want me to check flights?" That's not a chatbot anymore. That's an assistant living in your life, reaching out on its own.

The pieces are all technically feasible. Memory is storing. Browser Use can click pages. The only missing piece is the Proactivity scheduling layer.

And that's the most dangerous piece. Proactive triggers mean the model decides when to contact you and about what. OpenAI's GPT-5 system card will probably address safety constraints here — though we can't see any of it yet.

Plugins vs. Skills vs. Tools: three overlapping systems

This is the part that annoyed me most.

In under two years, OpenAI has shipped three separate tool-calling mechanisms:

  1. Plugins (GPT-4 era, 2023) — manifest files describing APIs for ChatGPT to call third-party services.
  2. GPTs + Actions (late 2023) — Actions embedded in custom GPTs, essentially Plugins evolved.
  3. Skills / Tools (2024–2025) — tool abstractions in the Agent SDK with a new function-calling schema.

Three systems running in parallel, and the names collide with everyone else's. "Skills" is a thing on Anthropic's Claude. "Tools" is core vocabulary in LangChain and LlamaIndex.

What does this mean for developers?

The integration you write today is probably obsolete next year. A Plugin I built for a client last year needed a rewrite as Actions this year, and the next migration is probably to Tools or Skills. Migration cost is real money.

It's a classic symptom of an unstable Agent infrastructure layer. A real platform has one clean tool-calling spec (MCP is one attempt, but it's Anthropic-driven — whether OpenAI adopts it is an open question). It doesn't have three overlapping systems covering each other.

OpenAI hasn't figured out how to converge the tool-calling layer. Three parallel systems reflects internal roadmap politics, not deliberate product design.

What reverse-engineering can't see

External reconstruction has natural limits. What Latent Space can see:

  • How the modules fit together (the seven-piece puzzle)
  • What works versus what's a placeholder
  • The direction of UI evolution

What Latent Space can't see:

  • Failure rates: Browser Use looks great in demos. How often does it click the wrong thing, stall, or crash in the wild?
  • Cost structure: How many tokens does an Agent burn per task? How does OpenAI plan to pay for a billion Agent-using users?
  • Safety constraints: On the Proactivity line, what guardrails has OpenAI set internally?

The last question is the most important. "Agent for a Billion Users" implicitly means a billion people waking up every day inside reach of an AI assistant that initiates contact. The regulatory pressure, compliance risk, and social implications — OpenAI has barely addressed any of it.

The article tells you what the pie looks like. It doesn't tell you whether you'll choke on it.

My read

After reading Latent Space's piece, here's where I land:

  1. ChatGPT Work is real, not a slide deck. Memory and Browser Use are running and functional.
  2. The "Agent for a Billion Users" tagline doesn't hold up yet. No visible Proactivity. No shipped Scheduling.
  3. The tool-calling layer's confusion is a real problem. Three parallel systems (Plugins, Skills, Tools), and OpenAI hasn't converged.
  4. The real risk lives in Proactivity, not Memory. Once that loop closes, AI assistants shift from reactive to proactive, and the social implications jump an order of magnitude.

OpenAI has shared what they can show. What they haven't shared — costs, failure rates, safety constraints — is what determines whether ChatGPT Work actually scales to a billion users.

Reverse-engineering gets you seven pieces of the puzzle. The rest has to come from OpenAI.

Source: https://www.latent.space/p/unpacking-chatgpt-work