A link hit the Hacker News front page yesterday with 187 points and 161 comments—DrivingBench, billing itself as "the first benchmark for evaluating LLM driving ability." The team also dropped a bombshell conclusion: their GPT-6 Astra model can already drive a car.
Hold on.
OpenAI's current product line tops out at GPT-5. The name "GPT-6" has never appeared in any official channel, let alone the codename "Astra." I dug into this, and there are several points worth unpacking.
What Is DrivingBench?
drivingbench.com positions itself as a benchmark that puts LLMs into end-to-end driving scenarios.
This is not the same as traditional evaluations like Waymo's Disengagement Report. Waymo measures takeover counts, mileage, and simulation scores from its own system; Tesla FSD's safety reports track crashes per mile. That data comes from real fleets or simulators.
DrivingBench takes a different angle: it puts an LLM directly in the driver's seat and evaluates how it performs in driving scenarios. The test subject is the model itself, not an autonomous driving system.
This angle isn't new—academia has had end-to-end AD benchmarks like Bench2Drive, nuScenes, and CARLA for years. But DrivingBench is packaged cleverly: it targets the LLM crowd, not AD vendors—getting language model people involved in driving evaluation.
From the project description, DrivingBench focuses on the multimodal / embodied intelligence intersection. The core question it tries to answer: can large models understand driving tasks in an embodied way?
What Is GPT-6 Astra?
This is the part I find most questionable.
"GPT-6"—OpenAI's public naming goes GPT-4 → GPT-4o → GPT-5, with the o1 / o3 series in between. "GPT-6" has never appeared in any official OpenAI channel.
"Astra"—Google has Astra (Project Astra), the codename for its Gemini multimodal assistant. OpenAI's codenames follow the GPT series + o-series + internal names (like Strawberry). "Astra" doesn't appear anywhere in OpenAI's public materials.
So "GPT-6 Astra" is almost certainly a fabricated alias. It could be to avoid NDAs on real models (common in third-party benchmark circles; Vellum's leaderboard also uses aliases), or it could just be an attention-grabbing hook.
Either way, using an unreleased next-gen model name plus an unverifiable codename for benchmark marketing is itself a red flag. The first thing readers see isn't DrivingBench's methodology—it's "GPT-6 can drive now."
Real Driving Is Not Playing GTA
This is the question DrivingBench most needs to answer but currently doesn't address anywhere visible: how is the LLM actually "driving"?
There are a few common approaches for hooking LLMs into driving:
- Running in a simulator—environments like CARLA and nuScenes feed visual streams / CAN signals to the model, which outputs steering angle, throttle, and brake. This is "true end-to-end," but there's still a large gap from real-world physics and sensor noise.
- Watching video and planning—feeding the model a front-camera video and asking for high-level plans ("there's a car ahead, change lanes"). This is VQA (visual question answering) extended; it's not driving, it's "commentary on driving."
- Connecting to a real CAN bus—directly controlling steering, throttle, and brake. This requires latency and reliability far beyond LLM inference speeds, and no production system dares do it.
DrivingBench doesn't explain on its homepage which approach it uses. If it's route 2, the "driving ability" label is watered down—no different from "GPT-5 can recognize traffic lights from images."
The more important distinction: embodied intelligence vs. visual question answering. True embodiment means the model is responsible for the consequences of its actions—if it's wrong, it crashes. In VQA mode, if the model is wrong, it just outputs wrong text, with zero cost. If DrivingBench's scoring is the latter, then no matter how intimidating the name, it's just a VLM benchmark wearing a driving costume.
What the HN Comments Are Arguing About
The HN thread with 161 comments converged on three points:
1. How "end-to-end" is end-to-end?
Some argue that LLMs driving is "AI cosplay"—you feed it visual input, it outputs text descriptions, and there's no control loop at all. Unless the output is shown to connect directly to vehicle actuators, it doesn't count as end-to-end.
2. Are the test scenarios adversarial enough?
Traditional AD benchmarks deliberately design corner cases (ghost probes, construction detours, sensor failures). DrivingBench's public samples don't show this kind of coverage. If it's just "sunny day highway driving," high scores prove nothing.
3. Can benchmark scores represent real ability?
One comment put it bluntly: "The biggest problem in the benchmark world right now is using leaderboard numbers to fake capability. No one has reproduced DrivingBench's conclusions in real scenarios—including that GPT-6 Astra." Transparency concerns came up repeatedly.
Others noted that DrivingBench's domain was newly registered and the team background is untraceable. This isn't the same tier as lm-evaluation-harness, the industry standard maintained by EleutherAI.
Editorial Take
This story is worth covering but can't be fully trusted—both judgments hold at the same time.
Worth covering because it represents an increasingly common phenomenon in AI: using unreleased next-gen models for benchmark marketing. From last year's "GPT-5 leaked early" to today's "GPT-6 Astra," naming bravado has become part of the attention economy. Readers see the IP value of "GPT-6," not the methodological value of DrivingBench.
Can't be fully trusted because DrivingBench's methodology transparency is insufficient—the definition of "end-to-end," test scenario coverage, and the specific form of model output are all unaddressed on the site. Until these hard questions are answered, the conclusion that "GPT-6 Astra has driving ability" is no different from an "AI driving" demo on a Snapchat/Clay filter.
My prediction: over the next 6 months, this combination of "unreleased model + hot task" will become more common. The reason is straightforward—next-gen models tested inside OpenAI / Anthropic / Google need PR, but the companies stay quiet; third-party teams seize this window and grab the first-mover story with aliases. By the time the real model launches, the benchmark has become an established fact, and "we tested it first" turns into a form of credibility capital.
What's different this time: it's an HN front-page event (187 points isn't small), and it touches the "embodied intelligence" narrative that's hot in 2025–2026. Waymo's disengagement numbers, Figure's household robot videos, FSD's v13 rollout—the entire embodied track is doing PR. DrivingBench just joined the game in a more questionable way.
Acknowledged uncertainty: I don't have DrivingBench's full paper / scoring table / dataset. If they later release details proving it's genuinely route 1 (true closed-loop in a simulator), I may have to retract some of my skepticism. Until then, I'm treating this as a marketing event, not a scientific one.
References: