DrivingBench · 2026-09-22 · major
DrivingBench — GPT-6 Astra is the only model to finish a real cone course
DrivingBench gives frontier models control of a real Toyota Corolla on a cone course, one command at a time. GPT-6 Astra finished it on its second try in 5:22. Claude Fable 5.1 got 45% of the way; Grok 4.6 and GPT-5.6 Sol barely started.
Four frontier models took turns steering a real car around cones; only GPT-6 Astra made it to the end.
Key specs
| Best finish time | 5:22 |
|---|---|
| Models that finished | 1 of 4 |
Quick facts
| Built by | Aditya Ramabadran, Simon Mahns, Tobias Gessler |
|---|---|
| Car | Toyota Corolla with a comma device running openpilot |
| Course | About 130 m of traffic cones in an empty lot |
| Speed cap | 3.5 m/s |
| Attempts | Up to 3, in one continuous chat session |
| Harness license | MIT |
Benchmarks
| GPT-6 Astra | 100% | |
|---|---|---|
| Claude Fable 5.1 | 45% | |
| Grok 4.6 | 11% | |
| GPT-5.6 Sol | 6% |
What is it?
DrivingBench is an independent test of whether general-purpose AI models can drive a real car, not a simulator. Three researchers put GPT-6 Astra, Claude Fable 5.1, Grok 4.6 and GPT-5.6 Sol in charge of a Toyota Corolla's steering, accelerator and brakes on a fixed course of traffic cones. Each model got up to three attempts, and the site scores how far along the course it got.
How does it work?
The open harness exposes three MCP tools: observe() returns a road camera image plus speed and steering angle, set_motion() asks for a steering direction, a speed between 0.5 and 3.5 m/s and a duration, and stop_now() brakes. Commands go through an HTTP gateway to a comma device running a modified openpilot, whose own 100 Hz control loop turns the request into wheel angles and enforces safety limits. A licensed driver sat in the driver's seat, foot over the brake, for every run.
Why does it matter?
Most model rankings measure text, code or screen clicks. This one checks whether a model can turn camera frames into safe physical actions with real latency, and the gap is large: one model finished, one got under halfway, two failed near the start. Co-author Aditya Ramabadran told The Register that using a frontier model out of the box for real driving 'is definitely not practical', so read it as a test of perception and planning, not a path to self-driving cars.
Who is it for?
embodied-AI and agent researchers
Frequently asked questions
- How long did GPT-6 Astra take to finish the DrivingBench course?
- GPT-6 Astra finished the DrivingBench cone course on its second attempt in 5 minutes 22 seconds. The Register reports it covered 134.7 meters at an average of 0.94 mph. The slow pace comes from the setup: the model drives one command at a time from inside a chat session, and speed is capped at 3.5 meters per second.
- How did Claude Fable 5.1 and Grok 4.6 do on DrivingBench?
- Claude Fable 5.1 was the closest rival on DrivingBench, reaching 45% of the course on its best attempt without finishing. Grok 4.6 reached 11% and GPT-5.6 Sol only 6%. The Register says the failed attempts mostly broke down at the first corner because of perception problems, meaning the models misread where the cones were.
- Can I run the DrivingBench harness myself?
- The DrivingBench harness is open source under the MIT license on GitHub. It needs a comma four device running openpilot v0.11.2, a supported Toyota with longitudinal control (it was tested on a Corolla), a macOS laptop with Python 3.12 and uv, and an empty private lot with cones. The README says a licensed, attentive safety driver must stay in the driver's seat at all times.
- How much did a DrivingBench run cost?
- The Register reports the DrivingBench runs used about 6.6 million tokens for $7.74 in API fees at discounted cache rates, plus $999 for the comma hardware. Counting both, the researchers put the cost at roughly $92.47 per mile driven, against about $0.18 per mile for fuel, which shows how far this is from a practical way to drive.
Try it
https://github.com/aditya-ramabadran/drivingbench_harness_v1