DeepLethe · 2026-10-15 · notable
FDE-Gym — can AI agents deliver a working system to a real customer?
FDE-Gym is a benchmark of 95 customer delivery tasks from 18 sectors, where agents must find unstated needs and ship a working system. The best of 13 model setups, Claude Opus 5.5, scores 0.56 and fully solves 14 tasks.
A new benchmark tests whether agents can deliver an AI system that actually improves a customer's numbers.
Key specs
| Tasks | 95 |
|---|---|
| Best mean score | 0.56 |
What is it?
FDE-Gym tests AI agents on the 'last mile' work of a forward deployed engineer: 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, adapted by industry experts from real projects. It ships as a paper, a Hugging Face dataset, an Apache-2.0 online judge you can self-host, and a public leaderboard.
How does it work?
Each task puts the agent in a simulated customer environment with scripted stakeholders and systems it can change. The score is how much the delivered system improves the customer's own success metric, set between 0 for an unchanged system and 1 for a reference solution, with a penalty for each stakeholder contact.
Why does it matter?
The paper finds that discovering unstated constraints, not coding, is the main bottleneck: leading models score 0.73 to 0.82 when the rules are given, but the best reaches only 0.56 in the realistic setting. Claude Sonnet 5.5 scores 0.49, GPT-6 Astra 0.38 and GPT-6.1 Sol 0.37.
Who is it for?
teams evaluating coding agents for client work, agent researchers
Try it
git clone https://github.com/deeplethe/fde-gym.git && cd fde-gym && python3 scripts/fetch_cases.py && docker compose up --build