How is Real-SWE different from SWE-bench and other public coding benchmarks?

Public coding benchmarks are built from open-source repositories that models have very likely seen in training. Real-SWE instead uses production codebases licensed from private companies, so the code and its fixes are not on the public internet. Specific Labs also reports a median of 11 files edited per reference solution, against 6 for FrontierCode and DeepSWE.

Can I download the Real-SWE tasks?

Real-SWE tasks are not a public download, because the codebases behind them are licensed from private companies. Specific Labs offers a sample on request through a form on the benchmark page. The tasks are packaged in Harbor format with injected verifiers, so each run is scored by whether the change actually works.

Which coding agent scored best on Real-SWE, and what did it cost?

Fable 5.1 running in Claude Code leads Real-SWE at a 38.8% resolution rate, and it is also the most expensive setup at an estimated $6.96 per rollout. Gemini 3.8 Flash in Gemini CLI costs $2.50 per rollout for 31.2%. Specific Labs notes that higher spend does not reliably buy a higher score.

Why do agents fail on Real-SWE tasks?

Specific Labs sorts failed Real-SWE runs into five kinds: building on an unverified assumption, missing a stated requirement, wiring a correct idea into the system incorrectly, breaking existing behavior, and editing a file the running app never calls. Missing a requirement dominates for Grok 4.6 at 67.2% of its failed runs.

Does giving an agent more time help on Real-SWE?

Not much, on the Real-SWE numbers. Rollouts that finished in under 10 minutes failed 71.4% of the time (70 of 98), and rollouts of 10 minutes or longer failed 73.4% of the time (398 of 542). Specific Labs reads this as triage difficulty rather than a time budget problem.

Specific Labs · 2026-09-12 · major

Real-SWE — coding agents tested on private company codebases

Real-SWE is a coding benchmark built on private production codebases that Specific Labs licensed from real companies. The best score is 38.8%, from Fable 5.1 running in Claude Code, well under what agents post on public benchmarks.

A coding-agent benchmark whose tasks come from real companies' private codebases, not from open-source repos.

Key specs

Top resolution rate	38.8%

Quick facts

Maker	Specific Labs (Y Combinator F25)
Task source	Private production codebases, licensed from real companies
Models tested	8, each in its own agent harness
Scoring	pass@1, averaged over 8 runs per task
Task difficulty	6 of the 10 sample tasks score under 15%
Cost per rollout	$2.50 to $6.96
Access	Sample tasks by request

Benchmarks

Real-SWE resolution rate

Fable 5.1 (Claude Code)		38.8%
GPT-6 Astra (Codex CLI)		33.8%
Gemini 3.8 Flash (Gemini CLI)		31.2%
GLM 5.3 (Claude Code)		28.8%
Grok 4.6 (Grok Build)		23.8%
Muse Spark 1.3 (Muse Code)		23.8%
Kimi K3 (Kimi Code)		18.8%
GPT-5.6 Sol (Codex CLI)		16.2%

source ↗

What is it?

Real-SWE scores coding agents on work that engineers were actually paid to do inside private companies. Specific Labs licensed the codebases — among them a Luma and Partiful competitor with over 200,000 users and a top-100 App Store ranking, a consumer fintech platform that processes more than 100,000 bank statements, and enterprise sales tools. Every task is lifted or adapted from a ticket an engineer on that team handled.

How does it work?

Each task hands the agent the repository plus the context a staff engineer would have had, then an injected verifier checks whether the finished change behaves correctly. Tasks ship in Harbor format and expose only the services the workflow needs, from Postgres and Redis to an AWS emulator, Kubernetes, GitHub, Linear and Slack. Models run inside their own native harness — Claude Code, Codex CLI, Gemini CLI, Grok Build — so the score covers the model and the tooling together, averaged over eight runs per task.

Why does it matter?

The gap is the point: the leading agent on Real-SWE resolves 38.8% of tasks, and six of the ten analysed tasks sit under 15%, with one at zero. Teams deciding how much company work to hand an agent now have numbers taken from code the model could not have memorised. The failure breakdown is just as useful, since it says where the agents break — usually a missed requirement or a guess that was never checked against the workspace.

Who is it for?

engineering leads evaluating coding agents

Frequently asked questions

How is Real-SWE different from SWE-bench and other public coding benchmarks?: Public coding benchmarks are built from open-source repositories that models have very likely seen in training. Real-SWE instead uses production codebases licensed from private companies, so the code and its fixes are not on the public internet. Specific Labs also reports a median of 11 files edited per reference solution, against 6 for FrontierCode and DeepSWE.
Can I download the Real-SWE tasks?: Real-SWE tasks are not a public download, because the codebases behind them are licensed from private companies. Specific Labs offers a sample on request through a form on the benchmark page. The tasks are packaged in Harbor format with injected verifiers, so each run is scored by whether the change actually works.
Which coding agent scored best on Real-SWE, and what did it cost?: Fable 5.1 running in Claude Code leads Real-SWE at a 38.8% resolution rate, and it is also the most expensive setup at an estimated $6.96 per rollout. Gemini 3.8 Flash in Gemini CLI costs $2.50 per rollout for 31.2%. Specific Labs notes that higher spend does not reliably buy a higher score.
Why do agents fail on Real-SWE tasks?: Specific Labs sorts failed Real-SWE runs into five kinds: building on an unverified assumption, missing a stated requirement, wiring a correct idea into the system incorrectly, breaking existing behavior, and editing a file the running app never calls. Missing a requirement dominates for Grok 4.6 at 67.2% of its failed runs.
Does giving an agent more time help on Real-SWE?: Not much, on the Real-SWE numbers. Rollouts that finished in under 10 minutes failed 71.4% of the time (70 of 98), and rollouts of 10 minutes or longer failed 73.4% of the time (398 of 542). Specific Labs reads this as triage difficulty rather than a time budget problem.

Try it

Request the sample task set at https://withspecific.com/benchmarks/real-swe/request-access