Specific Labs, an applied AI research lab, launched Real-SWE, a benchmark that tests AI coding models on tasks drawn from private production codebases licensed from real companies, according to the company's published results.
The benchmark ran eight model-and-harness combinations across 10 tasks, each attempted eight independent times, covering business-critical work such as billing logic, tax calculations and customer data migration. Instructions averaged about 1,742 characters, short enough that models had to discover implementation details inside the codebase on their own, Specific Labs said.
Fable 5.1 resolved the most tasks, at 38.8%, followed by GPT-6 Astra at 33.8% and Gemini 3.8 Flash at 31.2%. Gemini's results cost $2.50 per rollout against Fable's $6.96, and results varied sharply by task, from 67.2% resolution on one task to zero on another, according to the benchmark.
Missed requirements were the most common failure mode, accounting for 36.7% of the top model's failures, ahead of unverified assumptions and integration errors, Specific Labs said. Rollouts under 10 minutes failed about as often as longer ones, suggesting the gap is about codebase complexity rather than time spent.
Public coding benchmarks have been getting saturated for a while, and this is a sign of where the real gap sits: not writing plausible code, but working correctly inside a codebase the model never trained on. Anyone evaluating tools for internal engineering work should weight results like these well above leaderboard scores built on public repositories.