Specific Labs, a Y Combinator Fall 2025 company, announced Real-SWE on Sept. 10, 2026, in a post on X. The coding-agent benchmark uses private company codebases.
The company said it evaluates “coding agents on real software engineering tasks that engineers performed, using private, out-of-distribution codebases.” Source tasks include an app with 200K+ users, a fintech platform processing 100K+ bank statements and enterprise sales tools.
Specific Labs said each task “comes from a private production codebase that we licensed from a real-world company.” One sample task asks an agent to “fix invoice billing so each business charges the right tax and exempt customers aren't taxed.”
The company posted a Real-SWE leaderboard with eight model-and-harness pairs. Fable 5.1 with Claude Code scored 38.8%, followed by GPT-6 Astra with Codex CLI at 33.8%, Gemini 3.8 Flash with Gemini CLI at 31.2% and GLM 5.3 with Claude Code at 28.8%.
Grok 4.6 with Grok Build and Muse Spark 1.3 with Muse Code each scored 23.8%. Kimi K3 with Kimi Code scored 18.8%, and GPT-5.6 Sol with Codex CLI scored 16.2%. Resolution rate is pass@1 averaged over eight independent runs per task.
Six of the 10 public sample tasks had overall resolution rates below 15%, and the analytics stream reducer task scored 0.0% for every model. Specific Labs said no model solves every task. Reference solutions for sample tasks edited a median of 11 files, compared with six in FrontierCode and DeepSWE.
Agents ran in isolated sandboxes. Tasks use Harbor format, with verifiers injected at grading time, and the company scores model-and-harness combinations rather than models alone.
Co-founders Janak and Sid said they spent the past year acquiring and licensing operational data and codebases from real companies. Labs seeking an evaluation or the full task set for training were asked to email janak@withspecific.com; companies can contribute anonymized codebases while tasks remain private.
