Skip to content
Request access

Real-SWE 1.1

Benchmarking frontier AI models on private, real-world, enterprise codebases.

Read the blog

Leaderboard

October 2026
  1. Rank 1
    GPT 6.1 Sol
    Resolution rate: 47.20%, 95% confidence interval 42.5% to 51.9%
  2. Rank 2
    Claude Fable 5.1
    Resolution rate: 45.60%, 95% confidence interval 39.5% to 51.7%
  3. Rank 3
    GPT 6 Astra
    Resolution rate: 44.80%, 95% confidence interval 41.3% to 48.3%
  4. Tied for rank 4
    Claude Opus 5.5
    Resolution rate: 42.40%, 95% confidence interval 36.5% to 48.3%
  5. Tied for rank 4
    Muse Wonder 1.0
    Resolution rate: 42.40%, 95% confidence interval 37.4% to 47.4%
  6. Rank 6
    Gemini 4 Argon
    Resolution rate: 39.20%, 95% confidence interval 32.6% to 45.8%
  7. Rank 7
    DeepSeek V4.1 Flash
    Resolution rate: 38.40%, 95% confidence interval 33.3% to 43.5%
  8. Rank 8
    Hy4 preview
    Resolution rate: 33.60%, 95% confidence interval 28.3% to 38.9%
  9. Rank 9
    GLM-5.3
    Resolution rate: 26.40%, 95% confidence interval 21.4% to 31.4%
  10. Rank 10
    Grok 4.7
    Resolution rate: 25.60%, 95% confidence interval 20.6% to 30.6%
  11. Rank 11
    Qwen3.8-Max prime
    Resolution rate: 24.00%, 95% confidence interval 18.5% to 29.5%
  12. Rank 12
    Kimi K3
    Resolution rate: 11.20%, 95% confidence interval 6.8% to 15.6%
  13. Rank 13
    GPT 6 Luna
    Resolution rate: 10.40%, 95% confidence interval 6.7% to 14.1%
  14. Rank 14
    Inkling
    Resolution rate: 8.00%, 95% confidence interval 5.1% to 10.9%
  15. Rank 15
    Seed 2.1 Turbo
    Resolution rate: 5.60%, 95% confidence interval 2.7% to 8.5%
  16. Rank 16
    Nemotron 3 Ultra
    Resolution rate: 4.80%, 95% confidence interval 2.1% to 7.5%
RankModelResolution rate
1GPT 6.1 Sol
47.20%, 95% confidence interval 42.5% to 51.9%
2Claude Fable 5.1
45.60%, 95% confidence interval 39.5% to 51.7%
3GPT 6 Astra
44.80%, 95% confidence interval 41.3% to 48.3%
Tied for 4Claude Opus 5.5
42.40%, 95% confidence interval 36.5% to 48.3%
Tied for 4Muse Wonder 1.0
42.40%, 95% confidence interval 37.4% to 47.4%
6Gemini 4 Argon
39.20%, 95% confidence interval 32.6% to 45.8%
7DeepSeek V4.1 Flash
38.40%, 95% confidence interval 33.3% to 43.5%
8Hy4 preview
33.60%, 95% confidence interval 28.3% to 38.9%
9GLM-5.3
26.40%, 95% confidence interval 21.4% to 31.4%
10Grok 4.7
25.60%, 95% confidence interval 20.6% to 30.6%
11Qwen3.8-Max prime
24.00%, 95% confidence interval 18.5% to 29.5%
12Kimi K3
11.20%, 95% confidence interval 6.8% to 15.6%
13GPT 6 Luna
10.40%, 95% confidence interval 6.7% to 14.1%
14Inkling
8.00%, 95% confidence interval 5.1% to 10.9%
15Seed 2.1 Turbo
5.60%, 95% confidence interval 2.7% to 8.5%
16Nemotron 3 Ultra
4.80%, 95% confidence interval 2.1% to 7.5%
Resolution rate is equivalent to pass@1, averaged over five independent runs per task. 95% confidence intervals are shown.All runs use the highest effort possible for each model with its harness.

We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.

Resolution rate by estimated cost per rollout

Models in leaderboard order, with estimated cost per rollout
ModelResolution rateEstimated cost per rollout
1GPT 6.1 Sol47.20%$0.79
2Fable 5.145.60%$14.24
3GPT 6 Astra44.80%$3.78
4Opus 5.542.40%$13.95
5Muse Wonder 1.042.40%$3.76
6Gemini 4 Argon39.20%$4.66
7DeepSeek V4.1 Flash38.40%$0.19
8Hy4 preview33.60%$1.13
9GLM-5.326.40%$3.35
10Grok 4.725.60%$2.66
11Qwen3.8-Max prime24.00%$7.06
12Kimi K311.20%$2.31
13GPT 6 Luna10.40%$0.08
14Inkling8.00%$5.13
15Seed 2.1 Turbo5.60%$3.03
16Nemotron 3 Ultra4.80%$1.39
The dotted line connects the best observed trade-offs, not intermediate evaluations.

About the benchmark

Private codebases

Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet.

Work with business consequences

Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services.

Company-specific complexity

Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there.

Task examples

View analysis

  • TypeScript

    Tax jurisdiction

    Restore tax determination on invoices from each business's tax regime: own rate, authority quote or none, with exemptions and filing.

    Solved by 3 of 16 models

  • Ruby

    FX quote compatibility

    Restore FX quotes and finish the settlement lifecycle of a Rails API.

    Solved by 7 of 16 models

  • Go

    Live-chat block history

    Persistent live-chat history in sequential batches of 100 messages.

    Solved by 12 of 16 models

  • Python

    NSE and BSE instrument sync

    Sync financial instruments with a trading-system feed and the NSE and BSE exchange feeds.

    Solved by 1 of 16 models

One dot per model, in leaderboard order. A filled dot means the model passed the task at least once.

Evaluation setup

Each agent was run in an isolated sandbox. All tasks are in Harbor format, and verifiers are injected at grading time. The verifiers are inspired by existing test suites in the codebase or use those tests verbatim.

Read the full analysis

Request sample access

Request access