A task with its own rules.
Public single-attempt software engineering tasks drawn from repositories outside the original SWE-bench set.
- Organisation
- Scale AI
- Version
- v1
Loading benchmark header
Loading benchmark registry
Coding / Benchmark profile
Public single-attempt software engineering tasks drawn from repositories outside the original SWE-bench set.
Observed results
20 ranked models · higher is better · labels show rank and score
| Model | Rank | Model ID | Provider | Score | Unit |
|---|---|---|---|---|---|
| Claude Mythos 5 | #1 | claude-mythos-5 | Anthropic | 80.3 | percent |
| Claude Fable 5 | #2 | claude-fable-5 | Anthropic | 80 | percent |
| Claude Opus 5 | #3 | claude-opus-5 | Anthropic | 79.2 | percent |
| Sakana Fugu-Ultra | #4 | sakana-fugu-ultra | Sakana AI | 73.7 | percent |
| Claude Opus 4.8 | #5 | claude-opus-4-8 | Anthropic | 69.2 | percent |
| Qwen3.8 Max | #6 | qwen-3-8-max | Alibaba Cloud | 67.7 | percent |
| Grok 4.5 | #7 | grok-4-5 | xAI | 64.7 | percent |
| GPT-5.6 Sol | #8 | gpt-5-6-sol | OpenAI | 64.6 | percent |
| Claude Opus 4.7 | #9 | claude-opus-4-7 | Anthropic | 64.3 | percent |
| GPT-5.6 Terra | #10 | gpt-5-6-terra | OpenAI | 63.4 | percent |
| Claude Sonnet 5 | #11 | claude-sonnet-5 | Anthropic | 63.2 | percent |
| GPT-5.6 Luna | #12 | gpt-5-6-luna | OpenAI | 62.7 | percent |
| Qwen3.8-Flash-Next | #13 | qwen-3-8-flash-next | Alibaba Cloud | 62.5 | percent |
| Ornith-1.0-397B | #14 | ornith-1-0-397b | DeepReinforce AI | 62.2 | percent |
| GLM-5.2 | #15 | glm-5-2 | Z.AI | 62.1 | percent |
| Qwen3.8-27B | #16 | qwen-3-8-27b | Alibaba Cloud | 61.7 | percent |
| Muse Spark 1.1 | #17 | muse-spark-1-1 | Meta | 61.5 | percent |
| Qwen3.7-Max | #18 | qwen-3-7-max | Alibaba Cloud | 60.6 | percent |
| LongCat-2.0 | #19 | longcat-2-0 | Meituan | 59.5 | percent |
| Laguna S 2.1 | #20 | laguna-s-2-1 | Poolside | 59.4 | percent |
| MiniMax M3 | #21 | minimax-m3 | MiniMax | 59 | percent |
| Sakana Fugu | #22 | sakana-fugu | Sakana AI | 59 | percent |
| Gemini 3.6 Flash | #23 | gemini-3-6-flash | 58.7 | percent | |
| GPT-5.5 | #24 | gpt-5-5 | OpenAI | 58.6 | percent |
| Kimi K2.6 | #25 | kimi-k2-6 | Moonshot AI | 58.6 | percent |
| GLM-5.1 | #26 | glm-5-1 | Z.AI | 58.4 | percent |
| Hy3 | #27 | hy3 | Tencent | 57.9 | percent |
| GPT-5.4 | #28 | gpt-5-4 | OpenAI | 57.7 | percent |
| Qwen3.7-Plus | #29 | qwen-3-7-plus | Alibaba Cloud | 57.6 | percent |
| Qwen 3.6 Max (preview) | #30 | qwen3-6-max-preview | Alibaba Cloud | 57.3 | percent |
| MiMo-V2.5-Pro | #31 | mimo-v2-5-pro | Xiaomi | 57.2 | percent |
| Claude Opus 4.5 | #32 | claude-opus-4-5 | Anthropic | 57.1 | percent |
| GPT-5.3-Codex | #33 | gpt-5-3-codex | OpenAI | 56.8 | percent |
| Qwen3.6 Plus | #34 | qwen3-6-plus | Alibaba Cloud | 56.6 | percent |
| Step 3.7 Flash | #35 | step-3-7-flash | StepFun | 56.3 | percent |
| MiniMax M2.7 | #36 | minimax-m2-7 | MiniMax | 56.2 | percent |
| MiMo-V2.5 | #37 | mimo-v2-5 | Xiaomi | 56.1 | percent |
| Inkling-Small | #38 | inkling-small | Thinking Machines Lab | 55.9 | percent |
| GPT-5.2 | #39 | gpt-5-2 | OpenAI | 55.6 | percent |
| DeepSeek V4 Pro | #40 | deepseek-v4-pro | DeepSeek | 55.4 | percent |
| Gemini 3.5 Flash | #41 | gemini-3-5-flash | 55.1 | percent | |
| GLM-5 | #42 | glm-5 | Z.AI | 55.1 | percent |
| Inkling | #43 | inkling | Thinking Machines Lab | 54.3 | percent |
| Gemini 3.1 Pro Preview | #44 | gemini-3-1-pro-preview | 54.2 | percent | |
| Gemini 3.5 Flash-Lite | #45 | gemini-3-5-flash-lite | 54.2 | percent | |
| Qwen3.6 27B | #46 | qwen3-6-27b | Alibaba Cloud | 53.5 | percent |
| Claude Opus 4.6 | #47 | claude-opus-4-6 | Anthropic | 53.4 | percent |
| MAI-Thinking-1 | #48 | mai-thinking-1 | Microsoft | 52.8 | percent |
| DeepSeek V4 Flash | #49 | deepseek-v4-flash | DeepSeek | 52.6 | percent |
| Muse Spark | #50 | muse-spark | Meta | 52.4 | percent |
| Grok 4.20 | #51 | grok-4-20 | xAI | 51.8 | percent |
| Muse Glimmer 30B | #52 | muse-glimmer-30b | Meta | 51.2 | percent |
| Qwen3.5 397B A17B | #53 | qwen3-5-397b | Alibaba Cloud | 50.9 | percent |
| Kimi K2.5 | #54 | kimi-k2-5 | Moonshot AI | 50.7 | percent |
| Ornith-1.0-35B | #55 | ornith-1-0-35b | DeepReinforce AI | 50.4 | percent |
| Gemini 3 Flash | #56 | gemini-3-flash | 49.6 | percent | |
| Qwen3.6-35B-A3B | #57 | qwen3-6-35b-a3b | Alibaba Cloud | 49.5 | percent |
| Laguna M.1 | #58 | laguna-m-1 | Poolside | 49.2 | percent |
| Laguna XS 2.1 | #59 | laguna-xs-2-1 | Poolside | 47.6 | percent |
| Laguna XS.2 | #60 | laguna-xs-2 | Poolside | 46.3 | percent |
| Ornith-1.0-9B | #61 | ornith-1-0-9b | DeepReinforce AI | 42.9 | percent |
One best compatible score per canonical product · higher is better
| Rank | Model | Weights | Evidence | Score |
|---|---|---|---|---|
| 01 | Claude Mythos 5Anthropic | Closed weights | Source carrier | 80.3% |
| 02 | Claude Fable 5Anthropic | Closed weights | Source carrier | 80% |
| 03 | Claude Opus 5Anthropic | Closed weights | Source carrier | 79.2% |
| 04 | Sakana Fugu-UltraSakana AI | Closed weights | Source carrier | 73.7% |
| 05 | Claude Opus 4.8Anthropic | Closed weights | Source carrier | 69.2% |
| 06 | Qwen3.8 MaxAlibaba Cloud | Open weights | Direct source result | 67.7% |
| 07 | Grok 4.5xAI | Closed weights | Source carrier | 64.7% |
| 08 | GPT-5.6 SolOpenAI | Closed weights | Source carrier | 64.6% |
| 09 | Claude Opus 4.7Anthropic | Closed weights | Source carrier | 64.3% |
| 10 | GPT-5.6 TerraOpenAI | Closed weights | Source carrier | 63.4% |
Each distinct published measurement is retained. Same-snapshot canonical and configuration projections appear once.
| Published model / configuration | Score | Evidence & protocol | Source & dates |
|---|---|---|---|
Qwen3.8-Flash-Next public default xhigh configurationQwen3.8-Flash-NextExact identityqwen-3-8-flash-next-xhigh Canonical product: qwen-3-8-flash-next | 62.5% | Direct source resultsource-checkedVersion & systemv1 Claude Code | Qwen3.8-Flash-Next current official launch pageObserved Checked |
Qwen3.8-27B (xhigh default thinking configuration) as published in Qwen's Qwen3.8-Flash-Next comparison tableQwen3.8-27BExact identityqwen-3-8-27b-xhigh Canonical product: qwen-3-8-27b | 61.7% | Relative comparisonprovider-reportedVersion & systemv1 Claude Code | Qwen3.8-Flash-Next launch and official provider evaluationsObserved Checked |
DeepSeek-V4-Flash-0731 (provider-published configuration) as published in Qwen's Qwen3.8-Flash-Next comparison tableDeepSeek V4 Flash 0731Exact identitydeepseek-v4-flash-0731-deepseek-0813-release-unspecified Canonical product: deepseek-v4-flash-0731 | 56% | Relative comparisonprovider-reportedVersion & systemv1 Claude Code | Qwen3.8-Flash-Next launch and official provider evaluationsObserved Checked |
Qwen3.7-Plus (provider-published configuration) as published in Qwen's Qwen3.8-Flash-Next comparison tableQwen3.7-PlusExact identityqwen-3-7-plus-unspecified Canonical product: qwen-3-7-plus | 55.8% | Relative comparisonprovider-reportedVersion & systemv1 Claude Code | Qwen3.8-Flash-Next launch and official provider evaluationsObserved Checked |
Claude Opus 4.6 (Max) as published in Qwen's Qwen3.8-Flash-Next comparison tableClaude Opus 4.6Exact identityclaude-opus-4-6-max Canonical product: claude-opus-4-6 | 53.4% | Relative comparisonprovider-reportedVersion & systemv1 Claude Code | Qwen3.8-Flash-Next launch and official provider evaluationsObserved Checked |
Qwen3.8-27B (xhigh)Qwen3.8-27BExact identityqwen-3-8-27b-xhigh Canonical product: qwen-3-8-27b | 61.7% | Direct source resultprovider-reportedVersion & systemv1 Qwen3.8-27B official text table | Qwen3.8-27B official model cardObserved Checked |
Qwen3.7-Plus as published by QwenQwen3.7-PlusExact identityqwen-3-7-plus-unspecified Canonical product: qwen-3-7-plus | 57.6% | Relative comparisonprovider-reportedVersion & systemv1 Qwen3.8-27B official text table | Qwen3.8-27B official model cardObserved Checked |
Qwen3.6-27B as published by QwenQwen3.6 27BExact identityqwen3-6-27b-default Canonical product: qwen3-6-27b | 53.5% | Relative comparisonprovider-reportedVersion & systemv1 Qwen3.8-27B official text table | Qwen3.8-27B official model cardObserved Checked |
Claude Opus 4.6 Max as published by QwenClaude Opus 4.6Exact identityclaude-opus-4-6-max Canonical product: claude-opus-4-6 | 53.4% | Relative comparisonprovider-reportedVersion & systemv1 Qwen3.8-27B official text table | Qwen3.8-27B official model cardObserved Checked |
Muse Glimmer-30B; high reasoning; temperature=1.0; top_p=0.95; top_k=64Muse Glimmer 30BExact identitymuse-glimmer-30b-high Canonical product: muse-glimmer-30b | 51.2% | Public referenceprovider-reportedVersion & systemv1 / original 731-task set Meta agentic coding scaffold, averaged across four runs | Muse Glimmer Evaluation MethodologyObserved Checked |
From result to context
Public single-attempt software engineering tasks drawn from repositories outside the original SWE-bench set.
The summary shows one best compatible source result per canonical product. Versions, effort settings and execution systems remain attached to the underlying records.
Scores from different versions or harnesses may not be interchangeable. The published source rows preserve those distinctions and their original units. A source result is not automatically an input to a current index.
This catalogue entry preserves available source evidence. Current indices admit only their specifically reviewed tracks and configurations.
Contamination risk: Unknown. Lifecycle: Active.