Coding / Benchmark profile
Cognition's 100-task software-engineering benchmark for whether coding agents produce mergeable, production-quality pull requests, scored for correctness, tests, scope, style, and maintainability through maintainer-authored rubrics.
Observed results
13 ranked models · higher is better · labels show rank and score
| Model | Rank | Model ID | Provider | Score | Unit |
|---|---|---|---|---|---|
| Claude Fable 5 | #1 | claude-fable-5 | Anthropic | 53.5 | percent |
| Claude Opus 5 | #2 | claude-opus-5 | Anthropic | 53.4 | percent |
| Claude Opus 4.8 | #3 | claude-opus-4-8 | Anthropic | 46.5 | percent |
| Gemini 3.7 Flash | #4 | gemini-3-7-flash | 43.6 | percent | |
| GPT-5.5 | #5 | gpt-5-5 | OpenAI | 43 | percent |
| Claude Sonnet 5 | #6 | claude-sonnet-5 | Anthropic | 42.7 | percent |
| SWE-1.7 | #7 | swe-1-7 | Cognition | 42.3 | percent |
| GPT-5.6 Terra | #8 | gpt-5-6-terra | OpenAI | 41.3 | percent |
| Claude Opus 4.7 | #9 | claude-opus-4-7 | Anthropic | 38.5 | percent |
| Gemini 3.6 Flash | #10 | gemini-3-6-flash | 34.4 | percent | |
| GPT-5.4 mini | #11 | gpt-5-4-mini | OpenAI | 27 | percent |
| Claude Opus 4.6 | #12 | claude-opus-4-6 | Anthropic | 26.9 | percent |
| Claude Sonnet 4.6 | #13 | claude-sonnet-4-6 | Anthropic | 24.3 | percent |
One best compatible score per canonical product · higher is better
| Rank | Model | Weights | Evidence | Score |
|---|---|---|---|---|
| 01 | Claude Fable 5Anthropic | Closed weights | Source carrier | 53.5% |
| 02 | Claude Opus 5Anthropic | Closed weights | Public reference | 53.4% |
| 03 | Claude Opus 4.8Anthropic | Closed weights | Source carrier | 46.5% |
| 04 | Gemini 3.7 FlashGoogle | Closed weights | Public reference | 43.6% |
| 05 | GPT-5.5OpenAI | Closed weights | Source carrier | 43% |
| 06 | Claude Sonnet 5Anthropic | Closed weights | Public reference | 42.7% |
| 07 | SWE-1.7Cognition | Closed weights | Source carrier | 42.3% |
| 08 | GPT-5.6 TerraOpenAI | Closed weights | Public reference | 41.3% |
| 09 | Claude Opus 4.7Anthropic | Closed weights | Source carrier | 38.5% |
| 10 | Gemini 3.6 FlashGoogle | Closed weights | Public reference | 34.4% |
Each distinct published measurement is retained. Same-snapshot canonical and configuration projections appear once.
| Published model / configuration | Score | Evidence & protocol | Source & dates |
|---|---|---|---|
Gemini 3.7 FlashGemini 3.7 FlashExact identitygemini-3-7-flash-medium Canonical product: gemini-3-7-flash | 43.6% | Public referenceprovider-reportedVersion & system1.1 Main Official FrontierCode 1.1 public leaderboard | Google DeepMind permanent refresh sourceObserved Checked |
Claude Sonnet 5Claude Sonnet 5Exact identityclaude-sonnet-5-max Canonical product: claude-sonnet-5 | 42.7% | Public referenceprovider-reportedVersion & system1.1 Main Official FrontierCode 1.1 public leaderboard | Google DeepMind permanent refresh sourceObserved Checked |
GPT-5.6 TerraGPT-5.6 TerraExact identitygpt-5-6-terra-max Canonical product: gpt-5-6-terra | 41.3% | Public referenceprovider-reportedVersion & system1.1 Main Official FrontierCode 1.1 public leaderboard | Google DeepMind permanent refresh sourceObserved Checked |
Gemini 3.6 FlashGemini 3.6 FlashExact identitygemini-3-6-flash-high Canonical product: gemini-3-6-flash | 34.4% | Public referenceprovider-reportedVersion & system1.1 Main Official FrontierCode 1.1 public leaderboard | Google DeepMind permanent refresh sourceObserved Checked |
Exact BenchLM registry variant Claude Fable 5; bulk export does not retain a complete upstream harness configuration.Claude Fable 5Exact identitySource label without a registered configuration ID Canonical product: claude-fable-5 | 53.5% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Claude Opus 5; bulk export does not retain a complete upstream harness configuration.Claude Opus 5Exact identitySource label without a registered configuration ID Canonical product: claude-opus-5 | 53.4% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Claude Opus 4.8; bulk export does not retain a complete upstream harness configuration.Claude Opus 4.8Exact identitySource label without a registered configuration ID Canonical product: claude-opus-4-8 | 46.5% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant GPT-5.5; bulk export does not retain a complete upstream harness configuration.GPT-5.5Exact identitySource label without a registered configuration ID Canonical product: gpt-5-5 | 43% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Claude Sonnet 5; bulk export does not retain a complete upstream harness configuration.Claude Sonnet 5Exact identitySource label without a registered configuration ID Canonical product: claude-sonnet-5 | 42.7% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant SWE-1.7; bulk export does not retain a complete upstream harness configuration.SWE-1.7Exact identitySource label without a registered configuration ID Canonical product: swe-1-7 | 42.3% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
| Model | Result | Execution & attribution | Original source |
|---|---|---|---|
| Fable 5.1FrontierCode 1.1 Main · Cognition release table | 50.9percent (published) | Release-table configuration; per-model effort and harness vary. See original footnotes.provider measured competitorSource-native comparison only. Different settings are not silently substituted into retained index protocols. | Cognition ↗Published 2026-09-10 · checked 2026-09-16 |
| GPT-5.6 SolFrontierCode 1.1 Main · Cognition release table | 47.5percent (published) | Release-table configuration; per-model effort and harness vary. See original footnotes.provider measured competitorSource-native comparison only. Different settings are not silently substituted into retained index protocols. | Cognition ↗Published 2026-09-10 · checked 2026-09-16 |
| SWE-1.7FrontierCode 1.1 Main · Cognition release table | 42percent (published) | Release-table configuration; per-model effort and harness vary. See original footnotes.provider subjectSource-native comparison only. Different settings are not silently substituted into retained index protocols. | Cognition ↗Published 2026-09-10 · checked 2026-09-16 |
| Grok 4.6FrontierCode 1.1 Main · Cognition release table | 48percent (published) | Release-table configuration; per-model effort and harness vary. See original footnotes.provider measured competitorSource-native comparison only. Different settings are not silently substituted into retained index protocols. | Cognition ↗Published 2026-09-10 · checked 2026-09-16 |
| SWE-2FrontierCode 1.1 Main · Cognition release table | 50percent (published) | Release-table configuration; per-model effort and harness vary. See original footnotes.provider subjectSource-native comparison only. Different settings are not silently substituted into retained index protocols. | Cognition ↗Published 2026-09-10 · checked 2026-09-16 |
| Kimi K3FrontierCode 1.1 Main · Cognition release table | 44.2percent (published) | Release-table configuration; per-model effort and harness vary. See original footnotes.provider measured competitorSource-native comparison only. Different settings are not silently substituted into retained index protocols. | Cognition ↗Published 2026-09-10 · checked 2026-09-16 |
| GPT-6 AstraFrontierCode 1.1 Main · Cognition release table | 53.3percent (published) | Release-table configuration; per-model effort and harness vary. See original footnotes.provider measured competitorSource-native comparison only. Different settings are not silently substituted into retained index protocols. | Cognition ↗Published 2026-09-10 · checked 2026-09-16 |
From result to context
Cognition's 100-task software-engineering benchmark for whether coding agents produce mergeable, production-quality pull requests, scored for correctness, tests, scope, style, and maintainability through maintainer-authored rubrics.
The summary shows one best compatible source result per canonical product. Versions, effort settings and execution systems remain attached to the underlying records.
Scores from different versions or harnesses may not be interchangeable. The published source rows preserve those distinctions and their original units. A source result is not automatically an input to a current index.
A reviewed track from this benchmark is represented in the following current indices. Each index applies its own protocol and eligibility rules.
Contamination risk: Unknown. Lifecycle: Active.
