A task with its own rules.
An agent benchmark for completing automation workflows in reproducible task environments.
- Organisation
- Moonshot AI
- Version
- 2026
Loading benchmark header
Loading benchmark registry
Agents / Benchmark profile
An agent benchmark for completing automation workflows in reproducible task environments.
Observed results
10 ranked models · higher is better · labels show rank and score
| Model | Rank | Model ID | Provider | Score | Unit |
|---|---|---|---|---|---|
| DeepSeek V4 Pro 0813 | #1 | deepseek-v4-pro-0813 | DeepSeek | 31.8 | percent |
| Kimi K3 | #2 | kimi-k3 | Moonshot AI | 30.8 | percent |
| Claude Fable 5 | #3 | claude-fable-5 | Anthropic | 29.1 | percent |
| Claude Opus 4.8 | #4 | claude-opus-4-8 | Anthropic | 27.2 | percent |
| Claude Opus 5 | #5 | claude-opus-5 | Anthropic | 26 | percent |
| DeepSeek V4 Flash Vision Exp | #6 | deepseek-v4-flash-vision-exp | DeepSeek | 25.7 | percent |
| DeepSeek V4 Flash | #7 | deepseek-v4-flash | DeepSeek | 25.1 | percent |
| DeepSeek V4 Flash 0731 | #8 | deepseek-v4-flash-0731 | DeepSeek | 25.1 | percent |
| GLM-5.2 | #9 | glm-5-2 | Z.AI | 12.9 | percent |
| DeepSeek V4 Pro | #10 | deepseek-v4-pro | DeepSeek | 12.8 | percent |
One best compatible score per canonical product · higher is better
| Rank | Model | Weights | Evidence | Score |
|---|---|---|---|---|
| 01 | DeepSeek V4 Pro 0813DeepSeek | Open weights | Public reference | 31.8% |
| 02 | Kimi K3Moonshot AI | Open weights | Source carrier | 30.8% |
| 03 | Claude Fable 5Anthropic | Closed weights | Public reference | 29.1% |
| 04 | Claude Opus 4.8Anthropic | Closed weights | Public reference | 27.2% |
| 05 | Claude Opus 5Anthropic | Closed weights | Public reference | 26% |
| 06 | DeepSeek V4 Flash Vision ExpDeepSeek | Not documented | Public reference | 25.7% |
| 07 | DeepSeek V4 FlashDeepSeek | Open weights | Source carrier | 25.1% |
| 08 | DeepSeek V4 Flash 0731DeepSeek | Open weights | Public reference | 25.1% |
| 09 | GLM-5.2Z.AI | Open weights | Public reference | 12.9% |
| 10 | DeepSeek V4 ProDeepSeek | Open weights | Public reference | 12.8% |
Each distinct published measurement is retained. Same-snapshot canonical and configuration projections appear once.
| Published model / configuration | Score | Evidence & protocol | Source & dates |
|---|---|---|---|
DeepSeek Harness Minimal Mode; max effort; top_p=0.95; temperature=1.0DeepSeek V4 Flash Vision ExpExact identitydeepseek-v4-flash-vision-exp-max-harness Canonical product: deepseek-v4-flash-vision-exp | 25.7% | Public referenceprovider-reportedVersion & system2026 public set DeepSeek Harness Minimal Mode | DeepSeek-V4-Flash-Vision-Exp release and provider evaluationObserved Checked |
DeepSeek V4 Pro 0813DeepSeek V4 Pro 0813Exact identitydeepseek-v4-pro-0813-deepseek-0813-release-unspecified Canonical product: deepseek-v4-pro-0813 | 31.8% | Public referenceprovider-reportedVersion & systemPublic 600-task set, August 2026 AutomationBench public set; DeepSeek V4 Pro 0813 official GA comparison table | DeepSeek permanent refresh sourceObserved Checked |
Kimi K3Kimi K3Exact identitykimi-k3-deepseek-0813-release-unspecified Canonical product: kimi-k3 | 30.8% | Public referenceprovider-reportedVersion & systemPublic 600-task set, August 2026 AutomationBench public set; DeepSeek V4 Pro 0813 official GA comparison table | DeepSeek permanent refresh sourceObserved Checked |
Fable 5 (with fallback)Claude Fable 5Exact identityclaude-fable-5-deepseek-0813-release-unspecified Canonical product: claude-fable-5 | 29.1% | Public referenceprovider-reportedVersion & systemPublic 600-task set, August 2026 AutomationBench public set; DeepSeek V4 Pro 0813 official GA comparison table | DeepSeek permanent refresh sourceObserved Checked |
Claude Opus 4.8Claude Opus 4.8Exact identityclaude-opus-4-8-deepseek-0813-release-unspecified Canonical product: claude-opus-4-8 | 27.2% | Public referenceprovider-reportedVersion & systemPublic 600-task set, August 2026 AutomationBench public set; DeepSeek V4 Pro 0813 official GA comparison table | DeepSeek permanent refresh sourceObserved Checked |
DeepSeek V4 Flash 0731DeepSeek V4 Flash 0731Exact identitydeepseek-v4-flash-0731-deepseek-0813-release-unspecified Canonical product: deepseek-v4-flash-0731 | 25.1% | Public referenceprovider-reportedVersion & systemPublic 600-task set, August 2026 AutomationBench public set; DeepSeek V4 Pro 0813 official GA comparison table | DeepSeek permanent refresh sourceObserved Checked |
GLM-5.2GLM-5.2Exact identityglm-5-2-deepseek-0813-release-unspecified Canonical product: glm-5-2 | 12.9% | Public referenceprovider-reportedVersion & systemPublic 600-task set, August 2026 AutomationBench public set; DeepSeek V4 Pro 0813 official GA comparison table | DeepSeek permanent refresh sourceObserved Checked |
DeepSeek V4 Pro PreviewDeepSeek V4 ProExact identitydeepseek-v4-pro-deepseek-0813-release-unspecified Canonical product: deepseek-v4-pro | 12.8% | Public referenceprovider-reportedVersion & systemPublic 600-task set, August 2026 AutomationBench public set; DeepSeek V4 Pro 0813 official GA comparison table | DeepSeek permanent refresh sourceObserved Checked |
DeepSeek V4 Flash PreviewDeepSeek V4 FlashExact identitydeepseek-v4-flash-deepseek-0813-release-unspecified Canonical product: deepseek-v4-flash | 10.8% | Public referenceprovider-reportedVersion & systemPublic 600-task set, August 2026 AutomationBench public set; DeepSeek V4 Pro 0813 official GA comparison table | DeepSeek permanent refresh sourceObserved Checked |
Exact BenchLM registry variant Kimi K3; bulk export does not retain a complete upstream harness configuration.Kimi K3Exact identitySource label without a registered configuration ID Canonical product: kimi-k3 | 30.8% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
From result to context
An agent benchmark for completing automation workflows in reproducible task environments.
The summary shows one best compatible source result per canonical product. Versions, effort settings and execution systems remain attached to the underlying records.
Scores from different versions or harnesses may not be interchangeable. The published source rows preserve those distinctions and their original units. A source result is not automatically an input to a current index.
This catalogue entry preserves available source evidence. Current indices admit only their specifically reviewed tracks and configurations.
Contamination risk: Unknown. Lifecycle: Active.