A task with its own rules.
A benchmark for generating scientific-computing code from expert-authored specifications.
- Organisation
- SciCode authors
- Version
- Current / rolling
Loading benchmark header
Loading benchmark registry
Coding / Benchmark profile
A benchmark for generating scientific-computing code from expert-authored specifications.
Observed results
20 ranked models · higher is better · labels show rank and score
| Model | Rank | Model ID | Provider | Score | Unit |
|---|---|---|---|---|---|
| Claude Fable 5 | #1 | claude-fable-5 | Anthropic | 60.2 | percent |
| Gemini 3.1 Pro Preview | #2 | gemini-3-1-pro-preview | 58.9 | percent | |
| Kimi K3 | #3 | kimi-k3 | Moonshot AI | 58.7 | percent |
| Muse Spark 1.1 | #4 | muse-spark-1-1 | Meta | 58.2175925925926 | percent |
| Gemini 3.7 Flash | #5 | gemini-3-7-flash | 57.8703703703704 | percent | |
| GPT-5.4 | #6 | gpt-5-4 | OpenAI | 56.6 | percent |
| GLM-5.3 | #7 | glm-5-3 | Z.AI | 56.4814814814815 | percent |
| Muse Spark 1.2 | #8 | muse-spark-1-2 | Meta | 56.3657407407407 | percent |
| GPT-5.5 | #9 | gpt-5-5 | OpenAI | 56.1342592592593 | percent |
| GPT-5.6 Sol | #10 | gpt-5-6-sol | OpenAI | 56.1342592592593 | percent |
| Claude Opus 5 | #11 | claude-opus-5 | Anthropic | 55.7 | percent |
| GPT-5.2 Codex | #12 | gpt-5-2-codex | OpenAI | 54.6296 | percent |
| Claude Opus 4.7 | #13 | claude-opus-4-7 | Anthropic | 54.513888888889 | percent |
| Grok 4.5 | #14 | grok-4-5 | xAI | 54.1 | percent |
| GPT-5.6 Terra | #15 | gpt-5-6-terra | OpenAI | 53.9351851851852 | percent |
| Claude Sonnet 5 | #16 | claude-sonnet-5 | Anthropic | 53.6 | percent |
| Grok 4.6 | #17 | grok-4-6 | xAI | 53.587962962963 | percent |
| Claude Opus 4.8 | #18 | claude-opus-4-8 | Anthropic | 53.5 | percent |
| Kimi K2.6 | #19 | kimi-k2-6 | Moonshot AI | 53.5 | percent |
| Qwen3.7-Max | #20 | qwen-3-7-max | Alibaba Cloud | 53.5 | percent |
| GPT-5.3-Codex | #21 | gpt-5-3-codex | OpenAI | 53.2 | percent |
| Gemini 3.5 Flash | #22 | gemini-3-5-flash | 53.1 | percent | |
| Gemini 3.6 Flash | #23 | gemini-3-6-flash | 52.7 | percent | |
| GPT-5.6 Luna | #24 | gpt-5-6-luna | OpenAI | 52.546296296296305 | percent |
| GPT-5.2 | #25 | gpt-5-2 | OpenAI | 52.1 | percent |
| Claude Opus 4.6 | #26 | claude-opus-4-6 | Anthropic | 51.9 | percent |
| Muse Spark | #27 | muse-spark | Meta | 51.5 | percent |
| Qwen3.7-Plus | #28 | qwen-3-7-plus | Alibaba Cloud | 51.3 | percent |
| GLM-5.2 | #29 | glm-5-2 | Z.AI | 50.5 | percent |
| MiMo-V2.5-Pro | #30 | mimo-v2-5-pro | Xiaomi | 50.2 | percent |
| DeepSeek V4 Pro | #31 | deepseek-v4-pro | DeepSeek | 50 | percent |
| Gemini 3 Flash | #32 | gemini-3-flash | 49.9 | percent | |
| GPT-5.4 mini | #33 | gpt-5-4-mini | OpenAI | 49.9 | percent |
| Claude Opus 4.5 Thinking | #34 | claude-opus-4-5-thinking | Anthropic | 49.5 | percent |
| DeepSeek V4 Pro 0813 | #35 | deepseek-v4-pro-0813 | DeepSeek | 49.189814814814795 | percent |
| Kimi K2.5 | #36 | kimi-k2-5 | Moonshot AI | 49 | percent |
| Hy3 | #37 | hy3 | Tencent | 47.6 | percent |
| Hy3 Preview | #38 | hy3-preview | Tencent | 47.6 | percent |
| Kimi K2.7 Code | #39 | kimi-k2-7-code | Moonshot AI | 47.5 | percent |
| Grok 4.3 | #40 | grok-4-3 | xAI | 47.3 | percent |
| Claude Opus 4.5 | #41 | claude-opus-4-5 | Anthropic | 47 | percent |
| MiniMax M2.7 | #42 | minimax-m2-7 | MiniMax | 47 | percent |
| Claude Sonnet 4.6 | #43 | claude-sonnet-4-6 | Anthropic | 46.9 | percent |
| GPT-5.4 nano | #44 | gpt-5-4-nano | OpenAI | 46.9 | percent |
| Qwen 3.6 Max (preview) | #45 | qwen3-6-max-preview | Alibaba Cloud | 46.9 | percent |
| Qwen3.6 Max | #46 | qwen3-6-max | Alibaba Cloud | 46.875 | percent |
| Qwen3.8-Flash-Next | #47 | qwen-3-8-flash-next | Alibaba Cloud | 46.875 | percent |
| DeepSeek V4 Flash Vision Exp | #48 | deepseek-v4-flash-vision-exp | DeepSeek | 46.6435185185185 | percent |
| o4-mini | #49 | o4-mini | OpenAI | 46.5278 | percent |
| GLM-5 | #50 | glm-5 | Z.AI | 46.2 | percent |
| Inkling | #51 | inkling | Thinking Machines Lab | 46.1 | percent |
| GLM-5.3-Flash | #52 | glm-5-3-flash | Z.AI | 46.0648148148148 | percent |
| Grok 4 | #53 | grok-4 | xAI | 45.7176 | percent |
| Grok 4.20 | #54 | grok-4-20 | xAI | 45.6019 | percent |
| MiniMax M3 | #55 | minimax-m3 | MiniMax | 45.4 | percent |
| GLM-4.7 | #56 | glm-4-7 | Z.AI | 45.1389 | percent |
| DeepSeek V4 Flash | #57 | deepseek-v4-flash | DeepSeek | 44.9 | percent |
| Claude Sonnet 4.5 | #58 | claude-sonnet-4-5 | Anthropic | 44.6759 | percent |
| Grok 4 Fast | #59 | grok-4-fast | xAI | 44.213 | percent |
| Grok 4 Fast (Reasoning) | #60 | grok-4-fast-reasoning | xAI | 44.2 | percent |
| Grok 4.1 Fast (Reasoning) | #61 | grok-4-1-fast-reasoning | xAI | 44.2 | percent |
| GLM-5.1 | #62 | glm-5-1 | Z.AI | 43.8 | percent |
| GLM-5-Turbo | #63 | glm-5-turbo | Z.AI | 43.6 | percent |
| Muse Glimmer 30B | #64 | muse-glimmer-30b | Meta | 43.6 | percent |
| GLM-5V-Turbo | #65 | glm-5v-turbo | Z.AI | 43.5 | percent |
| Gemma 4 31B | #66 | gemma-4-31b | 43.4 | percent | |
| GPT-5.1 | #67 | gpt-5-1 | OpenAI | 43.3 | percent |
| MiMo-V2.5 | #68 | mimo-v2-5 | Xiaomi | 43.0556 | percent |
| Qwen3 Max | #69 | qwen3-max | Alibaba Cloud | 43.0556 | percent |
| GPT-5 | #70 | gpt-5 | OpenAI | 42.9398 | percent |
| Gemini 2.5 Pro | #71 | gemini-2-5-pro | 42.8241 | percent | |
| Nova 2 Pro | #72 | nova-2-pro | Amazon | 42.7083 | percent |
| MiniMax M2.5 | #73 | minimax-m2-5 | MiniMax | 42.5926 | percent |
| MiMo-V2-Pro | #74 | mimo-v2-pro | Xiaomi | 42.5 | percent |
| Kimi K2 Thinking | #75 | kimi-k2-thinking | Moonshot AI | 42.3611 | percent |
| Qwen3.5 122B | #76 | qwen3-5-122b | Alibaba Cloud | 42.0139 | percent |
| Qwen3.5 397B A17B | #77 | qwen3-5-397b | Alibaba Cloud | 42.0139 | percent |
| Qwen3.5-122B-A10B | #78 | qwen3-5-122b-a10b | Alibaba Cloud | 42.013888888889 | percent |
| Gemini 3.1 Flash-Lite | #79 | gemini-3-1-flash-lite | 41.9 | percent | |
| o3 | #80 | o3 | OpenAI | 41 | percent |
| GPT-5 mini | #81 | gpt-5-mini | OpenAI | 40.9722 | percent |
| Claude 4.1 Opus Thinking | #82 | claude-4-1-opus-thinking | Anthropic | 40.9 | percent |
| Gemini 3.5 Flash-Lite | #83 | gemini-3-5-flash-lite | 40.9 | percent | |
| Claude Opus 4.1 | #84 | claude-opus-4-1 | Anthropic | 40.8979 | percent |
| GPT-5 Codex | #85 | gpt-5-codex | OpenAI | 40.8565 | percent |
| MiniMax M2.1 | #86 | minimax-m2-1 | MiniMax | 40.7407 | percent |
| Qwen3.6 Plus | #87 | qwen3-6-plus | Alibaba Cloud | 40.7 | percent |
| DeepSeek V3.1 Terminus | #88 | deepseek-v3-1-terminus | DeepSeek | 40.625 | percent |
| Grok 3 mini | #89 | grok-3-mini | xAI | 40.625 | percent |
| Gemini 2.5 Flash | #90 | gemini-2-5-flash | 40.5093 | percent | |
| Qwen3.5 Omni Plus | #91 | qwen3-5-omni-plus | Alibaba Cloud | 40.5093 | percent |
| GPT-4.1 mini | #92 | gpt-4-1-mini | OpenAI | 40.4 | percent |
| DeepSeek-R1 | #93 | deepseek-r1 | DeepSeek | 40.3 | percent |
| Claude Sonnet 3.7 | #94 | claude-sonnet-3-7 | Anthropic | 40.2778 | percent |
| GPT-5.1-Codex | #95 | gpt-5-1-codex | OpenAI | 40.2 | percent |
| GPT-5.1-Codex-Max | #96 | gpt-5-1-codex-max | OpenAI | 40.2 | percent |
| Claude Sonnet 4 | #97 | claude-sonnet-4 | Anthropic | 40.0463 | percent |
| Gemma 4 26B | #98 | gemma-4-26b | 40.0463 | percent | |
| Gemma 4 26B A4B | #99 | gemma-4-26b-a4b | 40.046296296296 | percent | |
| Step 3.7 Flash | #100 | step-3-7-flash | StepFun | 40 | percent |
| Nemotron 3 Ultra | #101 | nemotron-3-ultra | NVIDIA | 39.9306 | percent |
| o3-mini | #102 | o3-mini | OpenAI | 39.9 | percent |
| Claude Opus 4 | #103 | claude-opus-4 | Anthropic | 39.8148 | percent |
| Qwen3.6 27B | #104 | qwen3-6-27b | Alibaba Cloud | 39.8148 | percent |
| Mistral Medium 3.5 128B | #105 | mistral-medium-3-5-128b | Mistral AI | 39.6 | percent |
| Qwen3.5 27B | #106 | qwen3-5-27b | Alibaba Cloud | 39.5 | percent |
| DeepSeek V3.1 | #107 | deepseek-v3-1 | DeepSeek | 39.12037037037 | percent |
| GPT-OSS 120B | #108 | gpt-oss-120b | OpenAI | 38.9 | percent |
| DeepSeek V3.2 | #109 | deepseek-v3-2 | DeepSeek | 38.8889 | percent |
| GLM-4.6 | #110 | glm-4-6 | Z.AI | 38.4259 | percent |
| Gemma 4 12B Unified | #111 | gemma-4-12b | 38.2 | percent | |
| GPT-4.1 | #112 | gpt-4-1 | OpenAI | 38.1 | percent |
| Mistral Small 4 | #113 | mistral-small-4 | Mistral AI | 38 | percent |
| Command A+ | #114 | command-a-plus | Cohere | 37.8472 | percent |
| Qwen3.5 35B | #115 | qwen3-5-35b | Alibaba Cloud | 37.7315 | percent |
| Qwen3.5-35B-A3B | #116 | qwen3-5-35b-a3b | Alibaba Cloud | 37.7 | percent |
| DeepSeek R1 Distill Qwen 32B | #117 | deepseek-r1-distill-qwen-32b | DeepSeek | 37.6 | percent |
| Claude 4 Sonnet | #118 | claude-4-sonnet | Anthropic | 37.3 | percent |
| Ling 2.6 1T | #119 | ling-2-6-1t | InclusionAI | 37.037 | percent |
| MiMo-V2-Omni | #120 | mimo-v2-omni | Xiaomi | 36.7 | percent |
| Grok Code Fast 1 | #121 | grok-code-fast-1 | xAI | 36.2269 | percent |
| Mistral Large 3 | #122 | mistral-large-3 | Mistral AI | 36.2 | percent |
| Trinity-Large-Thinking | #123 | trinity-large-thinking | Arcee AI | 36.111111111111 | percent |
| MiniMax M2 | #124 | minimax-m2 | MiniMax | 36.1111 | percent |
| Trinity-Large-Preview | #125 | trinity-large-preview | Arcee AI | 36.1 | percent |
| Nemotron 3 Super | #126 | nemotron-3-super | NVIDIA | 35.9954 | percent |
| o1 | #127 | o1 | OpenAI | 35.8 | percent |
| Qwen3.6-35B-A3B | #128 | qwen3-6-35b-a3b | Alibaba Cloud | 35.8 | percent |
| K-Exaone | #129 | k-exaone | LG AI Research | 35.6 | percent |
| LongCat-2.0 | #130 | longcat-2-0 | Meituan | 35.416666666667 | percent |
| DeepSeek V3 | #131 | deepseek-v3 | DeepSeek | 35.4 | percent |
| Nemotron Ultra 253B | #132 | nemotron-ultra-253b | NVIDIA | 34.7 | percent |
| Kimi K2 | #133 | kimi-k2 | Moonshot AI | 34.5 | percent |
| GPT-OSS 20B | #134 | gpt-oss-20b | OpenAI | 34.4 | percent |
| GPT-4o | #135 | gpt-4o | OpenAI | 33.3 | percent |
| Llama 4 Maverick | #136 | llama-4-maverick | Meta | 33.101851851852 | percent |
| Mistral Medium 3 | #137 | mistral-medium-3 | Mistral AI | 33.1 | percent |
| NVIDIA Nemotron 3.5 Lightning 30B-A3B | #138 | nemotron-3-5-lightning-30b-a3b | NVIDIA | 32.6 | percent |
| GPT-4 Turbo | #139 | gpt-4-turbo | OpenAI | 31.9 | percent |
| Kimi K2 0905 | #140 | kimi-k2-0905 | Moonshot AI | 30.6713 | percent |
| GLM-4.5-Air | #141 | glm-4-5-air | Z.AI | 30.6 | percent |
| Llama 3.1 405B | #142 | llama-3-1-405b | Meta | 29.9 | percent |
| Nemotron 3 Nano 30B | #143 | nemotron-3-nano-30b | NVIDIA | 29.62962962963 | percent |
| Grok 4.1 Fast | #144 | grok-4-1-fast | xAI | 29.6 | percent |
| Gemini 1.5 Pro | #145 | gemini-1-5-pro | 29.5 | percent | |
| Mistral Large 2 | #146 | mistral-large-2 | Mistral AI | 29.2 | percent |
| Nemotron 3 Nano Omni 30B A3B | #147 | nemotron-3-nano-omni-30b-a3b | NVIDIA | 27.8 | percent |
| Ling 2.6 Flash | #148 | ling-2-6-flash | InclusionAI | 27.1 | percent |
| Qwen2.5 Coder 32B Instruct | #149 | qwen2-5-coder-32b-instruct | Alibaba Cloud | 27.1 | percent |
| Sarvam 105B | #150 | sarvam-105b | Sarvam | 26.4 | percent |
| Phi-4 | #151 | phi-4 | Microsoft | 26 | percent |
| GPT-4.1 nano | #152 | gpt-4-1-nano | OpenAI | 25.925925925926 | percent |
| Exaone 4.0 32B | #153 | exaone-4-0-32b | LG AI Research | 25.2 | percent |
| Solar Pro 2 | #154 | solar-pro-2 | Upstage | 24.8 | percent |
| Gemma 4 E4B | #155 | gemma-4-e4b | 24.4 | percent | |
| Claude 3 Opus | #156 | claude-3-opus | Anthropic | 23.3 | percent |
| GPT-4o mini | #157 | gpt-4o-mini | OpenAI | 22.9 | percent |
| Gemma 3 27B | #158 | gemma-3-27b | 21.2 | percent | |
| Gemma 4 E2B | #159 | gemma-4-e2b | 20.9 | percent | |
| Nova Pro | #160 | nova-pro | Amazon | 20.8 | percent |
| Sarvam 30B | #161 | sarvam-30b | Sarvam | 19.2 | percent |
| Claude 3 Haiku | #162 | claude-3-haiku | Anthropic | 18.6 | percent |
| Llama 4 Scout | #163 | llama-4-scout | Meta | 17.013888888889 | percent |
| Gemini 1.0 Pro | #164 | gemini-1-0-pro | 11.7 | percent | |
| Granite-4.0-1B | #165 | granite-4-0-1b | IBM | 8.7 | percent |
| Granite-4.0-H-1B | #166 | granite-4-0-h-1b | IBM | 8.2 | percent |
| LFM2.5-8B-A1B | #167 | lfm2-5-8b-a1b | LiquidAI | 7.8 | percent |
| Exaone 4.0 1.2B | #168 | exaone-4-0-1-2b | LG AI Research | 7.4 | percent |
| LFM2.5-VL-1.6B-Extract | #169 | lfm2-5-vl-1-6b-extract | LiquidAI | 3 | percent |
| Granite-4.0-H-350M | #170 | granite-4-0-h-350m | IBM | 1.7 | percent |
| Granite-4.0-350M | #171 | granite-4-0-350m | IBM | 0.9 | percent |
One best compatible score per canonical product · higher is better
| Rank | Model | Weights | Evidence | Score |
|---|---|---|---|---|
| 01 | Claude Fable 5Anthropic | Closed weights | Source carrier | 60.2% |
| 02 | Gemini 3.1 Pro PreviewGoogle | Closed weights | Source carrier | 58.9% |
| 03 | Kimi K3Moonshot AI | Open weights | Public reference | 58.7% |
| 04 | Muse Spark 1.1Meta | Closed weights | Public reference | 58.2% |
| 05 | Gemini 3.7 FlashGoogle | Closed weights | Public reference | 57.9% |
| 06 | GPT-5.4OpenAI | Closed weights | Source carrier | 56.6% |
| 07 | GLM-5.3Z.AI | Open announced | Public reference | 56.5% |
| 08 | Muse Spark 1.2Meta | Open announced | Public reference | 56.4% |
| 09 | GPT-5.5OpenAI | Closed weights | Public reference | 56.1% |
| 10 | GPT-5.6 SolOpenAI | Closed weights | Public reference | 56.1% |
Each distinct published measurement is retained. Same-snapshot canonical and configuration projections appear once.
| Published model / configuration | Score | Evidence & protocol | Source & dates |
|---|---|---|---|
Claude Opus 4.7 (Adaptive Reasoning, Max Effort) (Artificial Analysis independent run)Claude Opus 4.7Exact identityclaude-opus-4-7-max Canonical product: claude-opus-4-7 | 54.5% | Public referenceindependently-verifiedVersion & system2024 Artificial Analysis SciCode evaluation | Claude Opus 4.7 (Adaptive Reasoning, Max Effort) individual evaluationsObserved Checked |
Claude Opus 4.6 (Adaptive Reasoning, Max Effort) (Artificial Analysis independent run)Claude Opus 4.6Exact identityclaude-opus-4-6-max Canonical product: claude-opus-4-6 | 51.9% | Public referenceindependently-verifiedVersion & system2024 Artificial Analysis SciCode evaluation | Claude Opus 4.6 (Adaptive Reasoning, Max Effort) individual evaluationsObserved Checked |
GLM-5.2 (max) (Artificial Analysis independent run)GLM-5.2Exact identityglm-5-2-max Canonical product: glm-5-2 | 50.5% | Public referenceindependently-verifiedVersion & system2024 Artificial Analysis SciCode evaluation | GLM-5.2 (max) individual evaluationsObserved Checked |
Kimi K2.5 (Reasoning) (Artificial Analysis completed independent run)Kimi K2.5Exact identitykimi-k2-5-thinking Canonical product: kimi-k2-5 | 49.0% | Public referenceindependently-verifiedVersion & system2024 Artificial Analysis SciCode evaluation | Kimi K2.5 (Reasoning) current model and individual-evaluation pageObserved Checked |
Qwen3.8-Flash-Next (Artificial Analysis independent run)Qwen3.8-Flash-NextExact identityqwen-3-8-flash-next-aa-unspecified Canonical product: qwen-3-8-flash-next | 46.9% | Public referenceindependently-verifiedVersion & system2024 Artificial Analysis SciCode evaluation | Qwen3.8-Flash-Next individual evaluationsObserved Checked |
Qwen3.5 122B A10B (Reasoning) (Artificial Analysis completed independent run)Qwen3.5-122B-A10BExact identityqwen3-5-122b-a10b-aa-reasoning-default Canonical product: qwen3-5-122b-a10b | 42.0% | Public referenceindependently-verifiedVersion & system2024 Artificial Analysis SciCode evaluation | Qwen3.5 122B A10B (Reasoning) current model and individual-evaluation pageObserved Checked |
Qwen3.5 397B A17B (Reasoning) (Artificial Analysis completed independent run)Qwen3.5 397B A17BExact identityqwen3-5-397b-thinking Canonical product: qwen3-5-397b | 42.0% | Public referenceindependently-verifiedVersion & system2024 Artificial Analysis SciCode evaluation | Qwen3.5 397B A17B (Reasoning) current model and individual-evaluation pageObserved Checked |
GPT-4.1 mini (Artificial Analysis completed independent run)GPT-4.1 miniExact identitygpt-4-1-mini-epoch-gpt-4-1-mini-2025-04-14 Canonical product: gpt-4-1-mini | 40.4% | Public referenceindependently-verifiedVersion & system2024 Artificial Analysis SciCode evaluation | GPT-4.1 mini current model and individual-evaluation pageObserved Checked |
Gemma 4 26B A4B (Reasoning) (Artificial Analysis completed independent run)Gemma 4 26B A4BExact identitygemma-4-26b-a4b-aa-reasoning-default Canonical product: gemma-4-26b-a4b | 40.0% | Public referenceindependently-verifiedVersion & system2024 Artificial Analysis SciCode evaluation | Gemma 4 26B A4B (Reasoning) current model and individual-evaluation pageObserved Checked |
Mistral Medium 3.5 (Artificial Analysis completed independent run)Mistral Medium 3.5 128BExact identitymistral-medium-3-5-128b-aa-reasoning-default Canonical product: mistral-medium-3-5-128b | 39.6% | Public referenceindependently-verifiedVersion & system2024 Artificial Analysis SciCode evaluation | Mistral Medium 3.5 current model and individual-evaluation pageObserved Checked |
| Model | Result | Execution & attribution | Original source |
|---|---|---|---|
| Qwen3.8 Max (0902)scicode · exact AA profile b112af07-3bd5-4647-b09f-b23204361bb2 | 52.083percent | AA current published profile; effort label not exportedindependent evaluatorRetained as additional evidence. Existing Core point is unchanged pending an accepted complete own-evidence amendment; incompatible versions remain separate. | Artificial Analysis ↗Publication date not recorded · checked 2026-09-16 |
From result to context
A benchmark for generating scientific-computing code from expert-authored specifications.
The summary shows one best compatible source result per canonical product. Versions, effort settings and execution systems remain attached to the underlying records.
Scores from different versions or harnesses may not be interchangeable. The published source rows preserve those distinctions and their original units. A source result is not automatically an input to a current index.
A reviewed track from this benchmark is represented in the following current indices. Each index applies its own protocol and eligibility rules.
Contamination risk: Low. Lifecycle: Active.