A task with its own rules.
A more robust extension of MMMU intended to reduce shortcut-based answering.
- Organisation
- MMMU-Pro authors
- Version
- Current / rolling
Loading benchmark header
Loading benchmark registry
Multimodal / Benchmark profile
A more robust extension of MMMU intended to reduce shortcut-based answering.
Observed results
20 ranked models · higher is better · labels show rank and score
| Model | Rank | Model ID | Provider | Score | Unit |
|---|---|---|---|---|---|
| Claude Opus 5 | #1 | claude-opus-5 | Anthropic | 84.7 | percent |
| Gemini 3.5 Flash | #2 | gemini-3-5-flash | 84.3 | percent | |
| Gemini 3.1 Pro Preview | #3 | gemini-3-1-pro-preview | 83.9 | percent | |
| GPT-5.6 Sol | #4 | gpt-5-6-sol | OpenAI | 83.41 | percent |
| Gemini 3.6 Flash | #5 | gemini-3-6-flash | 83.2 | percent | |
| Qwen3.8 Max | #6 | qwen-3-8-max | Alibaba Cloud | 82.3 | percent |
| Kimi K3 | #7 | kimi-k3 | Moonshot AI | 81.6 | percent |
| GPT-5.4 | #8 | gpt-5-4 | OpenAI | 81.2 | percent |
| GPT-5.5 | #9 | gpt-5-5 | OpenAI | 81.2 | percent |
| GPT-5.6 Terra | #10 | gpt-5-6-terra | OpenAI | 80.7 | percent |
| Muse Spark | #11 | muse-spark | Meta | 80.5 | percent |
| Qwen3.7-Plus | #12 | qwen-3-7-plus | Alibaba Cloud | 80.5 | percent |
| Grok 4.5 | #13 | grok-4-5 | xAI | 80.4 | percent |
| Kimi K2.6 | #14 | kimi-k2-6 | Moonshot AI | 79.4 | percent |
| Gemini 3.5 Flash-Lite | #15 | gemini-3-5-flash-lite | 79 | percent | |
| Claude Opus 4.7 | #16 | claude-opus-4-7 | Anthropic | 78.8 | percent |
| Qwen3.6 Plus | #17 | qwen3-6-plus | Alibaba Cloud | 78.8 | percent |
| Gemini 3 Flash | #18 | gemini-3-flash | 78.6 | percent | |
| GPT-5.6 Luna | #19 | gpt-5-6-luna | OpenAI | 78.6 | percent |
| MiniMax M3 | #20 | minimax-m3 | MiniMax | 78.6 | percent |
| GPT-5.3-Codex | #21 | gpt-5-3-codex | OpenAI | 78.5 | percent |
| Kimi K2.5 | #22 | kimi-k2-5 | Moonshot AI | 78.5 | percent |
| Grok 4.3 | #23 | grok-4-3 | xAI | 78.1 | percent |
| Claude Opus 4.6 | #24 | claude-opus-4-6 | Anthropic | 77.3 | percent |
| Claude Sonnet 5 | #25 | claude-sonnet-5 | Anthropic | 77.3 | percent |
| Qwen3.5 397B A17B | #26 | qwen3-5-397b | Alibaba Cloud | 77.3 | percent |
| Gemma 4 31B | #27 | gemma-4-31b | 76.9 | percent | |
| GPT-5.2 Codex | #28 | gpt-5-2-codex | OpenAI | 76.3006 | percent |
| Gemini 3.1 Flash-Lite | #29 | gemini-3-1-flash-lite | 75.5 | percent | |
| GPT-5.1 | #30 | gpt-5-1 | OpenAI | 75.5 | percent |
| MiMo-V2.5 | #31 | mimo-v2-5 | Xiaomi | 75.4335 | percent |
| Step 3.7 Flash | #32 | step-3-7-flash | StepFun | 75.3 | percent |
| Qwen3.5 27B | #33 | qwen3-5-27b | Alibaba Cloud | 75.0289 | percent |
| Qwen3.5-122B-A10B | #34 | qwen3-5-122b-a10b | Alibaba Cloud | 75 | percent |
| Qwen3.6-35B-A3B | #35 | qwen3-6-35b-a3b | Alibaba Cloud | 75 | percent |
| Qwen3.5 122B | #36 | qwen3-5-122b | Alibaba Cloud | 74.9711 | percent |
| Gemini 2.5 Pro | #37 | gemini-2-5-pro | 74.9133 | percent | |
| Qwen3.6 27B | #38 | qwen3-6-27b | Alibaba Cloud | 74.6243 | percent |
| Grok 4.20 | #39 | grok-4-20 | xAI | 74.5665 | percent |
| GPT-5 | #40 | gpt-5 | OpenAI | 74.3 | percent |
| Claude Opus 4.5 Thinking | #41 | claude-opus-4-5-thinking | Anthropic | 74 | percent |
| Muse Glimmer 30B | #42 | muse-glimmer-30b | Meta | 74 | percent |
| GPT-5 Codex | #43 | gpt-5-codex | OpenAI | 73.815 | percent |
| Inkling | #44 | inkling | Thinking Machines Lab | 73.5 | percent |
| GPT-5.4 mini | #45 | gpt-5-4-mini | OpenAI | 73.3 | percent |
| Gemini 2.5 Flash | #46 | gemini-2-5-flash | 73.1214 | percent | |
| GLM-5V-Turbo | #47 | glm-5v-turbo | Z.AI | 72.8 | percent |
| Qwen3.5-35B-A3B | #48 | qwen3-5-35b-a3b | Alibaba Cloud | 72.7 | percent |
| Qwen3.5 35B | #49 | qwen3-5-35b | Alibaba Cloud | 72.659 | percent |
| GPT-5.1-Codex | #50 | gpt-5-1-codex | OpenAI | 72.5 | percent |
| GPT-5.1-Codex-Max | #51 | gpt-5-1-codex-max | OpenAI | 72.5 | percent |
| Claude Opus 4.5 | #52 | claude-opus-4-5 | Anthropic | 71.2 | percent |
| Claude Sonnet 4.6 | #53 | claude-sonnet-4-6 | Anthropic | 70.6 | percent |
| Qwen3.5 Omni Plus | #54 | qwen3-5-omni-plus | Alibaba Cloud | 70.5202 | percent |
| o3 | #55 | o3 | OpenAI | 70.1 | percent |
| MiMo-V2-Omni | #56 | mimo-v2-omni | Xiaomi | 69.9 | percent |
| Gemma 4 12B Unified | #57 | gemma-4-12b | 69.7 | percent | |
| Gemma 4 26B | #58 | gemma-4-26b | 69.2486 | percent | |
| o4-mini | #59 | o4-mini | OpenAI | 69.2486 | percent |
| Gemma 4 26B A4B | #60 | gemma-4-26b-a4b | 69.2 | percent | |
| GPT-5 mini | #61 | gpt-5-mini | OpenAI | 68.8439 | percent |
| Grok 4 | #62 | grok-4 | xAI | 68.8439 | percent |
| Claude Sonnet 4.5 | #63 | claude-sonnet-4-5 | Anthropic | 68.7283 | percent |
| Claude Opus 4.1 | #64 | claude-opus-4-1 | Anthropic | 67.9191 | percent |
| Claude 4.1 Opus Thinking | #65 | claude-4-1-opus-thinking | Anthropic | 67.9 | percent |
| GPT-5.4 nano | #66 | gpt-5-4-nano | OpenAI | 66.1 | percent |
| Mistral Medium 3.5 128B | #67 | mistral-medium-3-5-128b | Mistral AI | 64.9 | percent |
| Mistral Medium 3.5 | #68 | mistral-medium-3-5 | Mistral AI | 64.86 | percent |
| Nova 2 Pro | #69 | nova-2-pro | Amazon | 64.5087 | percent |
| Grok 4.1 Fast (Reasoning) | #70 | grok-4-1-fast-reasoning | xAI | 63.3 | percent |
| Command A+ | #71 | command-a-plus | Cohere | 63.237 | percent |
| Claude 4 Sonnet | #72 | claude-4-sonnet | Anthropic | 62.4 | percent |
| Llama 4 Maverick | #73 | llama-4-maverick | Meta | 62.1 | percent |
| Grok 4 Fast (Reasoning) | #74 | grok-4-fast-reasoning | xAI | 61.8 | percent |
| Claude Sonnet 4 | #75 | claude-sonnet-4 | Anthropic | 61.7919 | percent |
| Grok 4 Fast | #76 | grok-4-fast | xAI | 61.7919 | percent |
| GPT-4.1 | #77 | gpt-4-1 | OpenAI | 61.2 | percent |
| GPT-4.1 mini | #78 | gpt-4-1-mini | OpenAI | 58.7 | percent |
| Mistral Small 4 | #79 | mistral-small-4 | Mistral AI | 56.82 | percent |
| Mistral Large 3 | #80 | mistral-large-3 | Mistral AI | 55.7 | percent |
| Claude Haiku 4.5 | #81 | claude-haiku-4-5 | Anthropic | 55.14 | percent |
| Gemini 1.5 Pro | #82 | gemini-1-5-pro | 55 | percent | |
| Nemotron 3 Nano Omni 30B A3B | #83 | nemotron-3-nano-omni-30b-a3b | NVIDIA | 53.2 | percent |
| Mistral Medium 3 | #84 | mistral-medium-3 | Mistral AI | 53 | percent |
| Llama 4 Scout | #85 | llama-4-scout | Meta | 52.9 | percent |
| GPT-4o (May '24) | #86 | gpt-4o-2024-05-13 | OpenAI | 51.9 | percent |
| Gemma 4 E4B | #87 | gemma-4-e4b | 51.4 | percent | |
| Grok 4.1 Fast | #88 | grok-4-1-fast | xAI | 48.4 | percent |
| Gemma 3 27B | #89 | gemma-3-27b | 48 | percent | |
| Gemma 4 E2B | #90 | gemma-4-e2b | 44.6 | percent | |
| Nova Pro | #91 | nova-pro | Amazon | 44.3 | percent |
| Gemini 1.5 Pro (May '24) | #92 | gemini-1-5-pro-may-2024 | 43.5 | percent | |
| GPT-4o mini | #93 | gpt-4o-mini | OpenAI | 41.5 | percent |
| GPT-4.1 nano | #94 | gpt-4-1-nano | OpenAI | 40.1 | percent |
| Claude 3 Haiku | #95 | claude-3-haiku | Anthropic | 30.8 | percent |
| LFM2.5-VL-1.6B-Extract | #96 | lfm2-5-vl-1-6b-extract | LiquidAI | 26.5 | percent |
One best compatible score per canonical product · higher is better
| Rank | Model | Weights | Evidence | Score |
|---|---|---|---|---|
| 01 | Claude Opus 5Anthropic | Closed weights | Source carrier | 84.7% |
| 02 | Gemini 3.5 FlashGoogle | Closed weights | Source carrier | 84.3% |
| 03 | Gemini 3.1 Pro PreviewGoogle | Closed weights | Source carrier | 83.9% |
| 04 | GPT-5.6 SolOpenAI | Closed weights | Direct source result | 83.4% |
| 05 | Gemini 3.6 FlashGoogle | Closed weights | Direct source result | 83.2% |
| 06 | Qwen3.8 MaxAlibaba Cloud | Open weights | Direct source result | 82.3% |
| 07 | Kimi K3Moonshot AI | Open weights | Direct source result | 81.6% |
| 08 | GPT-5.4OpenAI | Closed weights | Source carrier | 81.2% |
| 09 | GPT-5.5OpenAI | Closed weights | Public reference | 81.2% |
| 10 | GPT-5.6 TerraOpenAI | Closed weights | Source carrier | 80.7% |
Each distinct published measurement is retained. Same-snapshot canonical and configuration projections appear once.
| Published model / configuration | Score | Evidence & protocol | Source & dates |
|---|---|---|---|
Muse Glimmer-30B; high reasoning; temperature=1.0; top_p=0.95; top_k=64Muse Glimmer 30BExact identitymuse-glimmer-30b-high Canonical product: muse-glimmer-30b | 74% | Public referenceprovider-reportedVersion & system1,730-question set Artificial Analysis MMMU-Pro documented evaluation | Muse Glimmer Evaluation MethodologyObserved Checked |
Qwen3.8 Max (xhigh, default reasoning effort)Qwen3.8 MaxExact identityqwen-3-8-max-xhigh Canonical product: qwen-3-8-max | 82.3% | Public referenceprovider-reportedVersion & systemCurrent Qwen provider evaluation | Qwen3.8 Max official releaseObserved Checked |
Exact BenchLM registry variant Claude Opus 5; bulk export does not retain a complete upstream harness configuration.Claude Opus 5Exact identitySource label without a registered configuration ID Canonical product: claude-opus-5 | 84.7% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Gemini 3.5 Flash; bulk export does not retain a complete upstream harness configuration.Gemini 3.5 FlashExact identitySource label without a registered configuration ID Canonical product: gemini-3-5-flash | 84.3% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant GPT-5.6 Sol; bulk export does not retain a complete upstream harness configuration.GPT-5.6 SolExact identitySource label without a registered configuration ID Canonical product: gpt-5-6-sol | 83.4% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Gemini 3.6 Flash; bulk export does not retain a complete upstream harness configuration.Gemini 3.6 FlashExact identitySource label without a registered configuration ID Canonical product: gemini-3-6-flash | 83.2% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant GPT-5.6 Terra; bulk export does not retain a complete upstream harness configuration.GPT-5.6 TerraExact identitySource label without a registered configuration ID Canonical product: gpt-5-6-terra | 80.7% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Kimi K3; bulk export does not retain a complete upstream harness configuration.Kimi K3Exact identitySource label without a registered configuration ID Canonical product: kimi-k3 | 80.5% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Muse Spark; bulk export does not retain a complete upstream harness configuration.Muse SparkExact identitySource label without a registered configuration ID Canonical product: muse-spark | 80.5% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Qwen3.7 Plus; bulk export does not retain a complete upstream harness configuration.Qwen3.7-PlusExact identitySource label without a registered configuration ID Canonical product: qwen-3-7-plus | 80.5% | Source carriersource-checkedVersion & system2026 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
| Model | Result | Execution & attribution | Original source |
|---|---|---|---|
| Qwen3.8 Max (0902)mmmuPro · exact AA profile b112af07-3bd5-4647-b09f-b23204361bb2 | 82.775percent | AA current published profile; effort label not exportedindependent evaluatorRetained as additional evidence. Existing Core point is unchanged pending an accepted complete own-evidence amendment; incompatible versions remain separate. | Artificial Analysis ↗Publication date not recorded · checked 2026-09-16 |
From result to context
A more robust extension of MMMU intended to reduce shortcut-based answering.
The summary shows one best compatible source result per canonical product. Versions, effort settings and execution systems remain attached to the underlying records.
Scores from different versions or harnesses may not be interchangeable. The published source rows preserve those distinctions and their original units. A source result is not automatically an input to a current index.
This catalogue entry preserves available source evidence. Current indices admit only their specifically reviewed tracks and configurations.
Contamination risk: Unknown. Lifecycle: Active.