Knowledge / Benchmark profile
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Designed to be difficult even for skilled non-experts with access to Google.
Observed results
20 ranked models · higher is better · labels show rank and score
| Model | Rank | Model ID | Provider | Score | Unit |
|---|---|---|---|---|---|
| Sakana Fugu | #1 | sakana-fugu | Sakana AI | 95.5 | percent |
| Sakana Fugu-Ultra | #2 | sakana-fugu-ultra | Sakana AI | 95.5 | percent |
| GPT-5.6 Sol | #3 | gpt-5-6-sol | OpenAI | 94.6 | percent |
| Claude Opus 4.7 | #4 | claude-opus-4-7 | Anthropic | 94.2 | percent |
| Claude Mythos 5 | #5 | claude-mythos-5 | Anthropic | 94.1 | percent |
| Claude Opus 4.8 | #6 | claude-opus-4-8 | Anthropic | 93.6 | percent |
| GPT-5.5 | #7 | gpt-5-5 | OpenAI | 93.6 | percent |
| Kimi K3 | #8 | kimi-k3 | Moonshot AI | 93.5 | percent |
| GPT-5.6 Terra | #9 | gpt-5-6-terra | OpenAI | 92.9 | percent |
| GPT-5.4 | #10 | gpt-5-4 | OpenAI | 92.8 | percent |
| GPT-5.2 | #11 | gpt-5-2 | OpenAI | 92.4 | percent |
| Qwen3.7-Max | #12 | qwen-3-7-max | Alibaba Cloud | 92.4 | percent |
| GPT-5.6 Luna | #13 | gpt-5-6-luna | OpenAI | 92.3 | percent |
| Gemini 3.5 Flash | #14 | gemini-3-5-flash | 92.2 | percent | |
| Claude Opus 4.6 | #15 | claude-opus-4-6 | Anthropic | 91.3 | percent |
| GLM-5.2 | #16 | glm-5-2 | Z.AI | 91.2 | percent |
| Kimi K2.6 | #17 | kimi-k2-6 | Moonshot AI | 90.5 | percent |
| Qwen3.6 Plus | #18 | qwen3-6-plus | Alibaba Cloud | 90.4 | percent |
| Qwen3.7-Plus | #19 | qwen-3-7-plus | Alibaba Cloud | 90.3 | percent |
| DeepSeek V4 Pro | #20 | deepseek-v4-pro | DeepSeek | 90.1 | percent |
| Grok 4.3 | #21 | grok-4-3 | xAI | 90.1 | percent |
| Claude Sonnet 4.6 | #22 | claude-sonnet-4-6 | Anthropic | 89.9 | percent |
| Interfaze Beta | #23 | interfaze-beta | Interfaze | 89.9 | percent |
| Inkling-Small | #24 | inkling-small | Thinking Machines Lab | 89.5 | percent |
| Qwen3.5 397B A17B | #25 | qwen3-5-397b | Alibaba Cloud | 88.4 | percent |
| DeepSeek V4 Flash | #26 | deepseek-v4-flash | DeepSeek | 88.1 | percent |
| GPT-5.4 mini | #27 | gpt-5-4-mini | OpenAI | 88 | percent |
| Inkling | #28 | inkling | Thinking Machines Lab | 87.9 | percent |
| Qwen3.6 27B | #29 | qwen3-6-27b | Alibaba Cloud | 87.8 | percent |
| Kimi K2.5 | #30 | kimi-k2-5 | Moonshot AI | 87.6 | percent |
| Hy3 Preview | #31 | hy3-preview | Tencent | 87.2 | percent |
| Claude Opus 4.5 | #32 | claude-opus-4-5 | Anthropic | 87 | percent |
| Nemotron 3 Ultra | #33 | nemotron-3-ultra | NVIDIA | 87 | percent |
| Qwen3.5-122B-A10B | #34 | qwen3-5-122b-a10b | Alibaba Cloud | 86.6 | percent |
| GLM-5 | #35 | glm-5 | Z.AI | 86 | percent |
| Qwen3.6-35B-A3B | #36 | qwen3-6-35b-a3b | Alibaba Cloud | 86 | percent |
| GLM-4.7 | #37 | glm-4-7 | Z.AI | 85.7 | percent |
| Qwen3.5 27B | #38 | qwen3-5-27b | Alibaba Cloud | 85.5 | percent |
| Gemma 4 31B | #39 | gemma-4-31b | 84.3 | percent | |
| MAI-Thinking-1 | #40 | mai-thinking-1 | Microsoft | 84.2 | percent |
| Qwen3.5-35B-A3B | #41 | qwen3-5-35b-a3b | Alibaba Cloud | 84.2 | percent |
| Claude Sonnet 4.5 | #42 | claude-sonnet-4-5 | Anthropic | 83.4 | percent |
| Gemini 2.5 Pro | #43 | gemini-2-5-pro | 83 | percent | |
| GPT-5.4 nano | #44 | gpt-5-4-nano | OpenAI | 82.8 | percent |
| o1-pro | #45 | o1-pro | OpenAI | 79 | percent |
| Gemma 4 12B Unified | #46 | gemma-4-12b | 78.8 | percent | |
| Qwen3 235B 2507 | #47 | qwen3-235b-2507 | Alibaba Cloud | 77.5 | percent |
| o3-mini | #48 | o3-mini | OpenAI | 77.2 | percent |
| o1 | #49 | o1 | OpenAI | 75.7 | percent |
| Nemotron 3 Nano Omni 30B A3B | #50 | nemotron-3-nano-omni-30b-a3b | NVIDIA | 72.2 | percent |
| ZAYA1-8B | #51 | zaya1-8b | Zyphra | 71 | percent |
| GPT-4.1 | #52 | gpt-4-1 | OpenAI | 66.3 | percent |
| GPT-4.1 mini | #53 | gpt-4-1-mini | OpenAI | 64.2 | percent |
| Claude 3.5 Sonnet | #54 | claude-3-5-sonnet | Anthropic | 59.4 | percent |
| DeepSeek V3 | #55 | deepseek-v3 | DeepSeek | 59.1 | percent |
| Ling 2.6 Flash | #56 | ling-2-6-flash | InclusionAI | 59 | percent |
| Gemma 4 E4B | #57 | gemma-4-e4b | 58.6 | percent | |
| Mellum2-12B-A2.5B-Thinking | #58 | mellum2-12b-a2-5b-thinking | JetBrains | 57.6 | percent |
| ZAYA1-74B-Preview | #59 | zaya1-74b-preview | Zyphra | 57.3 | percent |
| GPT-4.1 nano | #60 | gpt-4-1-nano | OpenAI | 50.3 | percent |
| Gemma 4 E2B | #61 | gemma-4-e2b | 43.4 | percent | |
| Soofi S 30B-A3B | #62 | soofi-s-30b-a3b | Soofi Project | 43.4 | percent |
| Mellum2-12B-A2.5B-Instruct | #63 | mellum2-12b-a2-5b-instruct | JetBrains | 40.9 | percent |
| LFM2.5-VL-450M | #64 | lfm2-5-vl-450m | LiquidAI | 25.66 | percent |
| LFM2.5-230M | #65 | lfm2-5-230m | LiquidAI | 25.41 | percent |
One best compatible score per canonical product · higher is better
| Rank | Model | Weights | Evidence | Score |
|---|---|---|---|---|
| 01 | Sakana FuguSakana AI | Closed weights | Source carrier | 95.5% |
| 02 | Sakana Fugu-UltraSakana AI | Closed weights | Source carrier | 95.5% |
| 03 | GPT-5.6 SolOpenAI | Closed weights | Source carrier | 94.6% |
| 04 | Claude Opus 4.7Anthropic | Closed weights | Source carrier | 94.2% |
| 05 | Claude Mythos 5Anthropic | Closed weights | Source carrier | 94.1% |
| 06 | Claude Opus 4.8Anthropic | Closed weights | Source carrier | 93.6% |
| 07 | GPT-5.5OpenAI | Closed weights | Source carrier | 93.6% |
| 08 | Kimi K3Moonshot AI | Open weights | Source carrier | 93.5% |
| 09 | GPT-5.6 TerraOpenAI | Closed weights | Source carrier | 92.9% |
| 10 | GPT-5.4OpenAI | Closed weights | Source carrier | 92.8% |
Each distinct published measurement is retained. Same-snapshot canonical and configuration projections appear once.
| Published model / configuration | Score | Evidence & protocol | Source & dates |
|---|---|---|---|
Exact BenchLM registry variant Sakana Fugu; bulk export does not retain a complete upstream harness configuration.Sakana FuguExact identitySource label without a registered configuration ID Canonical product: sakana-fugu | 95.5% | Source carriersource-checkedVersion & system2023 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Sakana Fugu-Ultra; bulk export does not retain a complete upstream harness configuration.Sakana Fugu-UltraExact identitySource label without a registered configuration ID Canonical product: sakana-fugu-ultra | 95.5% | Source carriersource-checkedVersion & system2023 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant GPT-5.6 Sol; bulk export does not retain a complete upstream harness configuration.GPT-5.6 SolExact identitySource label without a registered configuration ID Canonical product: gpt-5-6-sol | 94.6% | Source carriersource-checkedVersion & system2023 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Claude Opus 4.7 (Adaptive); bulk export does not retain a complete upstream harness configuration.Claude Opus 4.7Exact identityclaude-opus-4-7-max Canonical product: claude-opus-4-7 | 94.2% | Source carriersource-checkedVersion & system2023 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Claude Mythos 5; bulk export does not retain a complete upstream harness configuration.Claude Mythos 5Exact identitySource label without a registered configuration ID Canonical product: claude-mythos-5 | 94.1% | Source carriersource-checkedVersion & system2023 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Claude Opus 4.8; bulk export does not retain a complete upstream harness configuration.Claude Opus 4.8Exact identitySource label without a registered configuration ID Canonical product: claude-opus-4-8 | 93.6% | Source carriersource-checkedVersion & system2023 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant GPT-5.5; bulk export does not retain a complete upstream harness configuration.GPT-5.5Exact identitySource label without a registered configuration ID Canonical product: gpt-5-5 | 93.6% | Source carriersource-checkedVersion & system2023 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Kimi K3; bulk export does not retain a complete upstream harness configuration.Kimi K3Exact identitySource label without a registered configuration ID Canonical product: kimi-k3 | 93.5% | Source carriersource-checkedVersion & system2023 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant GPT-5.6 Terra; bulk export does not retain a complete upstream harness configuration.GPT-5.6 TerraExact identitySource label without a registered configuration ID Canonical product: gpt-5-6-terra | 92.9% | Source carriersource-checkedVersion & system2023 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant GPT-5.4; bulk export does not retain a complete upstream harness configuration.GPT-5.4Exact identitySource label without a registered configuration ID Canonical product: gpt-5-4 | 92.8% | Source carriersource-checkedVersion & system2023 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
From result to context
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Designed to be difficult even for skilled non-experts with access to Google.
The summary shows one best compatible source result per canonical product. Versions, effort settings and execution systems remain attached to the underlying records.
Scores from different versions or harnesses may not be interchangeable. The published source rows preserve those distinctions and their original units. A source result is not automatically an input to a current index.
This catalogue entry preserves available source evidence. Current indices admit only their specifically reviewed tracks and configurations.
Contamination risk: Unknown. Lifecycle: Active.
