Instruction Following / Benchmark profile
IFBench evaluates precise instruction-following generalization on 58 challenging, verifiable out-of-domain constraints. Unlike IFEval which tests familiar constraint types, IFBench specifically measures how well models follow novel instructions they haven't been optimized for, exposing overfitting to common instruction patterns.
Observed results
17 ranked models · higher is better · labels show rank and score
| Model | Rank | Model ID | Provider | Score | Unit |
|---|---|---|---|---|---|
| MAI-Thinking-1 | #1 | mai-thinking-1 | Microsoft | 85 | percent |
| Inkling-Small | #2 | inkling-small | Thinking Machines Lab | 82.2 | percent |
| Nemotron 3 Ultra | #3 | nemotron-3-ultra | NVIDIA | 81.7 | percent |
| Grok 4.3 | #4 | grok-4-3 | xAI | 81.3 | percent |
| Inkling | #5 | inkling | Thinking Machines Lab | 79.8 | percent |
| Qwen3.7-Max | #6 | qwen-3-7-max | Alibaba Cloud | 79.1 | percent |
| Qwen3.7-Plus | #7 | qwen-3-7-plus | Alibaba Cloud | 79.1 | percent |
| Gemini 3.5 Flash | #8 | gemini-3-5-flash | 76.3 | percent | |
| Qwen3.6 Plus | #9 | qwen3-6-plus | Alibaba Cloud | 75.8 | percent |
| Nemotron 3 Nano Omni 30B A3B | #10 | nemotron-3-nano-omni-30b-a3b | NVIDIA | 74.2 | percent |
| Hy3 Preview | #11 | hy3-preview | Tencent | 63.1 | percent |
| Claude Opus 4.5 | #12 | claude-opus-4-5 | Anthropic | 58 | percent |
| Ling 2.6 Flash | #13 | ling-2-6-flash | InclusionAI | 57 | percent |
| LFM2.5-8B-A1B | #14 | lfm2-5-8b-a1b | LiquidAI | 56.47 | percent |
| ZAYA1-8B | #15 | zaya1-8b | Zyphra | 52.56 | percent |
| MiniCPM5-1B | #16 | minicpm5-1b | OpenBMB | 46.67 | percent |
| LFM2.5-230M | #17 | lfm2-5-230m | LiquidAI | 38.4 | percent |
One best compatible score per canonical product · higher is better
| Rank | Model | Weights | Evidence | Score |
|---|---|---|---|---|
| 01 | MAI-Thinking-1Microsoft | Not documented | Source carrier | 85% |
| 02 | Inkling-SmallThinking Machines Lab | Open weights | Source carrier | 82.2% |
| 03 | Nemotron 3 UltraNVIDIA | Open weights | Source carrier | 81.7% |
| 04 | Grok 4.3xAI | Closed weights | Source carrier | 81.3% |
| 05 | InklingThinking Machines Lab | Open weights | Source carrier | 79.8% |
| 06 | Qwen3.7-MaxAlibaba Cloud | Closed weights | Source carrier | 79.1% |
| 07 | Qwen3.7-PlusAlibaba Cloud | Closed weights | Source carrier | 79.1% |
| 08 | Gemini 3.5 FlashGoogle | Closed weights | Source carrier | 76.3% |
| 09 | Qwen3.6 PlusAlibaba Cloud | Closed weights | Source carrier | 75.8% |
| 10 | Open weights | Source carrier | 74.2% |
Each distinct published measurement is retained. Same-snapshot canonical and configuration projections appear once.
| Published model / configuration | Score | Evidence & protocol | Source & dates |
|---|---|---|---|
Exact BenchLM registry variant MAI-Thinking-1; bulk export does not retain a complete upstream harness configuration.MAI-Thinking-1Exact identitySource label without a registered configuration ID Canonical product: mai-thinking-1 | 85% | Source carriersource-checkedVersion & system2025 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Inkling-Small; bulk export does not retain a complete upstream harness configuration.Inkling-SmallExact identitySource label without a registered configuration ID Canonical product: inkling-small | 82.2% | Source carriersource-checkedVersion & system2025 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Nemotron 3 Ultra; bulk export does not retain a complete upstream harness configuration.Nemotron 3 UltraExact identitySource label without a registered configuration ID Canonical product: nemotron-3-ultra | 81.7% | Source carriersource-checkedVersion & system2025 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Grok 4.3; bulk export does not retain a complete upstream harness configuration.Grok 4.3Exact identitySource label without a registered configuration ID Canonical product: grok-4-3 | 81.3% | Source carriersource-checkedVersion & system2025 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Inkling; bulk export does not retain a complete upstream harness configuration.InklingExact identitySource label without a registered configuration ID Canonical product: inkling | 79.8% | Source carriersource-checkedVersion & system2025 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Qwen3.7 Max; bulk export does not retain a complete upstream harness configuration.Qwen3.7-MaxExact identitySource label without a registered configuration ID Canonical product: qwen-3-7-max | 79.1% | Source carriersource-checkedVersion & system2025 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Qwen3.7 Plus; bulk export does not retain a complete upstream harness configuration.Qwen3.7-PlusExact identitySource label without a registered configuration ID Canonical product: qwen-3-7-plus | 79.1% | Source carriersource-checkedVersion & system2025 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Gemini 3.5 Flash; bulk export does not retain a complete upstream harness configuration.Gemini 3.5 FlashExact identitySource label without a registered configuration ID Canonical product: gemini-3-5-flash | 76.3% | Source carriersource-checkedVersion & system2025 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Qwen3.6 Plus; bulk export does not retain a complete upstream harness configuration.Qwen3.6 PlusExact identitySource label without a registered configuration ID Canonical product: qwen3-6-plus | 75.8% | Source carriersource-checkedVersion & system2025 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
Exact BenchLM registry variant Nemotron 3 Nano Omni 30B A3B; bulk export does not retain a complete upstream harness configuration.Nemotron 3 Nano Omni 30B A3BExact identitySource label without a registered configuration ID Canonical product: nemotron-3-nano-omni-30b-a3b | 74.2% | Source carriersource-checkedVersion & system2025 Source-native system | BenchLM public datasets — 2026-08-01Observed Checked |
From result to context
IFBench evaluates precise instruction-following generalization on 58 challenging, verifiable out-of-domain constraints. Unlike IFEval which tests familiar constraint types, IFBench specifically measures how well models follow novel instructions they haven't been optimized for, exposing overfitting to common instruction patterns.
The summary shows one best compatible source result per canonical product. Versions, effort settings and execution systems remain attached to the underlying records.
Scores from different versions or harnesses may not be interchangeable. The published source rows preserve those distinctions and their original units. A source result is not automatically an input to a current index.
This catalogue entry preserves available source evidence. Current indices admit only their specifically reviewed tracks and configurations.
Contamination risk: Unknown. Lifecycle: Active.
