Agents / Benchmark profile
Fintech customer-support agent benchmark requiring knowledge-base navigation and multi-step tool use on banking workflows.
Observed results
20 ranked models · higher is better · labels show rank and score
| Model | Rank | Model ID | Provider | Score | Unit |
|---|---|---|---|---|---|
| Grok 4.6 | #1 | grok-4-6 | xAI | 50.7216494845361 | percent |
| GLM-5.3 | #2 | glm-5-3 | Z.AI | 50.3092783505155 | percent |
| GLM-5.3-Flash | #3 | glm-5-3-flash | Z.AI | 47.2164948453608 | percent |
| Kimi K3 | #4 | kimi-k3 | Moonshot AI | 45.979381443299 | percent |
| Qwen3.8-Flash-Next | #5 | qwen-3-8-flash-next | Alibaba Cloud | 45.360824742268 | percent |
| GPT-5.6 Sol | #6 | gpt-5-6-sol | OpenAI | 44.3298969072165 | percent |
| Claude Opus 5 | #7 | claude-opus-5 | Anthropic | 42.0618556701031 | percent |
| Grok 4.5 | #8 | grok-4-5 | xAI | 42.0618556701031 | percent |
| DeepSeek V4 Flash Vision Exp | #9 | deepseek-v4-flash-vision-exp | DeepSeek | 41.0309278350515 | percent |
| GPT-5.6 Terra | #10 | gpt-5-6-terra | OpenAI | 40.2061855670103 | percent |
| DeepSeek V4 Pro 0813 | #11 | deepseek-v4-pro-0813 | DeepSeek | 39.58762886597941 | percent |
| GPT-5.4 | #12 | gpt-5-4 | OpenAI | 39.58762886597941 | percent |
| GPT-5.5 | #13 | gpt-5-5 | OpenAI | 38.9690721649485 | percent |
| Claude Fable 5 | #14 | claude-fable-5 | Anthropic | 38.144329896907195 | percent |
| Claude Sonnet 5 | #15 | claude-sonnet-5 | Anthropic | 37.319587628866 | percent |
| Gemini 3.7 Flash | #16 | gemini-3-7-flash | 35.4639175257732 | percent | |
| Muse Spark 1.2 | #17 | muse-spark-1-2 | Meta | 34.8453608247423 | percent |
| Claude Opus 4.7 | #18 | claude-opus-4-7 | Anthropic | 34.639175257732 | percent |
| GLM-5.2 | #19 | glm-5-2 | Z.AI | 34.639175257732 | percent |
| Claude Opus 4.8 | #20 | claude-opus-4-8 | Anthropic | 34.226804123711304 | percent |
| Muse Spark 1.1 | #21 | muse-spark-1-1 | Meta | 31.7525773195876 | percent |
| GPT-5.6 Luna | #22 | gpt-5-6-luna | OpenAI | 31.1340206185567 | percent |
| Claude Sonnet 4.6 | #23 | claude-sonnet-4-6 | Anthropic | 30.5155 | percent |
| Gemini 3.6 Flash | #24 | gemini-3-6-flash | 29.8969072164948 | percent | |
| DeepSeek V4 Pro | #25 | deepseek-v4-pro | DeepSeek | 25.7732 | percent |
| Gemini 3.5 Flash | #26 | gemini-3-5-flash | 25.3608 | percent | |
| Muse Glimmer 30B | #27 | muse-glimmer-30b | Meta | 23.5 | percent |
| DeepSeek V4 Flash | #28 | deepseek-v4-flash | DeepSeek | 22.8866 | percent |
| GPT-5.4 mini | #29 | gpt-5-4-mini | OpenAI | 21.4433 | percent |
| GPT-5.4 nano | #30 | gpt-5-4-nano | OpenAI | 21.0309 | percent |
| Kimi K2.6 | #31 | kimi-k2-6 | Moonshot AI | 20.6186 | percent |
| Solar Open 2 250B | #32 | solar-open2-250b | Upstage | 19.6 | percent |
| GPT-5 | #33 | gpt-5 | OpenAI | 19.5876 | percent |
| Muse Spark | #34 | muse-spark | Meta | 19.5876 | percent |
| Claude Sonnet 4.5 | #35 | claude-sonnet-4-5 | Anthropic | 18.9691 | percent |
| DeepSeek V3.2 | #36 | deepseek-v3-2 | DeepSeek | 18.7629 | percent |
| Kimi K2.7 Code | #37 | kimi-k2-7-code | Moonshot AI | 18.1443 | percent |
| Qwen3.7-Plus | #38 | qwen-3-7-plus | Alibaba Cloud | 17.8694 | percent |
| Gemini 3 Flash | #39 | gemini-3-flash | 17.5258 | percent | |
| Gemini 3.5 Flash-Lite | #40 | gemini-3-5-flash-lite | 16.5 | percent | |
| Gemini 3.1 Pro Preview | #41 | gemini-3-1-pro-preview | 16.4948 | percent | |
| Qwen3.6 Plus | #42 | qwen3-6-plus | Alibaba Cloud | 16.4948 | percent |
| DeepSeek V3.1 Terminus | #43 | deepseek-v3-1-terminus | DeepSeek | 15.8763 | percent |
| Qwen3.5-122B-A10B | #44 | qwen3-5-122b-a10b | Alibaba Cloud | 15.257731958763 | percent |
| Qwen3.6 27B | #45 | qwen3-6-27b | Alibaba Cloud | 15.2577 | percent |
| Mistral Medium 3.5 128B | #46 | mistral-medium-3-5-128b | Mistral AI | 15.051546391753 | percent |
| Gemma 4 31B | #47 | gemma-4-31b | 15.0515 | percent | |
| Mistral Medium 3.5 | #48 | mistral-medium-3-5 | Mistral AI | 14.433 | percent |
| Kimi K2.5 | #49 | kimi-k2-5 | Moonshot AI | 14.226804123711 | percent |
| GPT-5.1 | #50 | gpt-5-1 | OpenAI | 14.0206 | percent |
| Claude Sonnet 4 | #51 | claude-sonnet-4 | Anthropic | 13.8144 | percent |
| Nemotron 3 Ultra | #52 | nemotron-3-ultra | NVIDIA | 13.8144 | percent |
| Qwen3.5 122B | #53 | qwen3-5-122b | Alibaba Cloud | 13.6082 | percent |
| Qwen3.5 397B A17B | #54 | qwen3-5-397b | Alibaba Cloud | 13.4021 | percent |
| LongCat-2.0 | #55 | longcat-2-0 | Meituan | 13.19587628866 | percent |
| MiniMax M3 | #56 | minimax-m3 | MiniMax | 12.9897 | percent |
| Grok 4.3 | #57 | grok-4-3 | xAI | 12.1649 | percent |
| Gemma 4 26B A4B | #58 | gemma-4-26b-a4b | 11.958762886598 | percent | |
| Gemma 4 26B | #59 | gemma-4-26b | 11.7526 | percent | |
| GLM-5.1 | #60 | glm-5-1 | Z.AI | 11.5464 | percent |
| Qwen3.7-Max | #61 | qwen-3-7-max | Alibaba Cloud | 10.9278 | percent |
| GLM-4.6 | #62 | glm-4-6 | Z.AI | 10.5155 | percent |
| Nemotron 3 Super | #63 | nemotron-3-super | NVIDIA | 10.1031 | percent |
| GLM-4.7 | #64 | glm-4-7 | Z.AI | 9.6907 | percent |
| NVIDIA Nemotron 3.5 Lightning 30B-A3B | #65 | nemotron-3-5-lightning-30b-a3b | NVIDIA | 9.28 | percent |
| Gemini 2.5 Pro | #66 | gemini-2-5-pro | 9.2784 | percent | |
| Qwen3.6-35B-A3B | #67 | qwen3-6-35b-a3b | Alibaba Cloud | 9.278350515464 | percent |
| Claude Haiku 4.5 | #68 | claude-haiku-4-5 | Anthropic | 9.0722 | percent |
| MiniMax M2.7 | #69 | minimax-m2-7 | MiniMax | 8.866 | percent |
| Gemini 3.1 Flash-Lite | #70 | gemini-3-1-flash-lite | 8.6598 | percent | |
| MiMo-V2.5-Pro | #71 | mimo-v2-5-pro | Xiaomi | 8.6598 | percent |
| Nova 2 Pro | #72 | nova-2-pro | Amazon | 7.6289 | percent |
| MiMo-V2.5 | #73 | mimo-v2-5 | Xiaomi | 6.5979 | percent |
| Nemotron 3 Nano 30B | #74 | nemotron-3-nano-30b | NVIDIA | 5.979381443299 | percent |
| Command A+ | #75 | command-a-plus | Cohere | 5.7732 | percent |
| Mistral Large 3 | #76 | mistral-large-3 | Mistral AI | 5.7732 | percent |
| Trinity-Large-Thinking | #77 | trinity-large-thinking | Arcee AI | 5.773195876289 | percent |
| GPT-4.1 mini | #78 | gpt-4-1-mini | OpenAI | 5.360824742268 | percent |
| Mistral Small 4 | #79 | mistral-small-4 | Mistral AI | 5.1546 | percent |
| Llama 4 Maverick | #80 | llama-4-maverick | Meta | 3.711340206186 | percent |
| GPT-4.1 nano | #81 | gpt-4-1-nano | OpenAI | 3.505154639175 | percent |
| Llama 4 Scout | #82 | llama-4-scout | Meta | 3.298969072165 | percent |
| Gemma 3 27B | #83 | gemma-3-27b | 0.824742268041 | percent |
One best compatible score per canonical product · higher is better
| Rank | Model | Weights | Evidence | Score |
|---|---|---|---|---|
| 01 | Grok 4.6xAI | Closed weights | Direct source result | 50.7% |
| 02 | GLM-5.3Z.AI | Open announced | Direct source result | 50.3% |
| 03 | GLM-5.3-FlashZ.AI | Open weights | Direct source result | 47.2% |
| 04 | Kimi K3Moonshot AI | Open weights | Direct source result | 46.0% |
| 05 | Qwen3.8-Flash-NextAlibaba Cloud | Open weights | Public reference | 45.4% |
| 06 | GPT-5.6 SolOpenAI | Closed weights | Direct source result | 44.3% |
| 07 | Claude Opus 5Anthropic | Closed weights | Direct source result | 42.1% |
| 08 | Grok 4.5xAI | Closed weights | Direct source result | 42.1% |
| 09 | DeepSeek V4 Flash Vision ExpDeepSeek | Not documented | Public reference | 41.0% |
| 10 | GPT-5.6 TerraOpenAI | Closed weights | Direct source result | 40.2% |
Each distinct published measurement is retained. Same-snapshot canonical and configuration projections appear once.
| Published model / configuration | Score | Evidence & protocol | Source & dates |
|---|---|---|---|
Qwen3.8-Flash-Next (Artificial Analysis independent run)Qwen3.8-Flash-NextExact identityqwen-3-8-flash-next-aa-unspecified Canonical product: qwen-3-8-flash-next | 45.4% | Public referenceindependently-verifiedVersion & system3 Artificial Analysis tau3-Banking evaluation | Qwen3.8-Flash-Next individual evaluationsObserved Checked |
Claude Opus 4.7 (Adaptive Reasoning, Max Effort) (Artificial Analysis independent run)Claude Opus 4.7Exact identityclaude-opus-4-7-max Canonical product: claude-opus-4-7 | 34.6% | Direct source resultindependently-verifiedVersion & system3 Artificial Analysis tau3-Banking evaluation | Claude Opus 4.7 (Adaptive Reasoning, Max Effort) individual evaluationsObserved Checked |
GLM-5.2 (max) (Artificial Analysis independent run)GLM-5.2Exact identityglm-5-2-max Canonical product: glm-5-2 | 34.6% | Direct source resultindependently-verifiedVersion & system3 Artificial Analysis tau3-Banking evaluation | GLM-5.2 (max) individual evaluationsObserved Checked |
Qwen3.5 122B A10B (Reasoning) (Artificial Analysis completed independent run)Qwen3.5-122B-A10BExact identityqwen3-5-122b-a10b-aa-reasoning-default Canonical product: qwen3-5-122b-a10b | 15.3% | Direct source resultindependently-verifiedVersion & system3 Artificial Analysis tau3-Banking evaluation | Qwen3.5 122B A10B (Reasoning) current model and individual-evaluation pageObserved Checked |
Mistral Medium 3.5 (Artificial Analysis completed independent run)Mistral Medium 3.5 128BExact identitymistral-medium-3-5-128b-aa-reasoning-default Canonical product: mistral-medium-3-5-128b | 15.1% | Direct source resultindependently-verifiedVersion & system3 Artificial Analysis tau3-Banking evaluation | Mistral Medium 3.5 current model and individual-evaluation pageObserved Checked |
Kimi K2.5 (Reasoning) (Artificial Analysis completed independent run)Kimi K2.5Exact identitykimi-k2-5-thinking Canonical product: kimi-k2-5 | 14.2% | Direct source resultindependently-verifiedVersion & system3 Artificial Analysis tau3-Banking evaluation | Kimi K2.5 (Reasoning) current model and individual-evaluation pageObserved Checked |
Qwen3.5 397B A17B (Reasoning) (Artificial Analysis completed independent run)Qwen3.5 397B A17BExact identityqwen3-5-397b-thinking Canonical product: qwen3-5-397b | 13.4% | Direct source resultindependently-verifiedVersion & system3 Artificial Analysis tau3-Banking evaluation | Qwen3.5 397B A17B (Reasoning) current model and individual-evaluation pageObserved Checked |
LongCat 2.0 (Artificial Analysis independent run)LongCat-2.0Exact identitylongcat-2-0-default Canonical product: longcat-2-0 | 13.2% | Direct source resultindependently-verifiedVersion & system3 Artificial Analysis tau3-Banking evaluation | LongCat 2.0 individual evaluationsObserved Checked |
Gemma 4 26B A4B (Reasoning) (Artificial Analysis completed independent run)Gemma 4 26B A4BExact identitygemma-4-26b-a4b-aa-reasoning-default Canonical product: gemma-4-26b-a4b | 12.0% | Direct source resultindependently-verifiedVersion & system3 Artificial Analysis tau3-Banking evaluation | Gemma 4 26B A4B (Reasoning) current model and individual-evaluation pageObserved Checked |
Qwen3.6 35B A3B (Reasoning) (Artificial Analysis completed independent run)Qwen3.6-35B-A3BExact identityqwen3-6-35b-a3b-aa-reasoning-default Canonical product: qwen3-6-35b-a3b | 9.3% | Direct source resultindependently-verifiedVersion & system3 Artificial Analysis tau3-Banking evaluation | Qwen3.6 35B A3B (Reasoning) current model and individual-evaluation pageObserved Checked |
From result to context
Fintech customer-support agent benchmark requiring knowledge-base navigation and multi-step tool use on banking workflows.
The summary shows one best compatible source result per canonical product. Versions, effort settings and execution systems remain attached to the underlying records.
Scores from different versions or harnesses may not be interchangeable. The published source rows preserve those distinctions and their original units. A source result is not automatically an input to a current index.
This catalogue entry preserves available source evidence. Current indices admit only their specifically reviewed tracks and configurations.
Contamination risk: Low. Lifecycle: Active.
