Claude Model Comparison on Finance Reasoning — Sonnet, Opus, Haiku

This site runs three separate eval tracks (see the full benchmark and methodology). Track 3 — LLM accuracy — is the one where comparing model tiers is actually meaningful: 22 finance-reasoning questions, scored by Claude-as-judge, rather than closed-form correctness tests. Until now this site had only ever run that suite against one Claude model at a time and never published a tier comparison. We fixed that and ran it for real, across four current Claude models, and are publishing what came back — including the parts that don't flatter the largest model.

Methodology

The same 22-question dataset — 10 domains: valuation, accounting, risk, compliance, FP&A, M&A, structured products, hallucination probes, advanced quant, and ethics boundaries — was sent to four models as independent CLI invocations:

• claude-sonnet-5
• claude-sonnet-5-5
• claude-opus-5-5
• claude-haiku-5-5

Every response — from all four models — was scored by the same judge, Claude Sonnet 5.5, using G-Eval style chain-of-thought reasoning across 4 scoring dimensions. Pass threshold is overall ≥ 7.0/10, consistent with the rest of Track 3. We deliberately excluded Fable 5.1 from this run — it requires usage credits not available in our eval environment, and we'd rather omit a model than publish a number for it we can't stand behind.

Results

The headline that jumps out — Haiku 5.5 edging out Opus 5.5 — is real in this data, and we're not going to round it away. But read it for what it is: this is a narrow set of well-specified, mostly single-answer reasoning tasks (DCF math, covenant tests, hallucination traps, compliance identification), not open-ended synthesis or judgment calls under ambiguity. Smaller models are frequently fully competitive on tightly-scoped tasks like these. The two Opus misses were an advanced-quant convertible bond question and an ethics-boundary scenario — both categories where more elaborate reasoning paths create more surface area for a scoring miss on a strict rubric, not necessarily a worse underlying model.

What This Does and Doesn't Show

What it shows: on this specific 22-question finance-reasoning suite, all four current Claude model tiers score well (8.45–9.18 average), and the smallest model tested is fully competitive with the largest on this task shape. That's a real, measured result worth knowing if you're choosing a model tier for narrowly-scoped finance Q&A or calculation-checking work, where latency and cost matter and the task is well-specified.

What it doesn't show: this is n=22 questions × 4 models, judged by one judge model — a pilot, not a comprehensive capability benchmark. It doesn't test longer documents, multi-turn analysis, ambiguous or underspecified prompts, adversarial inputs beyond the two hallucination/ethics probes included, or anything outside these 10 domains. It also doesn't yet include non-Claude models (GPT, Gemini) — that requires API keys this environment doesn't currently have configured; when it does, they'll be added to this same comparison rather than a separate claim.

Where We're Taking This Next

Same approach as our determinism reproducibility study: this is the first entry in an ongoing series, re-run and republished as models change, not a one-time marketing snapshot. Planned next: add GPT and Gemini models once API access is set up, widen the dataset past 22 questions, and track how scores shift as new Claude models ship.

Related

• The full FinSkilz eval benchmark — all 3 tracks
• Full eval methodology
• Is Claude Deterministic? A Reproducibility Study
• Verifiability vs. Determinism — what's the difference?
• Finance AI Model Validation Framework