Claude Haiku 5.5: evaluate small-model work by task cost
Anthropic’s release targets fast, high-volume workloads with pricing that depends on prompt length.
Claude Haiku 5.5 official release artwork. Credit: Anthropic. Original image ↗
Anthropic’s Haiku 5.5 announcement introduces a small model intended for fast, high-volume work. The release discusses coding, tool use, computer use, and agent tasks alongside cost and latency.
Evaluate cost per successful task
Token pricing helps estimate a request, but a workflow can include retries, tool calls, validation, and escalation. A smaller model that produces several invalid results may cost more to finish the job than its first response suggests.
Choose a narrow workload with a measurable result: extracting known fields from a document, assigning a label from a fixed set, or producing a patch that must pass existing tests. Keep a representative set of inputs and define acceptance before comparing models.
Performance: compare the task, then the model
Anthropic’s reported results compare different tasks. Choose one evaluation; rating scores and percentages use separate scales.
OSWorld 2.1 — offline subset
Focus the graph and use arrow keys, or tap a point, to inspect values. The complete data table is available below.
Zero baseline. Missing values remain unreported. Sonnet 5.5 is a reference model; its FrontierCode result uses Xhigh effort. Qentrah did not reproduce these evaluations. Aggregate scores do not reconstruct the source’s accuracy-versus-cost curves. Evaluation methodology ↗
| Model | OSWorld 2.1 — offline subset |
|---|---|
| Haiku 5.5 | 72.4% |
| Haiku 4.5 | 15.7% |
| GPT-6 Luna | 48.9% |
| Sonnet 5.5 | 83.9% |
Read the complete benchmark table
| Evaluation | Haiku 5.5 | Haiku 4.5 | GPT-6 Luna | Sonnet 5.5 |
|---|---|---|---|---|
| GDPval-AA v2.1 | 1620 | 735 | 1437 | 1840 |
| AA-Briefcase v1.1 | 1578 | 614 | 1336 | 1824 |
| OSWorld 2.1 — offline subset | 72.4% | 15.7% | 48.9% | 83.9% |
| Humanity’s Last Exam — no tools | 45.9% | 10.2% | Not reported | 56.9% |
| Humanity’s Last Exam — with tools | 57.4% | 18.7% | Not reported | 64.5% |
| Terminal-Bench 4.0 | 39.2% | 0% | 16.4% | 70.6% |
| FrontierCode 1.1 (Main) | 46.4% | Not reported | 42.4% | 52.1% |
| Chartography — no tools | 46.4% | 6.4% | 29.1% | 61.6% |
Pricing has two bands
As published at our October 12 source check:
| Prompt length | Input per million tokens | Output per million tokens |
|---|---|---|
| Up to 100,000 tokens | $0.10 | $0.50 |
| Over 100,000 tokens | $0.50 | $2.50 |
These are Claude Platform prices from the announcement. Consult the current pricing and provider terms before budgeting; caching, batches, and other platforms have their own conditions.
Token price is only part of the comparison
Anthropic notes a changed tokenizer, so token counts can differ from the previous Haiku model. Compare completed-task cost, not just the price of a nominal million tokens.
Our recommendation is to include retries, invalid tool calls, and escalation to a larger model in that comparison. A cheap first response can become expensive if it repeatedly needs repair.
Work through uncached request cost
Let I and O be input and output token counts, and pi and po their respective prices per million tokens. Ignoring cache use and other discounts, a simple request estimate is:
Using the announcement's shorter-prompt band, an illustrative 20,000 input tokens and 2,000 output tokens give C = (20,000 × 0.10 + 2,000 × 0.50) / 1,000,000 = $0.003. That is three-tenths of a cent. It is an arithmetic estimate using published prices, not a bill from an executed API request. Cache writes and reads, batching, tool charges, provider rules, and changes in pricing require their own accounting.
For a hypothetical prompt of 150,000 tokens and the same 2,000 output tokens, applying the displayed longer-prompt band gives (150,000 × 0.50 + 2,000 × 2.50) / 1,000,000 = $0.08. Verify the provider's treatment of the boundary and complete pricing conditions before budgeting. Splitting a prompt into shorter requests also changes available context and may change the result, so it is not a free substitute for the same computation.
Work through completed-task cost
Imagine ten extraction tasks, each with one request costing $0.003. Eight pass validation, and two need a separate $0.02 repair request. If all ten eventually succeed, total request cost is 10 × $0.003 + 2 × $0.02 = $0.07, or $0.007 per completed task. These are illustrative counts and repair prices, not measured Haiku failure rates or fallback-provider prices.
This accounting makes a migration trial concrete. Retain first-attempt validity, retries, final success, latency, and total billed usage as separate outcomes. A low token price may be valuable when the task is narrow and validation is inexpensive. A larger model may be preferable when it prevents repeated repair on a more complex task. The announcement's benchmark selector above supports task-specific reading; it cannot determine that application's completed-task cost without actual usage and acceptance data.
Run a controlled migration trial
- The announcement names
claude-haiku-5-5for the Claude Platform. Follow the migration resources linked from Anthropic’s release for the provider you use. - Keep the task instructions and validation rules consistent across the comparison.
- Measure response latency, success rate, usage, and the number of repair attempts.
- Inspect failures, especially when a tool argument or structured field looks valid but means the wrong thing.
- Add any fallback-model cost to the completed task’s total.
This is an evaluation plan, not a claim that we executed a Haiku benchmark. Record the actual provider and request configuration so that the trial can be repeated.
Watch the long-prompt boundary
The published pricing bands distinguish prompts up to 100,000 tokens from longer ones. The announced tokenizer change also means the same text can have a different token count from the previous model. Use measured usage for your payloads instead of assuming an old estimate carries over.
A migration decision should consider the quality you need and the work needed to reach it. Start with a small shadow evaluation before changing a live application, consult current provider pricing, and retain the examples that expose regressions. The announcement’s benchmarks describe Anthropic’s tests; your application’s acceptance criteria should determine whether this release is an improvement for you.
