We Reran 7 Production Workflows on Claude Opus 4 and Sonnet 4

On May 22, 2025, Anthropic shipped Claude Opus 4 and Sonnet 4 — both scoring around 72% on SWE-bench, the hardest mainstream coding benchmark. Benchmarks are one thing; our production bill is another. So we reran 7 real workflows we operate — for clients and for our own products — on the new models and measured what actually changed: task quality, cost per task, and whether migrating now beats waiting. Here are the numbers.

The 60-Word Answer

Claude Opus 4 and Sonnet 4 launched May 22, 2025. Opus 4 leads SWE-bench at 72.5% (Terminal-bench 43.2%); Sonnet 4 hits 72.7%. Across our 7 reran workflows, the big wins were agentic coding and long multi-step tasks. Our call: migrate coding and agent workflows to Sonnet 4 now, keep simple high-volume tasks on cheaper models, and reserve Opus 4 for the hardest jobs.

Why This Matters Now (May 2025)

Anthropic's Claude 4 announcement dropped May 22, 2025, calling Opus 4 "the best coding model in the world." Both are hybrid models — near-instant responses or extended thinking for harder problems — and both run on the API, Amazon Bedrock, and Google Cloud Vertex AI. For us, a model launch isn't news until we've rerun real workloads. A 72% SWE-bench score is meaningful only if it moves our cost-per-resolved-task. So we tested.

Our Test Method (So You Can Trust the Numbers)

We didn't run a synthetic benchmark. We replayed real production traffic. Here's the method — copy it for your own stack.

This is the same eval discipline we use to ship prompt changes safely — see our writeup on prompt eval pipelines. Without a blind, outcome-based eval, "the new model feels better" is how teams burn budget on a downgrade.

The Results: 7 Workflows

Here's what changed, workflow by workflow. Deltas are versus the model we ran before the migration test.

The pattern is sharp. Claude 4 pays off on hard, multi-step, agentic, and coding work — exactly where the SWE-bench and Terminal-bench gains predicted. On simple, structured, high-volume tasks, a cheaper model already hits the quality ceiling, and paying more buys nothing. This is why blanket "upgrade everything to the newest model" is usually wrong.

The Cost-Per-Task Reality

Per-token price is a trap. Here's cost per resolved task on the agentic bug-fix workflow — the metric that actually hits your bill.

Sonnet 4 cost more per token on some comparisons, yet less per resolved task — because it finished more jobs on the first try and needed fewer retries. That's the whole argument for outcome-based evals. A model that costs 20% more per token but fails 40% less can be the cheaper choice once you count only the work that actually got done.

There's a hidden cost the per-token view never shows: human takeover. As Hrishikesh, our CTO, puts it, the most expensive token in any agentic pipeline is the one that makes a senior engineer stop their own work to rescue a stuck agent. On the bug-fix workflow, every failed run didn't just cost a retry — it sometimes cost 20 minutes of a human's attention to diagnose and finish. When we priced that engineer time into "cost per resolved task," the gap between the old model and Sonnet 4 widened further than the token math alone suggested. If your eval ignores human-takeover time, you're undercounting the real cost of a model that fails often.

What We Migrated and What We Didn't

We applied this routing on our own products too. On TalkDrill, our in-house English-speaking app, conversational quality matters but so does per-session cost at scale — so we route the hard reasoning to a stronger model and the high-volume turns to a cheaper one. Our founder Vivek Kumar has written about the same migrate-vs-wait calculus from a builder's seat.

Your Model-Migration Checklist

Run this before moving any workflow to a freshly launched model.

• The workflow has an objective pass/fail or graded success metric
• You replayed 100–300 real historical inputs, not synthetic prompts
• The same prompt ran on old and new models — only the model changed
• Outputs were graded blind (grader didn't know which model produced which)
• You measured cost per resolved task, not cost per token
• You priced in human-takeover time for failed runs
• The decision was made per workflow, not blanket across the org
• Cheap models stayed on tasks already at the quality ceiling

Common Mistakes (How Teams Botch a Model Migration)

A fourth mistake: migrating the day of launch with no eval at all, on the assumption that newer is strictly better. It usually is on hard tasks and usually isn't on easy ones. The launch is your cue to test, not to ship.

A Real Example: A Mumbai SaaS Team's Coding-Agent Bill

A Mumbai SaaS team ran an internal coding agent that fixed bugs from their issue tracker. It worked, but cost-per-fix was creeping up as their codebase grew. After Claude 4 launched, they asked us whether to migrate.

That split decision — migrate the coding agent, keep the classifier — is the model-routing audit we now run as a standard service. It's the same architectural thinking behind systems like ExamReady, where different tasks demand different models. Our AI and automation team runs this for clients deciding whether a model launch is worth the migration cost.

FAQ

When did Claude Opus 4 and Sonnet 4 launch?

Anthropic launched Claude Opus 4 and Claude Sonnet 4 on May 22, 2025. Both are hybrid models offering near-instant responses or extended thinking for harder problems, and both are available on the Anthropic API, Amazon Bedrock, and Google Cloud Vertex AI from launch.

How good is Claude Opus 4 at coding?

Anthropic called Opus 4 the best coding model in the world at launch. It leads SWE-bench at 72.5% and Terminal-bench at 43.2%. In our reran production workflows, the gains showed up most on agentic, multi-file, and long multi-step coding tasks — the hard work, not simple structured jobs.

Is Claude Sonnet 4 worth migrating to?

For coding and agentic workflows, in our tests yes. Sonnet 4 scored 72.7% on SWE-bench and often lowered our cost per resolved task by finishing more jobs on the first attempt. For simple, high-volume tasks already at the quality ceiling, a cheaper model still wins.

Should I upgrade all my AI workflows to Claude 4?

No. Migrate by workflow, not org-wide. Our 7-workflow test showed clear gains on coding and multi-step agents and no benefit on classification, extraction, and routine summaries. Route each task to the cheapest model that clears its quality bar — blanket upgrades add cost without value on easy tasks.

How do I decide whether to migrate to a new model?

Run a blind, outcome-based eval. Replay real historical inputs through old and new models with identical prompts, grade outputs without knowing which model produced them, and measure cost per resolved task — not per token. "It feels better" is confirmation bias and a poor migration criterion.

What does Claude 4 cost?

At launch, pricing held steady with prior tiers, with Opus 4 priced above Sonnet 4. The cost that actually matters for your budget, though, is what you pay per successfully completed task, which depends on how often the model succeeds.