Grok 4 Just Launched: We Ran Our 6 Production Workflows on It Today

xAI released Grok 4 today, July 9, 2025 — and it is the first model to clear 50% on Humanity's Last Exam. So we did the obvious thing: we pulled six of our live client automation workflows off their current models and ran them on Grok 4 for an afternoon. We measured latency, output accuracy, and refusal rate against Claude Sonnet 4 and GPT-4o. This post is the raw scorecard, the cost-per-task math, and which of the six we would actually move.

Should you switch your AI workflows to Grok 4?

For most production workflows, not yet. In our six-workflow test on launch day, Grok 4 matched or slightly beat Claude Sonnet 4 on hard reasoning and long-context tasks, but it was slower on median latency and more expensive per task than GPT-4o for routine work. The honest answer: keep your cheap, fast model for high-volume simple tasks, and consider Grok 4 only for the one or two workflows where deep reasoning is the bottleneck.

Why this matters now (July 9, 2025)

Every model launch triggers the same reflex: a founder reads the benchmark tweet and asks why we are not already migrating. Grok 4's benchmarks are genuinely strong — xAI reports it as the first model past 50% on Humanity's Last Exam, with leading scores on competition-math and reasoning suites. But benchmark numbers and production behaviour are different animals.

What benchmarks do not tell you: how the model behaves on YOUR prompts, with YOUR data, under YOUR latency budget, at YOUR volume. A model that wins a math olympiad can still be the wrong choice for a workflow that summarizes 400 support tickets an hour, where a model that is half as smart but a quarter of the price and twice as fast wins on every axis that matters to the client's bill. The only test that counts is the one you run on your own workflows.

That is why we run this exact drill on every major launch. We keep six representative client workflows wired up behind a model-router flag, swap the model, and replay the same fixed inputs. It takes an afternoon and it replaces a week of speculation on Twitter.

The 6 workflows we tested

The scorecard (today's run)

These are directional results from a single afternoon on fixed inputs, not a peer-reviewed study — read them as "what we saw," not gospel. Latency is median time-to-completion on our prompts. Accuracy is our reviewer's pass rate on a 25-sample set per workflow. Refusal is how often the model declined a legitimate task.

The headline: Grok 4 clearly won exactly one of six — the SQL-from-English workflow, where its reasoning produced correct, runnable queries on schema-join cases the other two fumbled. On everything else it was competitive but not better enough to justify a higher per-task cost or slower response. That is a normal launch-day result, and it is fine. You do not need a new model to win everywhere — you need it to win the one place your current model is weak.

The cost-per-task math (where the decision really lives)

Per-token pricing is a distraction until you convert it to cost per task at your real token counts. Grok 4 sits well above GPT-4o on output-heavy work. For a high-volume workflow, that gap compounds into a meaningful monthly line item.

For 50,000 tickets a month, that gap adds up fast — for a task where all three models pass. This is why we never let a benchmark tweet drive a migration. The right model for a workflow is the cheapest one that clears your quality bar at your latency budget. We bake this logic into a model-router so each workflow runs on its own best-fit model, a pattern we ship as part of our AI automation work.

The DIY: run this test on your own workflows

When NOT to switch models

Do not migrate a working production workflow on launch day because a benchmark looks good. Benchmarks measure capability ceilings; your workflow lives in the floor — average behaviour on ordinary inputs. We have seen teams chase a 2-point benchmark gain into a 3x cost increase and a latency regression their users actually felt. If your current model passes your golden set, the burden of proof is on the new one.

Do not switch a tone-sensitive workflow — sales emails, customer replies — on accuracy numbers alone. Grok 4 was technically correct on our sales-email task and still read stiffer than GPT-4o. Tone does not show up in a benchmark. And do not adopt any model for regulated or sensitive data without checking its data-handling and retention terms first; capability is irrelevant if the data policy fails your compliance review.

Real example: a model-router for a logistics client

A 240-person logistics firm we built a custom HRMS and ops stack for runs four AI workflows: document classification, shipment-exception summaries, a Hindi-English support assistant, and SQL-from-English for their ops analysts. Before we added routing, all four ran on a single model "to keep it simple," and the bill reflected it.

We split them across models by fit. Classification and the support assistant went to the cheapest fast model; the exception summaries and SQL generation went to stronger reasoning models. The same logic we used today would now move their SQL workflow to Grok 4 on its merits. Net effect of routing: their monthly AI spend dropped about 38% with no drop in output quality, because they stopped paying premium-model prices for tasks a cheaper model handled. We use the same voice and conversational stack on TalkDrill, our in-house English-speaking app with 5,000-plus users, where per-interaction cost decides whether a feature is viable.

Frequently asked questions

Is Grok 4 better than Claude Sonnet 4 and GPT-4o?

On hard reasoning and competition-math benchmarks, Grok 4 leads. In our six-workflow production test it clearly won only the SQL-from-English task. For high-volume simple work, GPT-4o stayed cheaper and faster. "Better" depends entirely on the workflow — there is no single winner.

How much does Grok 4 cost compared to GPT-4o?

Grok 4 costs meaningfully more than GPT-4o on output-heavy tasks. For our ticket-triage workflow, Grok 4 came out well above GPT-4o per 1,000 tasks — for a task all three models pass.

What is Grok 4's context window?

Grok 4 ships with a 256K-token context window, which is comfortable for long documents like 30-page contracts. In our clause-extraction test it handled the full document well, tying with Claude Sonnet 4 on accuracy.

Should I switch all my workflows to one model?

No. The cheapest reliable approach is a model-router: each workflow runs on its own best-fit model. A single model forces you to overpay for simple tasks or under-serve hard ones. Routing typically cuts AI spend 30-40% with no quality loss.

How long does it take to test a new model on our stack?

If your workflows are behind a config switch and you have golden sets, an afternoon. The one-time cost is building the golden sets and the harness. After that, every future launch is a same-day decision instead of a week of speculation.

Can you set up model-routing for our existing AI workflows?

Yes. We wire a router into your stack, build the golden sets, and hand you a dashboard showing cost and quality per workflow per model. You then add or swap models yourself as launches happen. First call is technical, with the engineer who would build it.