New AI models keep arriving, each with a name, a price and a table of numbers showing it beat the last one. You probably recognise the names and the prices already, even if you have never opened a developer tool. Those numbers answer the vendor’s question, not yours: how to choose an AI model for your actual work.
Whatever happens with AI, treat it like any other job in your business. A supplier who quotes a delivery without saying what arrives gets asked what changes and how you would tell; a price per pallet with no promise of intact pallets would not survive the first meeting. A model quote is that kind of quote: names, a price per million tokens (the unit providers bill by), a claim about better output, and no workload result anywhere in it.
A renewal makes it concrete, and it is a decision about what to do next, whether to work with a new model at all. The question is “should we change?” Answer it in your favour: name one recurring task, name the person who can judge a result on it, and ask what evidence would show a different configuration handling it at least as well. The model name becomes a hypothesis, and your own work becomes the test. Each section below is one step of that decision: what done means, what to compare, what the example shows, what it costs, what the paperwork leaves out, what to bring to the meeting.
Write down what counts as done, before any model name
An evaluation, plainly, is repeated checks on representative examples of the actual job. Not a demo, not a benchmark table: your cases, your material, your grading rule. Representative means three kinds: the routine ones your team sees daily, the difficult ones that go wrong, and the ones that should be refused or escalated rather than answered. Anthropic’s own evaluation guidance asks for criteria that are specific, measurable, achievable and relevant, and for task-specific tests that include edge cases: vendor guidance, useful as method, not as independent proof (read here, November 2026).
What is a dataset? Not a research project. It is the cases you grade the candidates on: your own inputs, the accepted answer for each, and the rule that says which answers pass. Call it a test set if “dataset” sounds like data science.
How do you build one, and how should you think about it? Pull recent examples from the work as it runs, messy ones included; the person who does the job confirms the accepted answer for each; keep anything your data rules forbid out of the set. Think sample, not proof: it shows whether two candidates handle the same job differently, certifies nothing for production, and says nothing about anyone else’s work.
Can the process be broken into steps, and what happens at each? Six steps: define what counts as done, collect the cases, write the grading rule, run every candidate on the same cases, count what was accepted against what failed the critical-error rule, then decide. This section is the first three.
When AI runs throughout a workflow, do you test the whole workflow or divide it into stages? Divide it. Test the stage you mean to change with the rest held fixed: a whole-workflow run tells you something failed, not which stage. Test handoffs only once each stage passes alone, then run the complete workflow end to end against the business outcome: stages passing is diagnosis, the whole run passing is acceptance.
What are the tactics? Keep everything identical except the one thing you change: same prompt, material, tools and grading rule; record the model version and the reasoning setting (a dial that trades thinking time for quality) per run; set acceptance criteria before you run anything; count review time with the quality, not after it. OpenAI’s guidance says the same (the guide, read November 2026). Then record three decisions before testing:
- The accepted unit. Write the smallest output you would actually pay for: a checked record, a correctly routed request, a source-traceable draft. Attempts are not units.
- The permitted route. Which inputs may leave your systems, and through which service. A model that scores well on a disallowed data path has not passed anything.
- The critical-error rule. Name the mistakes that block the work rather than delay it, and count them separately from ordinary rejections.
Your comparison sheet then carries the rest: workload owner, baseline, candidate configuration, quality rule, review method, where the test cases came from, failed and accepted counts, human minutes, all-in cost, and the escalation or stop trigger. Two cautions: thresholds come from the workload owner and its process, never a model’s confidence statement; and a short pilot shows only what happened in the pilot, not that a rare event stays rare.
The worksheet’s first block: four fields, each owned by a person. Before any model is named, the work, the accepted unit, the permissions and the evidence are on paper.
Only after those conditions exist does it make sense to introduce candidates, because a disallowed data path never becomes acceptable when a model scores better.
Treat the light model as a hypothesis
Three testing levels are worth naming, and none is a ranking: a light, narrow candidate for the straightforward work; a general-purpose candidate as the current baseline; a stronger-reasoning candidate for difficult synthesis where an omission costs more than the tokens. These are hypotheses you test against your own sheet, not categories with entitlements.
A configuration here is one model and version at a given reasoning setting, plus the tools and permissions you give it. Change those dials separately: a model change and an effort change are different experiments, and mixing them tells you nothing about either. Keep three words apart that marketing tends to braid together: small, open and safe. None of them implies the others.
Three candidate levels as columns of one sheet. Each column is a full configuration; all are tested on the same workload and scored in accepted units.
There is historical precedent, and only that: in May 2023 researchers published FrugalGPT, a method that learns which combinations of models to use for different queries, to reduce cost and improve accuracy (the paper, submitted 9 May 2023). The idea is studied; its numbers are three years old, and none of them transfer to your work.
What you do with the three levels is management, not engineering. Test the lighter configuration against the current baseline on the same examples, then keep it, split the workload between configurations, escalate the difficult subset, or stop. No router, no new tooling and no code project is required to run that comparison. The durable parts around a changing model and the overhead a configuration carries are written up elsewhere (what you build when models keep changing, anatomy of an agent harness); this article needs only the sheet.
Four ordinary jobs on one sheet
The rows below are illustrative test plans: not results from evonomics, from a client, from a vendor benchmark or from a tested model. The difference between an initial candidate and an accepted assignment is the whole point of the table.
| Workload / accepted unit | Candidate level to compare with baseline | Failure cost / risk | Check and ownership | Escalation or stop rule / planned decision |
|---|---|---|---|---|
| Route a routine inbound request to a department; accepted unit is a correctly routed request with no invented instruction. | Light candidate first, with general-purpose baseline. | Wrong queue delays an answer; ambiguous or sensitive requests have different handling. | Operations owner uses labelled representative requests including ambiguity and refusal cases and records errors plus review minutes. | Unknown category or missing required data goes to a person. Retain baseline if the lighter candidate misses the agreed routing rule. No claimed saving before measurement. |
| Extract purchase-order fields from a supplied document; accepted unit is a checked record, not an attempted extraction. | Light candidate against general-purpose baseline on identical documents and field schema. | Wrong total, identifier or missing field contaminates downstream records. | Process owner checks exact fields against source; critical-field mismatches are counted separately. | Failed critical field blocks the downstream write and requires source review. The worked example below rejects Candidate A and keeps B for a further controlled trial; it does not approve autonomous operation. |
| Produce a management comparison from a supplied evidence pack; accepted unit is a source-traceable draft ready for named review. | General-purpose baseline versus stronger-reasoning candidate on the same pack and grading rule. | Omitted obligation or invented fact misleads the decision; shorter prose is not a quality metric. | Named subject-matter reviewer checks completeness, cited facts, qualifications and review/rework time. | Missing evidence or an unresolved contradiction returns to source review. Escalate the difficult subset only when the comparison supports it. |
| Recommend an action with material consequences for a person or a business commitment. | A candidate model may support a bounded preparatory subtask only. | Consequences and permissions can dominate capability and price. | An accountable human defines permitted use and verifies the process before any output influences action. | No model-tier choice authorises the consequential action. Stop or narrow the task when the process or authority is absent. This is a management boundary, not legal advice. |
Complexity, failure cost, deployment and model tier are four different axes in that table. No tier on its own permits autonomous action.
When you need synthetic data
Three situations make synthetic data essential. You are launching a new product or chatbot and have no customer data yet: generate your own first. You are testing before go-live and need to know whether the responses are correct: generate synthetic customers, behaviours and profiles as test data. The real dataset is confidential, patient or client records you may not expose: you know its structure and format, so you replace the content with synthetic equivalents of the same shape.
Hamel Husain and Shreya Shankar settle the when in their evaluation FAQ: start error analysis before you have enough production traffic, or to test a known failure that appears rarely in real data. The earlier evals post makes the same point for launches: you do not need to wait for production data to test your system, and educated guesses about how users will work, turned into generated cases, are the beginning. The limit in that same FAQ is the one this piece keeps: synthetic data cannot tell you how common a failure is in production, so compare it against real data as soon as real data exists.
The decision this feeds is simple: can the new model beat the old one on quality, cost and speed? The how-you-compare half is compact: two configurations, the same 100 purchase-order cases, cost per accepted record, review minutes, and a predeclared rule of at least 95 accepted with no critical mismatch. What follows is invented test data; the EUR 60 an hour review rate is an assumed figure, not a recommendation.
| Worked example: input and result | Candidate A (light test hypothesis) | Candidate B (baseline) |
|---|---|---|
| Cases attempted / accepted | 100 / 90 | 100 / 98 |
| Critical mismatches | 1 | 0 |
| All model and tool charges for the run | EUR 2 | EUR 8 |
| Review and rework time at the assumed EUR 60/hour | 30 min = EUR 30 | 10 min = EUR 10 |
| Allocated rollout, hosting or switching cost for this run | EUR 0 (explicit simplifying assumption) | EUR 0 (same assumption) |
| Run cost / cost per accepted record, rounded | EUR 32 / EUR 0.356 | EUR 18 / EUR 0.184 |
| Decision under the predeclared rule of at least 95 accepted and no critical mismatch | Reject for this task: the quality rule fails regardless of price. | Meets this predeclared rule; keep as baseline for a further representative comparison. |
The worked example in full: the cheaper model bill still produces the more expensive accepted record, and one critical mismatch rejects Candidate A before price enters the decision.
The lesson is the definition and the order of the decision, not the arithmetic. A’s model and tool charges are lower, but its review burden makes both the run and each accepted record more expensive, and its critical mismatch independently disqualifies it under the predeclared rule. B passing this rule is not proof that B suits your work; it earns another controlled comparison. Cases that were not accepted stay failures on the sheet, and every failed-attempt charge stays in the cost. In a real sheet the zero-overhead row becomes whatever the rollout and switching actually cost you.
Divide by the work you accepted
Before whether to change can be answered, both candidates need the same price tag. Turn the sheet’s observations into one comparable figure: total delivery cost per accepted unit, a plain management definition rather than a vendor metric.
total delivery cost per accepted unit = (model charges for all attempts + tools and hosting allocation + human review and rework + rollout and switching allocation) / units accepted against your predefined criteria
The denominator as the ledger sees it: total delivery cost of the run divided by the units the business accepted. Failed attempts sit in the numerator; acceptance failures are reported next to the cost, never folded in.
Three rules keep that number checkable. The denominator is accepted work, never attempts, so a configuration that produces a lot of output and little accepted output cannot look cheap. Failed attempts belong in the numerator, and acceptance failures are reported beside the cost, never folded in as a discount: a low unit price cannot cancel a failed quality or permission condition.
Put the unit and the denominator on the same line as the number: a cost per accepted record without the case count, the review minutes and the run date is a figure nobody can check, you included, six months later. Freeze the date, the model version, the reasoning setting, the token categories and the currency beside every comparison you keep.
Watch which lever you pull: an effort setting, an added review step and a model switch are three experiments with different cost profiles, and only the third changes the model you buy. Whether a cheap checking step is worth its cost is its own question (that distinction has its own article).
Buying API calls and operating a model are different cost structures: a usage bill versus capacity, patching and failure ownership. Neither is automatically cheaper for your workload, and the licence on the model does not settle the choice.
A licence is not a hosting plan
Take one model card as an example. The publisher’s repository for GLM-5.3-Flash carries an MIT licence: use, modification and distribution permitted as long as the notice is retained, software with no warranty (licence text, read 4 November 2026). That is permission to use the code. It is not a statement about where your inputs are processed, who runs the service, what a contract covers or which regulator applies, and other open models publish different terms.
The same card describes 320B total parameters with 18B active (model card, read 4 November 2026). Neither number tells you what it takes to store or run the thing, so no hardware recipe follows from them, and an active-parameter count is not a footprint.
Four questions do the procurement work here, yours to get answered, not mine to answer. Where are inputs and outputs processed and retained? Who operates the service and patches it? Who owns the failure when the output is wrong, and what does it cost you? How is switching tested before you need it? This is a procurement checklist, not legal advice, and nothing above certifies a deployment.
A licence settles permissions for one repository. Where data is processed, who operates the service and how switching is tested are separate questions, answered in writing before deployment.
Three questions for the decision meeting
The whole method compresses into three questions for the meeting that decides something: whether to change, and on what evidence. Put them to a supplier or your own team.
- Which configuration handles this named workload, what counts as accepted work, and what same-input evidence shows a lighter configuration failing?
- What does one accepted unit cost, including failed attempts, tools and hosting, review and rework, and switching, and which quality or critical-error failures remain?
- Where is data processed and retained, who owns review and exceptions, and what triggers escalation, rollback or a new comparison when the model, the terms, the price or the workload changes?
The three procurement questions close the sheet. The first comparison underneath is small: one task, one owner, the same cases, and a decision on record.
The next step is smaller than a project. Name the task and its owner, write the accepted unit and failure rules on one page, gather representative examples your data rules allow, compare a lighter configuration with the baseline on them, and record the verdict: keep, escalate or stop. Revisit it when the workload, the model version, the terms or a material price condition move. If you would rather design that first comparison with someone who has run the exercise before, that is the consulting work behind evonomics.
That one page is the whole prerequisite. The renewal can wait until it exists.