Evals: know whether your LLM system works before your customers find out
An LLM system isn't tested by reading ten answers and nodding. It is tested with real cases, graders you can audit and a threshold that blocks deployment when quality drops. Here is the stack we recommend, with its costs and its traps.
An eval is a repeatable test of an LLM system: fixed inputs, explicit criteria and a score you can compare across versions. Without one, every prompt change, model release or effort setting is a bet, and intuition can't measure it.
The stack that works has five layers: code checks, a versioned golden set, LLM judges calibrated against human labels, human review and production monitoring. Our rule: start with 20 to 50 real cases, gate every release on them and grow the set from production failures.
Why eyeballing fails
Reading a few answers feels like testing. It isn't.
The usual pre-launch test is a demo: someone types a dozen questions, the answers look fine, the team ships. The sample is tiny, biased and never re-run. When the prompt, the model or the index changes, nobody knows whether quality moved.
Perception is no more reliable. In METR's randomized trial (July 2025), experienced open-source developers took 19% longer with early-2025 AI tools while believing they were 20% faster. Public benchmarks don't fill the gap: in February 2026 OpenAI stopped reporting SWE-bench Verified after finding flawed tests in 59.4% of 138 audited hard tasks. And none of them measures the Spanish of a Bogotá insurer or the French of a Lyon retailer.
Surveys confirm it: in LangChain's survey of 1,340 practitioners (late 2025), quality was the top blocker to production (32%) and 89% had observability, but only 52.4% ran offline evals.
The eval stack: five layers
Each layer catches what the previous one misses.
| Layer | What it catches | When it runs | Limit |
|---|---|---|---|
| Code checks | Broken JSON, missing fields, wrong language, forbidden content, invalid tool arguments | Every commit | Says nothing about correctness |
| Golden set | Regressions on known cases with expected answers or criteria | Every prompt, model, effort or index change | Covers only what you included |
| LLM judge | Quality of open answers: faithfulness to sources, completeness, tone | With the golden set and on production samples | Biased until calibrated |
| Human review | Ambiguous or high-stakes cases, judge calibration | Weekly sample and major releases | Slow and expensive |
| Production monitoring | Drift, new intents, real failures, user feedback | Continuously, on sampled traces | Sees problems after users do |
Anthropic adds two useful distinctions. Capability evals track what the system can't do yet and start at low pass rates; regression evals protect what already works and should stay near 100%. For agents, pass@k (one of k attempts succeeds) differs from pass^k (all k succeed): only the second measures reliability.
Building the dataset
The golden set is the asset; tools are replaceable.
Anthropic recommends starting with 20 to 50 tasks drawn from real failures; OpenAI recommends mixing production data, including user feedback, with expert-written cases. Your set should look like your traffic, not like a benchmark.
- Real prompts from logs, support tickets and sales conversations. Logs often hold personal data under Colombia's Law 1581 of 2012 or the GDPR: anonymize them before copying anything into a test set.
- Edge and hostile cases: ambiguous requests, dirty data, broken tools, out-of-scope questions, instructions injected into documents. Anthropic stresses balance: test when a behavior should happen and when it shouldn't.
- Spanish and French cases written by native speakers, not translated: COP amounts like 1.250.000, day/month dates, formal French (vous), missing accents and mid-conversation language switches.
- An expected outcome per case: the exact answer when one exists; otherwise the criteria and the allowed sources.
Version everything together: dataset, prompt, pinned model identifier (not an alias), effort level and judge. Keep a held-out slice you never tune against, and retire saturated cases: Anthropic warns that an eval everyone passes stops giving signal.
Graders, biases and calibration
Use the cheapest grader that can actually decide.
Exact match handles classification, routing and normalized extraction. Programmatic checks go further than most teams expect: a JSON schema, SQL that must run, unit tests on generated code, a numeric tolerance, a citation that must exist in the retrieved passages. Rubric-based LLM judges cover the rest, and OpenAI's guide prefers pass/fail or pairwise verdicts over numeric scales.
| Bias | Evidence | Mitigation |
|---|---|---|
| Position | With ChatGPT as judge, reordering the answers let Vicuna-13B beat ChatGPT on 66 of 80 queries (Wang et al., 2023) | Judge both orders and average; a verdict that flips with the order counts as a tie |
| Verbosity | Judges favor longer answers (Zheng et al., 2023); controlling for length raised AlpacaEval's correlation with Chatbot Arena from 0.94 to 0.98 (Dubois et al., 2024) | Length limits in the rubric; compare answers of similar length |
| Self-preference | Judges recognize their own outputs and favor them more the better they recognize them (Panickssery et al., 2024) | A judge from a different model family than the system under test |
Calibrating the judge
- Two people label the same 50 to 100 cases independently with the rubric; their agreement is your ceiling.
- Run the judge on those cases and read every disagreement: most trace back to vague criteria, so rewrite them as yes/no questions.
- Trust the judge when its agreement with humans approaches theirs with each other (Zheng et al. measured over 80% for GPT-4, the human level). OpenAI advises against scaling a judge before that.
- Pin the judge version and recalibrate when it changes. Anthropic only commits to Claude Haiku 4.5 until at least October 15, 2026.
Regression gates in CI
An eval that can't block a deployment is a report, not a control.
Treat prompts and configuration as code: every change goes through a pull request that runs the evals, whether it is a prompt edit, a new model snapshot, another effort level, a re-indexed knowledge base or a changed tool schema.
Models change even when your code doesn't. Claude Opus 5.5, released on September 22, 2026, defaults to medium effort, one level below Opus 5: teams that never set effort explicitly got new behavior without touching a line. A gate turns that silent change into a failed build.
- Tier the suite: 30 to 50 critical cases on every pull request, the full set nightly and before each release.
- Set the gate in advance, for example zero failures on critical cases (safety, money, personal data) and no drop of more than two points in the overall pass rate.
- Handle randomness: run critical cases several times and require pass^k.
- Keep transcripts next to scores: a score you can't read can't be debugged.
Tooling, as documented
Choose for your stack; keep the dataset portable.
| Tool | Who, license | Documented focus | Worth knowing |
|---|---|---|---|
| Inspect | UK AI Security Institute with Meridian Labs, MIT | 200+ ready-made evals, agents and multi-agent setups, MCP tools, sandboxes (Docker, Kubernetes, Modal) | METR moved its time-horizon evaluations to it in January 2026 |
| promptfoo | MIT; OpenAI agreed to acquire it in March 2026 | Evals, red teaming and vulnerability scanning | Stays open source; about 25,000 stars |
| OpenAI Evals | OpenAI, open source | Framework and registry of evals | About 19,500 stars; the separate hosted Evals product in AgentKit closes on November 30, 2026 |
| DeepEval | Confident AI, Apache-2.0 | Open-source LLM evaluation framework | About 18,500 stars |
| Ragas | Apache-2.0 | Reference-free RAG metrics: retrieval, faithfulness, answer quality | About 15,900 stars; now at vibrantlabsai/ragas |
| LangSmith | LangChain | Online evaluators on production traces (LLM judge or code), with filters, sampling and backfill | Runs on traces logged in LangSmith |
| Braintrust autoevals | Braintrust, MIT | Open-source scorer library | About 1,000 stars |
Tools matter less than discipline. Keep datasets and rubrics in plain files (JSONL, YAML) in your own repository so you can switch frameworks without rebuilding the golden set. For RAG, add the retrieval metrics from our RAG guide.
What evals cost, and how to sample
Tokens are the small line; people's time is the big one.
An illustrative calculation, not a project measurement: 300 cases (100 each in Spanish, English and French), three trials per case. The system under test is Claude Opus 5.5 ($4 / $20 per million tokens) with 3,000 input and 700 output tokens per case, reasoning included; the judge is GPT-6 Sol ($2 / $10), from another family, reading 2,500 tokens and writing 500.
| Scenario | Volume | Cost |
|---|---|---|
| Full run, standard prices | 300 cases × 3 trials | $32.40 |
| Full run, Batch API | Same | $16.20 |
| Nightly for 30 days, batch | 30 full runs | $486 |
| Smoke set per pull request | 50 cases × 1 trial | $1.80 |
Token spend is modest; the expensive part is the hours people spend labeling, reviewing and fixing rubrics. And the judge's rubric is a stable prefix: OpenAI discounts cached input on GPT-6 by 90%.
Sampling production. LangSmith's documentation shows online evaluators running on a sample, such as 10% of traces. We stratify: a fixed share per route and language, plus every trace with negative feedback, an escalation, a refusal or an unusual length. Code checks can cover 100% of traffic. More levers in the real cost of AI.
Where our 30-prompt method fits, plus a checklist
Choosing a model is one eval; production needs the others.
Every Slash deployment starts with our model-selection method: 30 fixed, versioned prompts per client, five engines per run, in Spanish, English and French, scoring quality, observed failures, cost per inference and p95 latency. Thirty sits within the 20 to 50 cases Anthropic suggests for a first set.
That run decides where to start; it doesn't replace the gate, the calibrated judge or monitoring. We recommend turning those 30 prompts into the seed of the client's golden set.
Checklist for a team with no evals
- Pick one route and write down what a correct answer is.
- Collect 20 to 50 real, anonymized cases, with Spanish, French and hostile variants.
- Write code checks first: format, schema, language, forbidden content, citations.
- Add a pass/fail judge from another model family and calibrate it on 50 human labels.
- Block merges on critical failures and run the full set nightly.
- Add new production failures to the set every week and re-run everything when a vendor ships a model.
No new model, prompt or effort setting reaches production without passing the same versioned set as the version it replaces. If you can't state your current system's pass rate, that is the first number to produce.
Key takeaways
- Reading a few answers is not testing: METR measured developers 19% slower with AI while they believed they were 20% faster.
- Stack five layers: code checks, a versioned golden set, calibrated LLM judges, human review and sampled production monitoring.
- LLM judges have documented position, verbosity and self-preference biases: swap the order, control length and use a judge from another model family.
- Put a regression gate in CI and run it on every prompt, model, effort or index change, vendor releases included.
- A full run costs $16 to $32 in our example; the expensive and valuable part is human time spent on labels and transcripts.
Sources
- Demystifying evals for AI agents
- Evaluation best practices
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Large Language Models are not Fair Evaluators
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- LLM Evaluators Recognize and Favor Their Own Generations
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- Why we no longer evaluate SWE-bench Verified
- State of Agent Engineering
- Models overview
- API pricing
Editorial note: this analysis reflects the public information available on the review date. Models, prices and rules change fast; every third-party figure links to its source, and our opinions are labeled as such. Spotted an error? Write to contact@slash-digital.io.
The questions we hear often
What is an LLM eval?
A repeatable test of an LLM system: a fixed set of inputs, explicit criteria and graders (code, a model or people) that produce a score you can compare between versions of your prompt, model or settings.
How many test cases do I need to start?
Anthropic recommends starting with 20 to 50 tasks drawn from real failures. Start there, make them pass reliably, then grow the set from production traces rather than aiming for thousands of synthetic cases on day one.
Can I trust an LLM as a judge?
Only after calibration. Zheng et al. found GPT-4 agreed with humans over 80% of the time, as often as humans agree with each other, but the same work and later papers document position, verbosity and self-preference biases. Measure your judge against human labels before relying on it.
Which eval tool should we use?
The one that fits your stack: Inspect, promptfoo, DeepEval, Ragas, OpenAI Evals, LangSmith or Braintrust's autoevals are all documented options. Keep datasets and rubrics in your own repository so the choice stays reversible.
Do public benchmarks make our own evals unnecessary?
No. They measure generic tasks, mostly in English, and can be flawed: OpenAI stopped reporting SWE-bench Verified after finding flawed tests in 59.4% of 138 audited hard tasks. Your prompts, languages and failure modes need your own set.
More analysis to read next
How we choose a model
Language, code, agents, cost and privacy: the evaluation we run before every deployment, with 30 real prompts per client.
Agents and engineeringRAG in 2026: long context, hybrid search and agentic search
A million-token window does not retire retrieval. What works in 2026: contextual chunks, hybrid search, rerankers, agentic search, access control and a cost model.
Agents and engineeringAI agents in production: what works in 2026
Workflow or agent, the eight building blocks, adoption and failure data, computer use, design rules, costs, security and a launch checklist.
Where to next
Let's put it in production
Tell us your challenge. We reply within 24 business hours with an honest first read: if we can help, we'll say how; if not, we'll say who can.
I reply personally. No endless forms, no canned replies.