Key takeaways
- Write measurable success criteria before building the test set, and get the process owner to sign them off.
- Build the test set from real past cases, including awkward and adversarial ones, not only clean examples.
- Grade with code wherever the answer can be checked mechanically, and check any LLM judge against human ratings.
- Run each test case more than once, because the same input can produce different outputs.
- Treat the test set as part of the product and re-run it after every prompt, model, tool or data change.
Part of our guide: Practical AI for business operations: where it pays off first
What does it mean to evaluate an LLM application?
To evaluate an LLM application is to measure, on cases where you already know the right answer, how often it does what it should and how badly it fails when it does not. It is closer to a driving test than a demo: the examiner chooses the route, including the difficult junctions, and the pass mark is fixed in advance.
You evaluate the whole application, not just the model. The same model can do well with one prompt, retrieval setup and tool design and badly with another. The result also depends on your data, so public benchmark scores tell you little about how the system will behave on your documents, customers and questions.
This article is part of our guide to practical AI for business operations. If you have not chosen a use case yet, start with how to pick your first AI automation project.
How do you define success criteria?
Define success in terms the business cares about and the test can measure. "Answers questions well" is not testable. "Gives the correct order status for at least the agreed share of questions, and never reveals another customer's data" is.
Anthropic's guide to defining success criteria and building evaluations (opens in a new tab) recommends criteria that are specific and measurable, and test cases that mirror your real task mix, including edge cases. In practice, write one line per criterion with its measure and threshold, and have the process owner approve it before the first test run, so nobody moves the goalposts after seeing the results.
- Task quality: is the answer, extraction or classification correct?
- Grounding: is every claim supported by the sources or records the system retrieved?
- Safety: does it refuse or escalate what it should, and avoid actions it must not take?
- Privacy: does it keep each user to the data they are allowed to see?
- Experience: is the tone right, and is the response time acceptable?
- Cost: is the model usage per task within the budget the business accepts?
How do you build an LLM evaluation dataset?
Build it from real cases: past tickets, emails, documents or questions, with the correct answer written by someone who knows the work. Real cases carry the spelling mistakes, half-finished sentences and odd layouts that synthetic examples miss.
Cover the mix you expect in production, then deliberately add the hard parts: rare categories, ambiguous requests, questions with no answer in your data, questions about records the user should not see, long and rambling inputs, and inputs in each language you support. Add adversarial cases too, such as a document that contains instructions to the model. Keep a small held-back set that you do not use while tuning prompts, so the final score is not flattered by overfitting.
Size depends on the decision. A few dozen cases are enough to spot obvious problems during development. To claim a low error rate before launch, you need many more, because zero failures in a small sample still leaves a wide range of possible error rates. Our article on LLM document extraction accuracy works through that arithmetic for documents.
Which LLM evaluation metrics should you use?
Choose metrics by what the application does. A classifier, an extractor, a document assistant and an agent fail in different ways, so they need different measures. Keep the list short: a handful of metrics that map to your success criteria beats a long dashboard nobody reads.
| Application | Core metrics | Typical failure to test |
|---|---|---|
| Classification or routing | Accuracy, per-class precision and recall | Rare classes routed to the common one |
| Data extraction | Field accuracy, whole-document accuracy | Plausible but wrong numbers |
| RAG assistant | Answer correctness, grounding, retrieval hit rate | Confident answers not in the sources |
| Drafting replies | Rubric scores for accuracy, tone, policy | Promises the business cannot keep |
| Agent with tools | Right tool, right parameters, task completed | Unapproved or repeated actions |
Code, LLM as a judge or human grading?
Use code wherever the answer can be checked mechanically, an LLM judge for qualities that need reading, and people for setting the standard and checking the judge. Each has a place, and most evaluations combine all three.
Code-based grading covers exact matches, numeric comparisons, schema checks and whether the right tool was called with the right parameters. It is fast, cheap and repeatable. An LLM judge grades against a written rubric, such as whether a reply is supported by the retrieved policy and free of commitments. It scales well, but it can be biased or inconsistent, so have people grade a sample of the same outputs and compare. If the judge and the people disagree often, fix the rubric before trusting the scores. Human grading is the slowest, but it is the reference everything else is calibrated against.
Remember that LLM outputs vary between runs. Run each case more than once, at the settings you will use in production, and look at the spread as well as the average. A system that passes a case two runs out of three has a reliability problem the average hides.
How do you red team an LLM application?
Red teaming means trying to make the system fail on purpose, using the risks that apply to your design. The OWASP Top 10 for LLM Applications (opens in a new tab) is a practical checklist to work from, because each item suggests concrete tests.
- Prompt injection: put instructions inside documents, emails or records the system reads, and check they are ignored.
- Sensitive information disclosure: ask for other users' data, internal prices or personal details, and check the system declines.
- System prompt leakage: ask the system to reveal its instructions, and check nothing secret is in them anyway.
- Excessive agency: ask an agent to take actions outside its role, and check the tool layer refuses.
- Improper output handling: check that outputs passed to other systems are validated, not executed blindly.
- Unbounded consumption: send very long inputs or loops, and check limits stop runaway cost.
- Misinformation: ask questions with no answer in your data, and check the system says so instead of inventing one.
How do you decide whether to launch?
Launch when every criterion meets its agreed threshold on the held-back set, no critical safety test fails, and the operational checks pass. Operational checks are easy to forget: response time under load, behavior when the AI provider is slow or down, cost per task at expected volume, and a working way for users to report a bad answer.
Make the go-live decision in a short written review with the process owner. If a criterion narrowly misses, the usual options are to keep human review on that part, narrow the scope, or fix and re-test. The NIST AI Risk Management Framework describes this kind of measured, documented decision under its Measure and Manage functions, which is useful language when auditors or clients ask how the system was approved.
What is LLM regression testing after launch?
LLM regression testing means re-running your full test set whenever anything changes, to confirm that a fix in one place has not broken another. Changes include prompt edits, a new model version, new tools, changes to the documents behind a RAG assistant, and upgrades to the systems it connects to, such as a new Odoo version for an Odoo AI assistant.
Between changes, monitor live behavior: sample real conversations or outputs for review, track user reports and corrections, and watch cost and response time. Add every real failure to the test set so the same mistake is caught next time. Over time, that growing set becomes the most valuable part of the system, because it lets you change models or providers with evidence rather than hope.
When we evaluate an LLM application at Digihub, the test set is written with the client's staff before the prompts and handed over with the system. Our LLM integration and AI agents services include this evaluation work, and our software testing and QA team covers the surrounding application. Fixed packages are listed on our AI automation pricing page.
Sources and further reading
Official documentation and guidance if you want to read more about this topic.
Related guides and services
Questions readers ask
How do you evaluate an LLM application?
Write measurable success criteria, collect real cases with confirmed answers, and score the application on them using code checks where possible, a rubric-based LLM judge for qualities that need reading, and human review to calibrate. Add safety tests based on the OWASP LLM risks and operational checks for cost and outages. Agree the pass thresholds before you see the results.
Is using an LLM as a judge reliable?
It can be useful at scale, but only after you check it. Give the judge a clear rubric, have people grade a sample of the same outputs, and compare the two. Where they disagree often, improve the rubric or keep human grading for that criterion. Never use an LLM judge as the only check for safety-critical behavior.
How is RAG evaluation different?
RAG evaluation checks two stages: whether retrieval found the right sources, and whether the answer is correct and supported by those sources. A wrong answer can come from either stage, so measure both. Include questions whose answer is not in your documents, and check that the assistant says it does not know rather than inventing an answer.
How many test cases does an LLM evaluation dataset need?
Enough to cover your real mix of inputs and every important edge case, and enough to support the claim you want to make. A few dozen cases catch obvious problems during development. Showing a low error rate before launch takes many more, because a handful of passes cannot rule out a meaningful failure rate. Grow the set with every real failure.
Do we need to re-test when the AI provider updates its model?
Yes. A new model version can change behavior on your cases even if general benchmarks improve. Pin the model version where the provider allows it, re-run your full test set on the new version before switching, and compare results criterion by criterion. Switch only when the new version meets every threshold your current one does.