How to build an evaluation framework for production AI

An AI evaluation framework is the evidence system behind a production decision. It defines what good looks like, assembles representative tests, combines automated and human assessment, and sets the thresholds that determine whether a system can be released.

It is not a single accuracy score. Production AI can fail through an incorrect answer, a missing fact, an unsafe action, an unusable interface, an unexpected cost or a workflow that users avoid. The framework needs to reflect the task and its consequences.

Evaluate the system, not just the model

A production application usually combines instructions, a model, retrieved information, business rules, user input, tools, integrations and human review. Model benchmarks can help compare general capabilities, but they do not test that whole system.

Evaluate at three levels:

1. Component level: retrieval, classification, extraction, tool selection and model output.
2. System level: the complete result produced from a realistic input, including citations, rules and connected tools.
3. Workflow level: the effect on user performance, operational outcomes and risk.

For example, a complaint-summary system may retrieve the correct documents and produce fluent text. It can still fail if it omits the customer’s requested resolution or causes the handler to spend longer checking the case.

The NIST AI Resource Center places testing, evaluation, verification and validation within a broader risk-management process. That is the right mindset: evaluation supports a decision in context, not a leaderboard.

Step 1: write the evaluation contract

Begin with a one-page contract for the use case. It should specify:

• The intended users and eligible cases.
• The decision or task the system supports.
• Inputs the system may use.
• Outputs and actions it may produce.
• Actions it must never take.
• Material harms and failure modes.
• Human review and fallback.
• The business outcome being pursued.

This prevents a common problem: the system is tested as a writing assistant and then used as a decision tool.

Step 2: turn risks and needs into criteria

Each criterion should answer a specific question and have a method for judging it.

Task success

Does the output complete the assigned task? For extraction, this may be field-level precision and recall. For drafting, it may be coverage of required content and adherence to the requested format.

Groundedness

Are factual statements supported by the authorised source material? A fluent but unsupported claim should fail even when it appears plausible.

Completeness

Are all material items present? For operational summaries, omission can matter more than writing style.

Instruction and policy compliance

Does the system follow business rules, permissions and required wording? Test conflicting instructions and attempts to bypass controls.

Safety and fairness

Could the output cause harm, reveal sensitive data or create systematically different performance for important groups? Choose measures relevant to the use case rather than applying a generic fairness score.

Human usability

Can a user understand, check and act on the result? Measure review time, correction effort and confidence, not satisfaction alone.

Operational performance

Does the system meet latency, availability and cost requirements at expected volume? An output that arrives too late has failed the workflow.

The Google Cloud generative AI evaluation framework similarly encourages teams to tailor evaluation to the use case and consider quality, workflow efficiency and risk together.

Step 3: build a representative evaluation set

The test set should mirror the production distribution and deliberately over-sample important failure modes.

Use several sources:

• A random sample of normal historical cases.
• High-volume case types.
• Difficult and ambiguous cases.
• Rare cases with severe consequences.
• Examples from known incidents and complaints.
• Adversarial tests and malformed inputs.
• Synthetic cases created to fill a justified gap.

Create metadata for each example, such as product, channel, document type, complexity, language and risk tier. This lets you inspect performance by segment instead of trusting an aggregate.

Keep a protected holdout set that developers do not repeatedly tune against. Otherwise evaluation performance can improve while real-world generalisation does not.

The AWS quality-evaluation guidance recommends a human-curated gold-standard dataset and evaluation across multiple facets. A gold standard is not automatically permanent. Update it when policy, data or the workflow changes.

Step 4: define reference answers carefully

Some tasks have one correct label. Generative tasks often allow several good answers. Use the appropriate reference type:

• Exact reference: a known field, category or numerical result.
• Required facts: a checklist of points that must appear.
• Rubric: criteria and scoring anchors for acceptable output.
• Pairwise preference: which of two outputs better supports the task.
• Prohibited content: claims, actions or disclosures that must not appear.

Ask subject-matter experts to resolve disagreements. Track inter-rater agreement. If qualified reviewers cannot agree on a reference, the task or rubric may be underspecified.

Step 5: combine evaluation methods

No single method is sufficient.

Deterministic checks

Use code for schema validation, required fields, allowed values, citation existence, numerical consistency and prohibited patterns. These checks are repeatable and cheap.

Model-based graders

A model judge can score large volumes against a rubric or compare variants. Validate it against expert judgements before trusting it. Check performance on ambiguous cases and by segment. Do not ask a judge model to decide whether its own unsupported factual claim is true without providing evidence.

Expert review

Use subject-matter experts for high-consequence correctness, policy interpretation and release decisions. Calibrate reviewers with examples and scoring anchors.

User testing

Observe real users performing the task. Measure time, corrections, misuse and decision quality. Users can reveal failures that output-only evaluation misses.

Red-team and abuse testing

Test hostile instructions, data leakage, inappropriate tool use, unusual inputs and attempts to escape the approved scope.

The mix should reflect risk. A low-impact internal drafting aid may need lighter expert sampling than a customer-affecting recommendation.

Step 6: set release thresholds before looking at the result

Define thresholds in advance to reduce pressure to reinterpret a disappointing score.

Use at least three types:

• Minimum overall performance: the normal quality bar.
• Segment floors: the lowest acceptable performance for important case types or groups.
• Critical failure tolerance: often zero or near zero for specific severe failures.

Add operational thresholds for latency, cost, fallback rate and review effort. A system should not pass because strong writing quality hides an unaffordable or unusable workflow.

Document the action for each threshold breach: block release, reduce scope, require review, investigate or monitor.

Step 7: run controlled workflow trials

Offline tests are necessary but cannot prove operational value. Run a controlled trial with eligible cases and authorised users.

Compare against the current process or another credible counterfactual. Measure:

• Task time including checking and correction.
• Output acceptance and material edit rate.
• End-to-end process time.
• Error and escalation rate.
• Appropriate adoption and non-use reasons.
• User understanding of limitations.
• The intended business outcome.

Separate model errors from data, integration and workflow errors. The owner and remedy may be different.

Step 8: create an evaluation report that supports a decision

The report should be concise enough for decision-makers and detailed enough to audit.

Include:

1. Approved use, users and exclusions.
2. Evaluation-set composition and limitations.
3. Criteria, methods and thresholds.
4. Results overall and by important segment.
5. Critical failures with examples.
6. Workflow-trial results.
7. Residual risks and controls.
8. Recommendation: go, refine or stop.
9. Monitoring plan and next review date.

Do not reduce the conclusion to an average score. Explain what can still go wrong and why the remaining risk is acceptable for the proposed scope.

Step 9: make evaluation continuous

Production creates new evidence. Capture sampled outputs, user corrections, overrides, complaints, incidents, cost and latency. Protect personal data and apply appropriate access controls and retention.

Add difficult production cases to a regression set after they have been reviewed. Re-run critical tests when any of these change:

• Model or model version.
• System instructions or prompts.
• Retrieval source or embedding method.
• Business policy or approved scope.
• Connected tools and permissions.
• User interface or review process.

Version the evaluation set, rubric, system and result together. Without that lineage, a passing score cannot be tied to the system in production.

A minimum viable evaluation pack

A small organisation does not need an enormous platform to begin. A defensible first pack can contain:

• A one-page evaluation contract.
• A versioned set of representative examples.
• Required facts and prohibited failures for each example.
• A scoring rubric with reviewer guidance.
• A simple results log by criterion and segment.
• A release threshold and signed decision record.
• A production sampling and incident process.

Start manually where judgement is valuable. Automate stable checks as volume grows. The objective is trustworthy evidence, not an elaborate evaluation stack.

The key design principle

An evaluation is useful only when it changes a decision. Every metric should connect to a release gate, control, workflow improvement or monitoring action.

That makes evaluation more than model testing. It becomes the operating discipline that allows AI to improve without silently moving outside its approved purpose.

Explore Sorsana’s AI evaluation and governance approach, or talk to us about building your evaluation framework.