Beyond Vibes: A Practical Guide to Evaluating AI Agents in Production

There's a dirty secret hiding inside most enterprise AI agent deployments: nobody really knows if they're working.

Not "working" in the sense of running without errors. Working in the sense of actually doing the right thing — producing correct outputs, making sound decisions, using tools appropriately, and staying within the boundaries your business requires. Most teams ship agents on a combination of optimism and demo-room confidence. A few impressive runs, a thumbs-up from a stakeholder, and suddenly it's in production handling real customer queries or executing real business workflows.

That's not engineering. That's gambling.

Evaluation — systematic, repeatable, quantitative evaluation — is the discipline that separates agent deployments that earn trust from those that quietly erode it. And it's one of the most underinvested areas in the entire AI agent stack.

This post is a practical guide to doing it properly.

---

Why Evaluating Agents Is Fundamentally Different

Evaluating a traditional software system is relatively straightforward: you define inputs, specify expected outputs, run tests, and check whether assertions pass. The system is deterministic. A function that adds two numbers either returns the right sum or it doesn't.

AI agents break every assumption in that model.

Non-determinism. The same prompt, the same tools, and the same input can produce meaningfully different outputs across runs — especially when temperature is above zero or when the agent is reasoning through an open-ended task. A test that passes once might fail the next time without any change to the code.

Multi-step trajectories. An agent doesn't just produce an output — it executes a sequence of decisions: which tool to call, what to pass to it, how to interpret the result, whether to retry, when to escalate. A wrong decision at step two can cascade into a catastrophically wrong final answer, even if every individual step looks plausible in isolation.

Subjective quality. Many agent tasks don't have a single correct answer. Was the customer support response helpful? Was the research summary accurate enough? Was the tone appropriate? These require evaluation rubrics, not binary assertions.

Tool and environment coupling. Agents interact with external systems — APIs, databases, browsers, email. Evaluation has to account for what happens when those systems are slow, unavailable, or return unexpected data.

Any serious evaluation framework has to grapple with all four of these challenges simultaneously.

---

The Four Dimensions of Agent Quality

Before you can measure agent quality, you need to define what quality means. We find it useful to decompose it into four orthogonal dimensions.

1. Correctness

Did the agent produce the right answer or take the right action? This is the most obvious dimension, but also the hardest to measure for open-ended tasks. Correctness evaluation typically requires either a ground-truth dataset with known correct answers, or an LLM-as-judge approach where a separate model scores the output against a rubric.

2. Reliability

Does the agent behave consistently across repeated runs, edge cases, and adversarial inputs? A correct agent that fails 20% of the time is not a reliable agent. Reliability evaluation involves running the same scenarios many times, stress-testing with unusual inputs, and measuring variance in outcomes.

3. Safety and Compliance

Does the agent stay within its defined boundaries? Does it refuse inappropriate requests? Does it avoid leaking sensitive data, making unauthorised tool calls, or taking irreversible actions without confirmation? Safety evaluation is non-negotiable for any agent operating in a regulated industry or touching sensitive systems.

4. Efficiency

Is the agent achieving its goals at a reasonable cost in tokens, latency, and tool calls? An agent that takes 47 LLM calls to answer a question that should require three is not just expensive — it's a signal that the reasoning architecture is broken. Efficiency evaluation catches bloated pipelines before they become budget problems.

---

Building Your Evaluation Dataset

Every evaluation framework starts with data. You need a corpus of test cases that reflects the real distribution of tasks your agent will encounter in production — not just the easy cases that make demos look good.

Golden datasets are curated collections of inputs paired with verified correct outputs or trajectories. Building them is labour-intensive, but they're the foundation of reproducible evaluation. Start with 50–100 cases covering your core use cases, common edge cases, and known failure modes. Grow the dataset continuously as new failure patterns emerge in production.

Trajectory datasets go further: instead of just capturing the final output, they record the full sequence of agent actions — every tool call, every intermediate reasoning step, every decision point. Trajectory evaluation lets you catch agents that arrive at the right answer via the wrong path, which is a reliability risk even if it doesn't show up in output-level metrics.

Adversarial datasets are specifically designed to break your agent. Include prompt injection attempts, malformed inputs, requests that probe the edges of the agent's permission boundaries, and scenarios designed to trigger hallucination. If your agent can't handle these gracefully in evaluation, it will encounter them in production.

Synthetic data generation using LLMs can dramatically accelerate dataset construction, but synthetic data should always be reviewed and filtered by humans before being used as ground truth. LLMs generate plausible-sounding test cases, not necessarily correct ones.

---

Evaluation Methodologies

Unit Testing for Agent Components

Not everything needs end-to-end evaluation. Individual components of an agent pipeline — tool call parsers, retrieval modules, prompt templates, output formatters — can and should be unit tested like any other software. Isolate components, mock dependencies, and verify behaviour deterministically. This catches a large class of bugs before they ever reach the agent runtime.

End-to-End Scenario Testing

Run complete agent workflows against your golden dataset and measure outcomes. Because agents are non-deterministic, run each scenario multiple times (at least five to ten runs) and report pass rates rather than binary pass/fail. A scenario that passes 9/10 times is meaningfully different from one that passes 10/10 — and both are meaningfully different from 6/10.

LLM-as-Judge Evaluation

For tasks where correctness is subjective, use a separate, high-capability LLM as an automated evaluator. Provide the judge with the original task, the agent's output, and a detailed scoring rubric. LLM-as-judge evaluation scales to thousands of test cases without human review, though it introduces its own biases — particularly a tendency to prefer verbose, confident-sounding outputs regardless of accuracy. Calibrate your judge against human ratings regularly.

Human Evaluation

For high-stakes workflows, there is no substitute for human review. Structured human evaluation — where reviewers score agent outputs against a consistent rubric — is slower and more expensive than automated methods, but it catches the subtle failures that automated evaluators miss: tone issues, cultural insensitivity, reasoning that is technically correct but practically unhelpful. Build human evaluation into your release process for any agent that touches customer-facing or compliance-sensitive workflows.

Regression Testing

Every time you change a prompt, swap a model, add a tool, or modify the agent's reasoning architecture, run your full evaluation suite. Regression testing ensures that improvements in one area don't silently degrade performance in another. This is especially important for multi-agent systems where a change to one agent can have unexpected downstream effects on others.

---

Metrics That Actually Matter

Avoid vanity metrics. "The agent answered correctly" is not a metric — it's an anecdote. Here are the measurements that give you real signal:

• Task completion rate: Percentage of scenarios where the agent successfully completes the assigned task without human intervention.
• Correctness score: For scored evaluations, the mean score across your test dataset, broken down by task type and difficulty tier.
• Trajectory efficiency: Average number of LLM calls and tool invocations per completed task. Compare against a human baseline or a hand-crafted optimal trajectory.
• Failure mode distribution: Categorise failures by type — hallucination, wrong tool selection, infinite loops, out-of-scope actions, refusals. The distribution tells you where to invest engineering effort.
• P95 latency: The 95th percentile end-to-end task completion time. Averages hide the long tail of slow runs that destroy user experience.
• Cost per task: Total token spend divided by completed tasks. Track this over time and across model configurations.
• Safety violation rate: Number of scenarios where the agent took a prohibited action or produced a non-compliant output, expressed as a percentage of total runs.

---

Continuous Evaluation in Production

Pre-deployment evaluation is necessary but not sufficient. Agent behaviour in production diverges from evaluation benchmarks for reasons that are hard to anticipate: real user inputs are messier than synthetic ones, external tool APIs change, model providers push silent updates, and edge cases accumulate in ways that no golden dataset fully captures.

Continuous evaluation means treating production as an ongoing evaluation environment.

Shadow mode testing. Run a new agent version in parallel with the production version, comparing outputs without serving the new version to users. Flag divergences for review before promoting the new version.

Sampling and scoring. Automatically sample a percentage of production interactions and score them using your LLM-as-judge pipeline. Track scores over time and alert on degradation.

User feedback signals. Thumbs up/down ratings, escalations to human agents, and task abandonment rates are all weak but real signals of agent quality. Correlate them with your automated evaluation scores to calibrate your judges.

Canary deployments. Roll out agent changes to a small percentage of traffic first, monitor quality metrics, and expand rollout only when metrics hold.

---

How Mindra Supports Evaluation at Scale

Mindra's orchestration platform is built with evaluation as a first-class concern, not an afterthought.

Every agent execution on Mindra produces a full structured trace — every tool call, every model response, every decision branch, every token consumed. These traces are the raw material for evaluation: you can replay them, score them, and diff them across versions without re-running live workflows.

Mindra's pipeline versioning means every prompt change, model swap, or tool update is tracked as a discrete version. Running your evaluation suite against a new version before promoting it to production is a one-click operation, with results surfaced directly in the dashboard alongside historical benchmarks.

For teams building evaluation pipelines programmatically, Mindra exposes execution traces via API, making it straightforward to pipe data into external evaluation frameworks or custom scoring infrastructure.

And because Mindra's orchestration layer manages the full agent lifecycle — from initial task intake through multi-step reasoning to final output — evaluation can happen at any granularity: individual tool calls, reasoning steps, full task trajectories, or aggregated workflow outcomes.

---

Getting Started: A Practical Checklist

If you're starting from zero, here's a pragmatic sequence:

1. Define your success criteria before writing a single test. What does "working correctly" mean for your specific agent and use case? Write it down.
2. Build a golden dataset of 50 representative scenarios, including at least 10 edge cases and 5 adversarial inputs.
3. Instrument your agent to capture full execution traces on every run.
4. Implement LLM-as-judge scoring for subjective quality dimensions, calibrated against at least 20 human-rated examples.
5. Run your evaluation suite on every code change, model update, and prompt revision. Automate it as a CI step.
6. Set up production sampling to score a percentage of live interactions continuously.
7. Review failure distributions weekly and use them to prioritise the next round of improvements.

Evaluation is not a one-time activity. It's an ongoing engineering practice — and the teams that invest in it early are the ones whose agents actually get better over time instead of quietly getting worse.

---

The move from "it seemed to work in the demo" to "we can prove it works in production" is one of the most important maturity jumps an AI engineering team can make. It's also one of the most neglected.

Don't ship on vibes. Evaluate.