Agent Evals and Guardrails
How I think about improving production reliability for AI agents with evals, policy guardrails, and escalation paths.
I used to think of evals and guardrails as the platform layer around an AI feature. I do not think that anymore.
Once an agent can affect a customer, a workflow, or a system of record, these things are part of the feature itself. "The model is usually right" is not enough. The team needs a way to check expected behavior before release and a way to limit unsafe behavior while the system is running.
The split
Evals ask: did the agent do the right thing?
Guardrails ask: is the agent allowed to do this thing here?
One without the other creates failure modes.
Evals without guardrails can tell you later that the agent failed. Guardrails without evals can block obvious bad actions while still allowing poor decisions. Most useful systems need both loops.
What I want evals to cover
I want evals for normal tasks, edge cases, prompt-injection style cases, and regressions from real incidents.
If a scenario can break in production, it should exist in evals.
How I split eval cases
I usually split cases into four buckets:
| Bucket | What I expect |
|---|---|
| Golden path | complete the task with little ambiguity |
| Ambiguous path | ask for clarification instead of guessing |
| Unsafe path | refuse, escalate, or request approval |
| Regression path | avoid failures already seen in logs, tickets, or incident reviews |
This keeps the eval set from becoming a demo script. It also makes failures easier to discuss: was this a capability issue, an instruction issue, a policy issue, or missing context?
Guardrails I care about
The usual list is not fancy: tool allowlists, scoped data access, restrictions on destination and action, approval gates for high-risk actions, and an audit trail.
Example: Calendar Agent
A calendar agent that reads availability is low risk. A calendar agent that books external meetings, edits invitees, or sends follow-up notes has a different risk profile.
For that workflow, I would split actions by risk:
| Action | Guardrail |
|---|---|
| Read free/busy slots | allowed with scoped calendar access |
| Draft a meeting proposal | allowed, but mark as draft |
| Invite external attendees | require confirmation |
| Cancel or move a customer meeting | require explicit approval and audit log |
The goal is not to slow every action down. It is to make the risky boundary visible.
A runtime shape I like
- Pre-check policy layer
- Agent execution layer
- Post-check validation layer
- Human escalation layer
- Feedback to eval set
Reliability improves when production failures become test cases.
Signals I watch after launch
After launch, I watch refusal rate by task type, approval rate by action type, tool-call error rate, rollback or correction count, new eval cases created from real failures, and whether escalations carry enough context for a human to act quickly.
If none of these are measured, the team is guessing. That may be fine for a prototype. It is too weak for an agent that can take real actions.
A bad order of operations
The bad version is: build a powerful agent first, then add evals only when someone asks, "How do we know this is safe?"
By then, the team has usually mixed prompts, tools, and product flows in ways that are hard to test separately.
I prefer starting with a small eval set before the first demo. Even 20 concrete cases force useful design choices: what the agent may do, what it must refuse, what it should ask a human, and what success actually means.
My default now: if an agent can act in production, the eval and policy work ships with the feature. It is not a cleanup project for after the first scary failure.