How to Check Whether Your AI Agent Really Resolved Conversations
Many AI agents count a conversation as resolved when the customer stops replying. Here is how to sample, score and report what really happened.
To check whether your AI agent's resolved conversations were really resolved, take a random sample of the conversations it marked resolved, score each one against a written rubric (truly resolved, partly resolved, deflected, abandoned, wrong answer) using the transcript and any follow-up contact from the same customer, and report the audited resolution rate with its margin of error next to the rate the vendor reports. A sample of 200 to 400 conversations a month is enough to measure the overall rate to within about four to seven points.
The steps below cover how to draw the sample, how to score it, which signals to use, and how to report the result.
Why the agent's own count needs checking
Many AI agents and bot platforms mark a conversation resolved when the customer stops replying, confirms, or reaches the end of a flow. That captures the good outcomes and some bad ones: a customer who gave up, or one who got a confident wrong answer, also stops replying. If you pay per resolution or per outcome, the same definition sets the bill. Our post on containment, resolution and deflection rates compares the definitions, including how one vendor words its billing unit.
Step 1: draw a random sample of "resolved" conversations
Start from everything the agent marked resolved in the period. Not the ones someone flagged, and not the latest ones only. A random draw is what lets you generalize from the sample.
Choose the sample size from the precision you need. The chart below shows the 95% margin of error for a measured rate at different sample sizes.
Stratify by topic if you want per-topic rates. A 300-conversation sample spread across 10 topics gives about 30 per topic, which is too few for a reliable rate on any single topic. Sample each important topic separately, or audit every conversation with a model and use the hand-read sample to check it.
Hide the agent's label from the reviewer. Knowing the system's label can sway a reviewer.
Sample size
Under about 200 conversations, the audited rate can be off by seven points or more
Margin of error at 95% confidence, in percentage points, for a measured rate near 50% and near 80%, by number of conversations audited.
True rate near 50%True rate near 80%
Step 2: score each conversation against the rubric
The rubric
Five outcomes for a conversation the agent marked resolved
Score each sampled conversation as exactly one of these, using the transcript and what happened next.
| Outcome | Definition | What to look for |
|---|---|---|
| Truly resolved | The customer got a correct, complete answer or the action was done | Customer confirms, or the last turn is a thanks; no repeat contact on the same issue within seven days |
| Partly resolved | Some of what the customer asked for was handled, not all | A second question left unanswered; "ok, and what about..." with no reply |
| Deflected | The agent sent the customer elsewhere without solving it | A link, phone number or "contact us" with no answer; the same question arrives by email or phone |
| Abandoned | The customer left before getting an answer | Last message is the customer's, or the customer goes quiet after a fallback or a loop |
| Wrong answer | The agent answered with something incorrect or against policy | Answer conflicts with the help center or policy; a refund, complaint or correction follows |
Write a short definition and two examples for each outcome before you start, and have two people score the same 40 or 50 conversations. Where they disagree, tighten the definition. When the rubric is stable, a language model given the same rubric can score every resolved conversation, not just the sample, and the hand-scored sample becomes the check on the model.
Step 3: use signals from the transcript and from what happened next
The transcript shows most of it:
Who spoke last. A conversation that ends on the customer's question was not resolved, whatever the status says.
Repeated questions or loops. The customer asking the same thing twice, or the agent giving the same answer twice.
Requests for a person. "Agent", "human", "speak to someone", even if no handoff happened.
The answer itself. Check it against your help center and policies. This is the only way to catch a wrong answer the customer accepted.
Follow-up contacts catch what the transcript cannot. Match each sampled customer on a shared ID across your other channels and look for, within seven days:
a new chat, ticket, email or call on the same issue;
a reopened conversation;
a refund, chargeback, cancellation or complaint;
a bad survey score or a review that mentions the bot.
A conversation with a repeat contact on the same issue is not truly resolved, even if the transcript looked fine. We use seven days; pick yours and keep it fixed so months compare.
Step 4: report the audited rate next to the reported one
Report four things each month:
The audited resolution rate with its range. For example, if 142 of 200 sampled "resolved" conversations were truly resolved, the audited rate is 71.0%, with a 95% range of about 64% to 77%.
The mix of the rest. How many were partly resolved, deflected, abandoned or wrong. Wrong answers usually deserve their own line, because they carry the most risk.
Rates by topic. Where the agent's resolutions hold up and where they do not, with a range on each so small topics are not over-read.
The gap to the vendor's figure. Multiply the number of conversations the vendor counted as resolved by the audited share to estimate how many were truly resolved. Compare that with what you are billed for.
Illustrative example
142 of 200 truly resolved is 71.0%, and the honest report is 64% to 77%
The same 71.0% audited rate at four sample sizes, with its 95% range (Wilson interval). The rust row is the example in the text.
Our sample chatbot report, run on synthetic data for a fictional cinema chain, shows what the per-topic view looks like. Across 4,961 customer needs in 3,901 conversations, 43.5% were not met; refunds and exchanges failed at 71.5% and showtimes at 58.8%, both significantly above average.
Sample report, synthetic data
Refunds and exchanges needs fail at 71.5%, against 43.5% across all needs
Share of each topic's customer needs that were not met (partly answered counts half), drawn as the distance from the 43.5% average across all needs. Rust: significantly worse than average. Dark: significantly better. Grey: within chance.
Ranked by share of all unmet needs, three topics (showtimes, refunds and exchanges, and membership) held 70.6% of them, which is where to start.
Sample report, synthetic data
Where the unresolved needs are: three topics hold 70.6% of them
Each topic's share of the needs the bot did not meet, largest first. Dark bars: the three topics to fix first.
Product loop
The report and the deck, built by the agent
Once every conversation is scored, the monthly audit report and slides are generated from the same data.
FAQ
How many AI agent conversations should I audit each month?
Two hundred to four hundred gives a margin of about five to seven points on the overall rate. For per-topic rates, sample each topic separately or have a model score every conversation and audit a sample of the model's scores.
What is the difference between deflected and resolved?
A resolved conversation solved the problem. A deflected one sent the customer somewhere else, such as a help article, form or phone number, without solving it. Both can end without a handoff, which is why they need separating.
Can a model do the audit?
Yes, if it gets a fixed rubric and you check it. Score a sample by hand, compare with the model's scores, and fix the rubric where they disagree. The model can then score every resolved conversation, which gives per-topic rates the hand sample is too small for. We describe the same checks for support tickets in how to analyze Zendesk tickets with AI.
Dimension Labs reads every AI agent conversation, scores how each one really ended against your rubric, matches follow-up contacts across channels, and reports audited resolution by topic, with ranges (confidence intervals on the Prove plan and up), on a schedule. See AI agent analytics and how to measure why customers escalate, or start with one file of transcripts. Plans start at $299 a month, with no demo or sales call needed.