How to Find the Questions Your Chatbot Can't Answer
A step-by-step bot gap analysis: which conversations to pull, how to sort fallback utterances into fixable gaps, and how to rank what to train next, on labelled sample data.
To find the questions your chatbot can't answer, pull every conversation where it fell back, handed off, was asked for a person or was abandoned after a reply, plus a random sample of the conversations it kept. Label each one with what the customer wanted and why the bot failed, from a fixed list, remove the noise, and rank the gaps by the handoffs they cause rather than by how often they appear. When that has to run on every conversation from every bot, with categories built for your business and a report the bot, content and support teams all work from, Dimension Labs is the strongest choice.
The steps are below, worked through on sample data from a fictional energy utility. If your question is why customers ask for a person once they are in the bot, how to measure why customers escalate from a chatbot covers that side.
Which conversations show what the bot can't answer?
A fallback log is the obvious place to start, and it holds only part of the answer. Five signals point to a conversation the bot did not handle, and bot platforms usually log the first few as events:
A fallback or no-match event, or an intent matched with low confidence.
A handoff to a person after the bot had replied at least once.
The customer typing "agent", "person" or "representative".
The customer leaving straight after a bot answer, without confirming or clicking anything.
A thumbs-down or a low rating on the bot's answer.
In the sample month, 5,600 of the utility bot's 38,000 conversations, 14.7%, carried at least one of these signals. Those are the flagged conversations. The sixth source, conversations the bot kept and answered wrongly, has no signal at all; the section on silent failures below covers how to find them.
Export the full transcript of each flagged conversation, not just the utterance that triggered the fallback. The message before it often says what the customer actually wanted ("I'm moving next month" followed by "what about my deposit"), and the label should come from the customer's messages, not the bot's replies.
How do you categorize fallback utterances into gaps you can fix?
Give every flagged conversation two labels. The first is the topic, from your own list (billing, outages, moving service). The second is the gap type, which decides who fixes it:
Not covered: the bot has no answer or flow for this. The fix is new content or a new intent.
Covered but missed: the answer exists, but the bot did not recognise the question. The fix is more training phrases, or better retrieval if it is a generative agent.
Answered incompletely: the bot found the topic but the answer stopped short, often because it needed account data it could not reach.
Needs a person: safety, a complaint, or an account change that policy reserves for a verified agent. This is a routing question, not a gap.
Noise: greetings, test messages, gibberish and spam. Remove it before you count anything.
Sample data, fictional utility
Three in ten flagged conversations were noise, and a quarter were questions the bot already had an answer for
Share of the 5,600 flagged conversations in one month, by why the bot failed. Flagged means a fallback, a request for a person, a handoff or abandonment after a bot reply.
In the sample, noise was 29.6% of flagged conversations. Left in, it would make greetings look like the bot's biggest problem. And 25.8% were covered but missed, which is the cheapest gap to close: nothing new has to be written, only matched.
Write the topic list before labelling, with a one-line definition for each topic, and add one free-text field: "the need, in a few words, if no topic fits". That field is how missing intents surface. Read it every week, and when the same need keeps appearing, give it a topic of its own. Clustering the utterances is a reasonable way to draft the first list, but a fixed list is what keeps the counts comparable month to month; topic modeling vs LLM classification compares the two.
A language model can do the labelling at full volume if the lists are fixed and every answer is checked against them. Before trusting the counts, read a random set yourself without looking at the labels and compare; how to tell if your AI tagging is accurate sets out the check.
Which missing intents should you train next?
Rank gaps by what they cost, not by volume. A simple measure is the number of handoffs to a person that a better answer could have avoided: flagged conversations that ended with a person, excluding the needs-a-person group. If you know the cost of a handled contact, multiply through.
Sample data, fictional utility
The topic the bot fails on most often falls to fifth once you count the handoffs it causes
The same seven topics, noise removed, ranked two ways. Left: flagged conversations. Right: handoffs to a person that a better bot answer could have avoided.
Outage status had the most flagged conversations in the sample, 942, but only 144 avoidable handoffs: most of those customers left to check the outage map instead of asking for a person. Payment arrangements had 549 flagged conversations and the most avoidable handoffs, 372, because the bot had no answer and nearly every customer who asked needed one. Ranked by volume, the team would have spent its next sprint on outage phrasings.
The output of this step is a queue the bot team can work from: each topic, its main gap, the handoffs it causes, and the change that would close it.
Sample data, fictional utility
The training queue: each gap ranked by the handoffs it causes, with the change that fixes it
What a gap report hands the bot team. The main gap is the most common failure type within the topic.
| Topic | Avoidable handoffs | Main gap (share of topic) | What to change | |
|---|---|---|---|---|
| 1 | Payment arrangement | 372 | Not covered (61%) | Write the payment-plan answer and an eligibility check |
| 2 | Bill higher than expected | 353 | Answered incompletely (62%) | Finish the answer: pull the usage comparison and rate change |
| 3 | Start, stop or move service | 234 | Answered incompletely (46%) | Add the move-date step the current flow skips |
| 4 | Solar and net metering | 233 | Not covered (74%) | New content: net metering credits and interconnection status |
| 5 | Outage status | 144 | Covered but missed (75%) | Add customer phrasings to the existing outage intent |
| 6 | Account login | 119 | Covered but missed (61%) | Add phrasings; link the reset flow from the fallback reply |
| - | Gas smell or safety | 4 | Needs a person (96%) | Not a bot gap: route to a person on the first message |
Across the six topics with a real gap, that queue covers 1,455 avoidable handoffs in the month. Re-run the same labels a month after each fix ships and compare the topic's handoffs before and after, so the queue reorders itself on evidence.
How do you find the questions the bot answered wrong?
Some failures leave no trace in the logs: the bot matched an intent, gave a confident answer that was wrong or out of date, and the customer left. These are counted as contained. The only way to find them is to read a random sample of contained conversations against a written rubric.
In the sample, a hand-read audit of 400 contained conversations found 50 with a wrong or incomplete answer, 12.5% (95% interval 9.3% to 15.7%). Applied to the month's 32,400 contained conversations, that is roughly 3,000 to 5,100 silent failures, about as many as the 3,943 flagged conversations with a real need. Label the silent failures with the same topic and gap lists, so they join the same queue. How to check whether your AI agent really resolved conversations covers sample sizes and the rubric, and containment vs resolution vs deflection explains why the contained count runs high.
What goes in a bot gap analysis report?
The report that keeps a self-service program improving is short and the same every month:
Flagged conversations, with noise removed and shown separately, by bot and by language.
The gap-type mix, so the team can see whether the problem is content, matching or routing.
The training queue, ranked by avoidable handoffs, with an owner for each fix.
New needs from the free-text field that do not fit any topic yet.
The silent-failure rate from the audit, with its interval.
Before and after for every fix shipped last month.
Splitting by language matters more than most teams expect. The American Red Cross story describes high fallback and not-handled intent in Spanish-language conversations, which a single overall rate would have hidden.
For the full set of bot metrics that sit around this report, see chatbot analytics: 13 metrics to track. If you are choosing software to run it, we compare the options in which chatbot and AI agent analytics tools are worth using.
What should a bot team use to find every unanswered question, across every bot?
Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.
For a gap analysis, that means four things:
It is fully omni-channel. Bot transcripts sit next to the tickets, calls and emails customers send after the bot fails, so a gap shows up wherever the customer went next.
Integration is easy. Intercom connects in one click, Ada and your own model's logs come in through the universal API, and on Prove each conversation can be joined to account and billing data.
Every conversation is enriched the same way: topic, gap type, outcome and the customer's own words, with custom Dimensions for your business from the Discover plan.
The visual analytics layer builds the monthly gap report, the training queue and the before-and-after charts, as decks, dashboards or PDFs for the bot, content and support teams.
Intuit's QuickBooks Assistant, more than a million sessions a month across 50+ bots, used this to find the causes of mishandled and fallback intents, and reported a 35.3% reduction in its fallback (not handled) intent rate and a 57% reduction in human escalations. The American Red Cross uses it to pull unresolved intents and new training examples from live conversations during disasters. "When disaster strikes, we have to quickly interpret conversations at scale and adjust to new requests and topics. Dimension Labs gives us the insights to provide the support needed to those who need it most." (Hailey Burgess, Product Manager, American Red Cross)
See AI agent and chatbot analytics for what the setup covers.
FAQ
How do I find what customers ask our chatbot that it can't answer?
Export every conversation with a fallback, a handoff, a request for a person or abandonment after a bot reply. Label each with the customer's need and the reason the bot failed, from fixed lists, and add a free-text field for needs that fit no topic. Then read a random sample of contained conversations to catch wrong answers the logs never flag.
How should I categorize chatbot fallback utterances?
Twice: by topic, from your own list, and by gap type (not covered, covered but missed, answered incompletely, needs a person, noise). The topic says where to work, and the gap type says what kind of fix it needs and who owns it.
Can I just cluster the fallback utterances?
For a first pass, yes. Clusters are a quick way to draft a topic list. They shift every time you re-run them, though, so a fixed list is better for counting month over month, with a free-text field to catch what the list misses.
How often should we run a bot gap analysis?
Monthly works for most teams, and weekly in the first months after launch or during a seasonal surge. Run the same lists each time so the before-and-after on each fix is a fair comparison.
What is the best tool for chatbot gap analysis?
For gaps across several bots and the channels customers move to next, ranked by what they cost, Dimension Labs. If you run one bot platform, start with its own reports; Ada, for example, documents a knowledge gaps view built from unresolved conversations. For prompt and model debugging, an observability tool fits better.
Dimension Labs reads every bot and AI agent conversation, labels each one with the need and the gap, and builds the training queue for you. See pricing, or start with one file of bot transcripts. Plans start at $299 a month with a 14-day free trial.