How to Compare Two Chatbot Vendors on Your Own Conversation Data
Each chatbot vendor counts success its own way. Here is how to compare two of them on your real conversations, with one rubric and an outcome measured outside the bot.
To compare two chatbot or AI agent vendors fairly, run both on the same kinds of customer conversations, export the full transcripts from each, label every conversation with one rubric that neither vendor wrote, and compare them on outcomes measured outside the bot, such as whether the customer came back within a week. The vendors' own dashboards cannot settle it, because each one counts success its own way. Dimension Labs is built for exactly this referee role: it reads transcripts from both platforms with the same Dimensions, joins them to what customers did next, and turns the result into a report your team can act on.
Below: why the dashboards disagree, how to set up the test, what to label on every conversation, and a worked example on sample data for a team moving from a rules-based bot to an LLM bot.
Why can't we compare the two vendors' dashboards?
Because the numbers on them answer different questions. A rules-based bot usually reports containment, meaning the conversation never reached a person. LLM agents are often priced and reported on resolutions or outcomes, and each vendor writes its own definition. Intercom, for example, counts a Fin outcome when a customer confirms the issue is resolved, when they don't ask for more help after Fin responds, or when Fin completes a workflow, including handoffs. That is a sensible billing rule, and it is measured inside the chat. A head-to-head between two vendors needs one definition you control, applied to both, plus an outcome from outside the bot.
From each vendor's own pages
Each platform counts success its own way, so two dashboards rarely line up
How three platforms describe what they bill for or report, in their own words, read November 12, 2026.
| Platform | What it counts |
|---|---|
| Intercom Fin | $0.99 per outcome. An outcome is counted when a customer confirms the issue is resolved, when they don't ask for more help after Fin responds, or when Fin completes a workflow, including handoffs. At most one per conversation. |
| Ada | Reports an "automated resolution rate" for its customers; no price is published, and the page offers an ROI model grounded in your volume. |
| Dialogflow CX (Conversational Agents) | Bills the count of requests for chat agents: $0.007 a request for flows and $0.012 for playbooks. |
The three terms in containment vs resolution vs deflection are worth agreeing on before the test starts, so the report uses the same words as the people reading it.
How do we set up a fair test on our real conversations?
There are two designs, and the choice depends on whether you can run both bots at once.
- Split traffic. Send a random share of sessions to each bot over the same weeks. This is the cleanest comparison, because both bots meet the same customers, the same promotions and the same outages.
- Before and after. When you are replacing one bot with another, compare a window before the switch with a window after it. Keep the windows the same length, avoid a launch or a holiday on one side only, and compare contact reason by contact reason, because the mix of questions shifts between periods.
Either way, give both bots the same knowledge base and the same handoff rules, and decide the outcome measures in writing before anyone sees results.
What should we measure on every conversation?
Label each conversation with a short, fixed set of fields, the same way for both bots:
- Contact reason, from a closed list (see how to build a contact reason taxonomy).
- Resolved, judged from the transcript: did the customer get what they came for, in their own words.
- Asked for a person, and whether they got one.
- Failure mode when it failed: wrong answer, no answer, a loop of the same reply, or a policy the bot could not apply.
- Effort: how many turns and how much repetition it took.
Then add one outcome from outside the transcript, such as a repeat contact within seven days from your help desk, or a survey score. That is the measure a vendor's definition cannot move. How to check whether your AI agent really resolved conversations has a scoring rubric you can reuse for the resolved field.
Worked example: a rules-based bot against an LLM bot
Sample data, generated for this post: 2,000 conversations from four weeks before a switch and 2,000 from four weeks after.
By each bot's own count, success rose from 72.3% to 85.5%, a gain of 13.2 points. Labelled with one rubric, the share resolved went from 48.5% to 64.3%. Those two numbers are not comparable with each other, and the second still flatters the new bot, because the after period had more order-status questions, which are easy for any bot.
Worked example, sample data
The LLM bot resolves more account and product questions, and fewer billing disputes
Share of conversations resolved, judged from the transcript with one rubric, by contact reason. Synthetic data: 2,000 conversations per bot.
Rules-based bot, 4 weeks beforeLLM bot, 4 weeks after
Comparing reason by reason tells the real story. The LLM bot resolves far more account and product questions, but billing disputes went from 22.0% resolved to 15.6%. Weighting both bots to the same mix of questions, the gain is 10.1 points, and repeat contacts within seven days fell by 8.4 points.
Worked example, sample data
Measured fairly, the gain is 10.1 points, smaller than either quick reading
Change in success rate from the rules-based bot to the LLM bot, measured three ways on the same sample conversations.
That is a clear win for the switch, with one fix to make before you call it done: route billing disputes to a person sooner. The value of labelling every conversation is that it gives you both answers at once, the verdict and the next fix.
We are moving from a rules-based bot to an LLM bot. How do we compare quality before and after?
The main difficulty is that intent reports disappear. A rules-based bot logs which intent fired and when it fell back; an LLM bot usually has no intents to report. Measuring both from the conversation text solves this, because the same fields can be read from either kind of transcript. Dimension Labs' documentation says its chatbot measures work for intent-based bots, LLM bots and a mix of both.
Two practical points. Export the old bot's history before you switch it off, so the before window survives the migration. And send the old bot's intent and handoff flags as extra fields, because a flag your platform recorded is more reliable than one inferred from text. For whether the new agent changes what customers do later, see is your AI agent hurting retention.
How do we get the transcripts out of each platform?
- Intercom is a one-click connector: you authorize access once and conversations are pulled on a schedule (Intercom integration).
- Ada conversations come through Ada's Data Export API and the universal API, message by message, with the customer's side and the agent's side kept apart (Ada integration).
- Dialogflow CX conversations come from its Conversations API, turn by turn. Conversation history has to be turned on for the agent, because Dialogflow only stores conversations from the time it is on, so switch it on before the test window starts (Dialogflow CX integration).
Keep a session or conversation id and a customer id on every message, so you can join each conversation to the repeat contact or survey that followed.
Why use Dimension Labs as the referee between two bots?
Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.
For a vendor comparison, that means four things:
- Every channel the bots touch sits in one place: the bot transcripts, the human chats they hand off to, the tickets and calls that follow, and the surveys.
- Integration with Intercom, Ada, Dialogflow CX and the other chat, help desk and call platforms on the integrations list, plus your CRM, so the repeat-contact outcome is a join, not a spreadsheet exercise. The join to CRM and revenue data is on the Prove plan ($2,999 a month) and up.
- One set of Dimensions, written for your business, applied the same way to both bots and to every team's questions, from support to product.
- A visual analytics layer: the agent builds the comparison as a deck, a dashboard or a PDF for the steering group, and shares it as a link you can revoke.
Intuit QuickBooks used Dimension Labs across more than 50 bots and reports a 35.3% reduction in its fallback intent rate and 57% fewer human escalations. Dropbox reports a 43% reduction in unhandled bot conversations. Both started from the same raw material you have: transcripts. More on the product side is on AI agent and chatbot analytics.
FAQ
How many conversations do we need for a fair comparison?
It depends on the difference you want to detect. To tell a 55% resolution rate from 60% with the usual 95% confidence and 80% power, you need about 1,534 conversations per bot. Smaller differences need far more, and a reason-by-reason comparison needs enough conversations in each reason, so plan the window around your volume.
Can we trust a vendor's resolution rate?
Trust it as that vendor's definition, used for billing and its own reporting. For choosing between vendors, use a resolved field you define and score both bots against it, plus an outcome from outside the bot such as a repeat contact.
What is the best tool to compare chatbot vendors on our own data?
For a team that wants both bots scored with one rubric, joined to repeat contacts or revenue, and written up for leadership, Dimension Labs. If you only run one platform and want its own reporting, that platform's built-in analytics is the simpler choice. For a one-off check of a few hundred conversations, a careful hand review with a shared rubric works. The wider field is covered in chatbot and AI agent analytics tools compared.
Should we run both bots at once or one after the other?
At once, with randomly split traffic, if the platforms allow it. A before-and-after test works too, as long as you compare contact reason by contact reason and keep the windows free of one-sided events such as a launch.
To score your current bot before you switch, export a month of transcripts with a conversation id and start with one file. Plans are on the pricing page.