How to Check AI Summaries of Customer Comments Before You Send Them
An AI summary can overstate a theme, flip its direction or miss the second-biggest complaint. Here is how to check one claim by claim against a labelled sample before it goes to managers.
Before an AI summary of customer comments goes out, split it into its separate claims, label a random sample of the underlying comments by hand, and test each claim against the sample: does the theme exist, is its size right, does it point the right way, is every quote verbatim, and is any large theme missing. Send the claims that pass, with a count beside each. The better long-term fix is to have AI label every comment first and write the summary from those counts, which is how Dimension Labs builds the summaries a team sends to store managers and executives.
Below: the five ways summaries go wrong, the labelled sample test step by step, a worked example on sample data, and what our own validation research says about checking AI readings of customer text.
What goes wrong in AI summaries of customer comments?
A summary compresses hundreds of comments into a paragraph, and the errors hide in the compression. The five to look for:
- Size words. "Most", "many" and "several" with nothing behind them. A theme in 4 comments out of 10 is not "most".
- Direction. A feature that comes up often gets described as liked, when most of the mentions are complaints.
- Quotes that aren't quotes. A tidy phrase in quotation marks that no customer wrote.
- Invented or tiny themes. One comment about produce becomes "a highlight".
- Omissions. The theme the summary left out, which is the hardest to see because there is nothing on the page to check.
Each one can send a store manager after the wrong problem, which is why the check is worth the hour it takes.
How do I spot-check an AI summary of customer feedback for errors?
- List the claims. Break the summary into one line per claim, and note the size word and direction in each.
- Draw a random sample of comments. Not the first rows and not the ones the summary quotes. A random draw is what lets the sample stand in for the whole file.
- Label the sample by hand, theme by theme, without looking at the summary. Have a second person label 40 or 50 of the same comments, and fix any definition the two of you read differently. How to tell if your AI tagging is accurate covers that agreement check.
- Score each claim. Holds, overstated, wrong direction, unsupported, or a paraphrase shown as a quote. Search the full file for every quoted phrase.
- Look for what is missing. List the largest themes in your sample and check that each one appears in the summary.
How big a sample
200 labelled comments pin a share down to about 7 points either way
95% margin of error for a share near 50%, from a simple random sample of comments.
For a store-level summary, 200 comments is a practical sample: about 7 points either way on a share near 50%, and less on smaller shares. If the store has fewer comments than that, label all of them.
Worked example: one store's summary, checked claim by claim
Sample data, generated for this post: 1,200 survey comments for one store, an AI-written summary of six claims, and 200 comments labelled by hand.
Worked example, sample data
Two of six claims hold up as written, and the second-biggest complaint is missing
An AI summary of 1,200 survey comments for one store, checked claim by claim against 200 comments labelled by hand.
| The summary said | Sample shows | Verdict | |
|---|---|---|---|
| 1 | Most customers praised friendly, helpful staff | 40.5% (81 of 200); interval 33.9% to 47.4% | Overstated: say about 4 in 10 |
| 2 | Long checkout waits were the top complaint | 18.5% (37 of 200), ahead of out-of-stock items at 11.0% | Holds |
| 3 | Many customers liked the new self-checkout | 22 comments mention it; 68.2% of them are complaints | Wrong direction |
| 4 | Several asked for longer opening hours | 2.0% (4 of 200) | Holds; give the count |
| 5 | Shoppers called the store "clean and easy to shop" | No comment uses those words; 18 mention cleanliness | Paraphrase shown as a quote |
| 6 | Produce quality was a highlight | 0.5% (1 of 200) | Too small to claim |
| + | (not mentioned) | Out-of-stock items: 11.0% (22 of 200) | Omission |
Two claims hold as written. The staff claim is the most common kind of error: 40.5% of the sample praise the staff, with an interval from 33.9% to 47.4%, so "most" is wrong even at the top of the range. The self-checkout claim points the wrong way, and the produce claim rests on one comment. The out-of-stock complaint, in 11.0% of comments, is the second-biggest complaint and is not in the summary at all.
Worked example, sample data
"Most customers" needs more than half, and the sample says about 4 in 10
Share of 200 hand-labelled comments naming each theme, with a 95% interval. Out-of-stock items, left out of the summary, in rust.
Half of all comments
The corrected summary is short and every line carries a count: staff praised in about 4 in 10 comments; checkout waits the top complaint, then items out of stock; self-checkout mentioned in about 1 in 10 comments, two thirds of them complaints. A manager can act on that, and anyone who doubts a line can open the comments behind it.
Can I trust AI-generated summaries of survey comments enough to send them to store managers?
Trust a summary whose every claim traces to counted comments. That changes the order of work: have the model label each comment with a fixed set of themes first, check the labels on a sample, and then write the summary from the counts. Each sentence can then show its number and link to the rows behind it, and the check takes minutes rather than an afternoon.
Three rules make store-level summaries safe to send:
- A minimum count per store. Below it, say there is too little to summarize this month, or roll up to a longer window.
- Quotes copied, never written. Every quoted phrase is pulled from a comment, with its date.
- The same template for every store. So managers compare like with like, as in how to send every location its own feedback report.
What does our research say about checking AI readings of customer comments?
Our published study of AI-predicted experience ratings gave GPT-4.1 the text of about 10,000 open-ended fan responses from five Major League Baseball teams and asked it to predict each fan's 0 to 10 rating. Two-thirds of predictions (67%) fell within one point of the fan's own rating and 36% matched exactly, and three independent runs agreed exactly 87% of the time. The same study found the predictions ran about one point below what fans reported, because a comment describes the memorable moments while the rating is the fan's overall verdict.
Two lessons carry over to summaries. Run the summary more than once and compare the claims, since a claim that appears in one run and not the next is not yet a finding. And look for a consistent lean, not only random errors: a reading of comments can be steady and still sit one step away from what customers would say overall. Knowing the lean is what lets you correct for it, and it is why the full study and the rest of our research publish the misses next to the matches.
Why do teams that send comment summaries to managers use Dimension Labs?
Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.
For summaries that go out to managers, the four parts work together:
- Every channel feeds the same summary: survey comments, reviews, tickets and calls for a store are read together, so the summary is not built from whichever source was easiest to export.
- Integration with survey platforms such as Qualtrics and Medallia, review sources and help desks, plus the store list and operational data, so every comment carries its location.
- Enrichment with Dimensions written for your business: each comment gets the same theme labels, and the agent can preview them on a sample of 30 to 50 records before the full run.
- A visual analytics layer that writes the summary from the counts and builds the store report, the regional deck or the executive PDF, on a schedule and shared by a link you can revoke.
The NBA moved its insight review cycle from about 60 days to under 6 hours, analyzing 100,000+ open-ended responses within hours instead of months. At Stake.com, "Teams don't wait for reports anymore. They explore insights themselves and use them to set priorities."
FAQ
How many comments should I check by hand?
About 200 per summary you plan to send, drawn at random, which gives a margin of about 7 points on a share near 50%. If a claim turns on a small theme, label extra comments that mention it.
Is it enough to check the comments the summary quotes?
No. Those are the comments the model chose, so they will agree with the summary. Size and omissions only show up in a random sample.
What is the best way to produce summaries we don't have to check every time?
Label every comment against a fixed list of themes, check the labels once on a sample and again after any change, and generate the summary from the counts. Dimension Labs works this way across every channel. If you only have a few dozen comments a month, reading them yourself is quicker than any tool.
Does the summary change if I run it again?
It can. Run it two or three times on the same comments and keep only the claims that appear every time, or better, write it from labelled counts so the numbers stay fixed.
Export one month of survey comments with the store attached, and start with one file; the agent labels every comment and writes the summary from the counts.