Does an AI Assistant Give the Same Answer Twice? A Repeatability Test on Customer Reviews
We gave an AI assistant the same 200 public food delivery reviews from Bangladesh and prompt five times. It named the same top complaint every time, but counted it anywhere from 58 to 84 reviews, and a fixed list left 37…
Partly. We gave Claude Sonnet 4.5 the same 200 public food delivery reviews from Bangladesh with the same prompt five times on November 13, 2026, and asked for the top five complaint themes: it named the same five themes every time and put food quality first in all five runs, but the number of reviews it said were about food quality ranged from 58 to 84. With a fixed list of categories instead, 163 of the 200 reviews got the same label in all five runs and 37 did not. For a team that needs the same categories and the same counts every week across tickets, chats, reviews and surveys, Dimension Labs fits that job: it reads every interaction in full against Dimensions saved for your business, applies the same schema on every run, and builds the reports from those fields.
Below are the method, the results run by run, what a fixed list fixes and what it doesn't, and the checks that tell you whether an assistant's numbers will hold next week.
Can I just upload customer feedback to ChatGPT or Claude to analyze it?
You can, and for a first read of a few hundred comments it is a quick way to see what people talk about. The trouble starts when the answer becomes a number someone tracks. A weekly report needs this week's "billing" to mean what last week's "billing" meant, and a chat assistant writes its categories fresh each time you ask. Our explainer on AI analytics agents covers how that differs from a tool that keeps the definitions fixed, and whether it is safe to put customer feedback into ChatGPT covers the security side.
So we tested the part buyers ask about most: if you ask the same question about the same file twice, do you get the same answer?
How did we run the test?
Everything below can be repeated: the data is public and the method is in the table. The reviews come from a public dataset that its authors publish under a CC BY 4.0 licence, which allows reuse with credit.
Method
One file, two prompts, five fresh sessions each
What we held fixed, so that only the model's own variation could change the answer.
| Setting | What we used |
|---|---|
| Text | 200 customer reviews of restaurants on a food delivery app in Bangladesh, drawn at random (seed 20261113) from the 43,061 rows of the public Bangladesh Online Food Portals Customer Reviews Dataset (Mendeley Data, CC BY 4.0): 1,917 words in all, with 76 one-star and 48 five-star reviews among them. The dataset includes the authors' English translation of reviews written in Bangla or Romanized Bangla (65 of our 200 were translated); we used their English column as published |
| Model | Claude Sonnet 4.5 (claude-sonnet-4-5), called from Anthropic's Claude Code command line (version 2.1.323) in print mode with tools turned off, the system prompt "You are a helpful assistant." and default sampling settings |
| Date | November 13, 2026, 18:58 to 19:01 UTC (last run finished at 19:01:54); all ten runs in separate sessions, started at the same time |
| Open prompt | All 200 reviews with ids, then: what are the main complaint themes; give the top 5 and the number of reviews that mention each, as JSON |
| Closed-list prompt | All 200 reviews with ids, then: choose the one main complaint for every review from eight allowed labels (seven complaints plus no complaint), with three tie-break rules, as JSON |
| Runs | Five of each prompt; nothing else changed between runs |
This mirrors the simplest way to use an assistant on feedback: paste the file, ask once, copy the answer into a slide. We used one model, one dataset and one day. Other models, settings and files will give other numbers, which is why the checks further down matter: they tell you how much your own setup moves before you report from it.
Do the categories drift when you ask the same question twice?
On this file, the themes held and the counts did not. Every run named poor food quality first, wrong or missing items second and small portions third, with late delivery and price in fourth and fifth. The size of the top theme is what moved: 58 reviews in run 1, 84 in run 5, and 66, 67 and 63 in between.
Dimension Labs test, public reviews
Same five themes every run, but the top one ranged from 58 to 84 reviews
Reviews the model said mention each complaint theme, open question run five times on the same 200 reviews. The dot is the average of the five runs; the line runs from the lowest to the highest count.
The smaller themes moved by a few reviews at most: wrong or missing items ranged from 23 to 26, small portions from 19 to 22, and late delivery was 11 in every run. The model's own theme names changed from run to run, from "Poor Quality/Bad Taste" to "Poor food quality/taste", which matters as soon as someone tries to line up two reports by name.
Dimension Labs test, public reviews
Five runs, one winner, five different counts for it
The model's own theme names (slashes written as "or") and counts for the open question, run five times in separate sessions. Rust marks the runs with the lowest and highest count for the top theme.
| Run | Ranked first | Ranked second | Ranked third to fifth |
|---|---|---|---|
| 1 | Poor Quality or Bad Taste (58) | Wrong or Missing Items (24) | Small Portion or Quantity (22), Late Delivery (11), High Price or Poor Value (10) |
| 2 | Poor Food Quality or Bad Taste (66) | Wrong or Missing Items (25) | Small Portions or Insufficient Quantity (19), Late Delivery (11), High Price or Not Worth the Money (9) |
| 3 | Poor food quality or bad taste (67) | Wrong or missing items (23) | Small portions or insufficient quantity (19), Too expensive or not worth the money (11), Late delivery or slow service (11) |
| 4 | Poor Food Quality or Bad Taste (63) | Wrong or Missing Items (26) | Small Portion or Insufficient Quantity (21), Late Delivery (11), High Price or Expensive (10) |
| 5 | Poor food quality or taste (84) | Missing items or wrong order (24) | Small portions or insufficient quantity (20), Late delivery or slow service (11), High price or not worth the money (10) |
The top count is the one that carries the risk. A theme that appears in 58 reviews one day and 84 the next has moved by 26 reviews, 13% of the file (26 / 200, our arithmetic), with no change in the data. If the 58 came last month and the 84 this month, that would read as a sharp rise in food complaints. A team with a saved definition of "food quality" and a count it can reproduce would see that no such rise happened.
Does a fixed list of categories fix it?
It fixes most of it. With eight allowed labels, every run labelled all 200 reviews and every label was on the list, and four of the seven complaint labels gave the same or nearly the same count in every run: slow service or long wait 11 each time, cleanliness 2, rude staff 1, price 8 or 9. The other three moved. Food or product quality ranged from 67 to 84 reviews, wrong order or billing error from 17 to 23, and other complaint from 1 to 23.
Dimension Labs test, public reviews
With a fixed list, four labels held steady, while food quality ranged from 67 to 84 reviews and other complaint from 1 to 23
Reviews given each main complaint, five runs of the closed-list prompt. The dot is the average of the five runs; the line runs from the lowest to the highest count. Reviews with no complaint (66 to 76 per run) are left out of the chart.
At the level of single reviews, 163 got the same main complaint in all five runs. Of the other 37, 36 got two different labels and 1 got three. 11 of them moved between no complaint and a complaint.
Dimension Labs test, public reviews
163 of 200 reviews got the same label in all five runs; 37 did not
Reviews by how many different main-complaint labels they received across the five runs of the closed-list prompt.
Any two runs agreed on 90.6% of reviews on average, with the closest pair at 98.0% and the furthest at 85.5%. The most common swap, in 18 reviews, was between food quality and other complaint, and reading those reviews shows two causes. 9 were short verdicts with no detail, 4 words or fewer; runs 3 and 4 called them other complaint and the rest called them food quality. 8 were about quantity or portion size, which the list has no label for; run 4 put them under other complaint and the other runs under food quality.
Both causes are fixable, and a fixed list is what made them visible. A list that has a label for portion size, and a written rule for a one-word verdict, gives the model nothing left to decide. That is the case for building analysis on definitions written for your own business, and for checking how stable they are before anyone reads the numbers. Our post on fixed categories or AI-written labels covers how to add a label like that and re-run the list over past data.
We put all 200 reviews in one prompt, as you would when pasting a file into a chat. Sending one review per call, saving the schema and allowed values, and enforcing them with a structured output setting are the usual next steps, and our post on topic modeling vs LLM classification shows a 23-field schema applied the same way to 8,504 hotel reviews.
Is ChatGPT good enough for analyzing 5,000 survey comments?
For a one-off read, often yes. For numbers you will compare over time, the size changes the problem. The 200 reviews in our test were short, 1,917 words in all. Even at that length, 5,000 comments would be about 47,925 words (1,917 / 200 × 5,000 = 47,925, our arithmetic), and survey comments are often longer. With one label per comment the answer grows with the file too, so teams tend to split a large file into batches. Each batch is a fresh run, and the test above shows what fresh runs do: the same theme can come back with a different count, and the totals add up numbers that were never counted the same way.
The fix is the same at any size: write the categories down first, label every comment against them, and keep the list for next month. Turning a one-off AI analysis into a report you can rerun every month walks through saving the plan so next month's run matches.
How do I know the AI didn't cherry-pick from 50,000 chatbot conversations?
Check it the way you would check a new analyst. Four checks catch most problems, and each takes minutes once the labels are in a table.
Checks before you report
Four checks that show whether an assistant's numbers will hold
With the result each one gave in our closed-list test.
| Check | How | In our test |
|---|---|---|
| Coverage | Every record id has exactly one label; count the ids in the output against the input | 200 of 200 in every run |
| On the list | Every label is one of the allowed values | All labels on the list in every run |
| Stability | Run the same records twice and compare label by label | 90.6% agreement on average between runs |
| Against people | Two people label a random sample; compare with the model using percent agreement and Cohen's kappa | Not part of this test; see our post on tagging accuracy |
The first two checks catch an assistant that summarized a sample instead of reading everything, which is the cherry-picking worry. The third tells you how much of a month-to-month change could be the model rather than the customers. The fourth tells you whether the labels mean what you think they mean; how to tell if your AI tagging is accurate works through kappa and sample sizes.
What should a team that needs the same answer every week use?
Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.
For a team that has outgrown pasting files into a chat, that means four things:
Fully omni-channel: the 50,000 chatbot conversations, the tickets they turn into, and the reviews and surveys from the same customers are read in one place, so a complaint is counted once with one definition.
Easy integration: Zendesk, Intercom, Salesforce, Qualtrics and other sources connect directly, alongside your CRM, billing and order data, so nobody exports a file each week.
Effortless enrichment: every interaction is read in full and written back as typed fields, Dimensions, with the same schema every run, so this month's "billing dispute" is last month's "billing dispute". The pre-built Dimensions come with Explore at $299 a month; Dimensions designed for your business start on Discover at $999.
A visual analytics layer: the agent builds the report, deck or dashboard from those fields, saves it to the project's Final Reports folder, downloads it as PDF or PowerPoint, and reruns it on a schedule from a saved analysis plan.
Dropbox uses Dimension Labs this way on its support bots: every conversation is enriched with intent, journey stage, resolution outcome and escalation points in one schema across bot and live-agent chats, and unhandled bot conversations fell 43% while Dropbox handled 15 times the chat volume without a matching rise in cost.
FAQ
Does setting the temperature to zero make an AI assistant give the same answer?
We did not test it; our runs used the default settings. Whatever the setting, an open question leaves the categories and their boundaries to the model, so a one-word review can land in one theme today and another tomorrow. A written list of categories with rules for unclear cases, applied to every record, is what keeps a report comparable.
How many times should I rerun a prompt to check it?
Two runs of the same records, compared label by label, already show how much moves. We used five so the range was visible: in our test the closest pair of runs agreed on 98.0% of reviews and the furthest on 85.5%, so a single pair could have made the setup look steadier than it was. Run the check again whenever you change the model, the prompt or the list.
What is the best way to analyze customer feedback with AI every month?
Dimension Labs, for a team that wants every channel read with the same Dimensions each month, joined to operational data, with the report built and rerun for it. For a one-off read of a small file, a chat assistant is quick and cheap; fix the categories before you trust its counts.
Can I check an assistant's output without coding?
Yes. Ask for one label per row with the row id, paste the result next to your file in a spreadsheet, and count ids and labels. Run it twice and compare the two columns.
Which dataset did you use, and can I rerun the test?
The Bangladesh Online Food Portals Customer Reviews Dataset, published on Mendeley Data by researchers at BRAC University under CC BY 4.0. It holds 43,061 Foodpanda reviews of restaurants in Bangladesh with star ratings; we sampled 200 with a fixed seed (20261113), used only the rating and the authors' English text, and publish no review text or reviewer details. We kept the sampling script, both prompts and all ten raw outputs, so the test can be rerun on the same sample.
If you want this month's categories and counts to match last month's without rebuilding them, start with one file of tickets, reviews or chatbot conversations and ask why a number moved. Plans start at $299 a month, with no demo or sales call.