How Many Survey Responses or Reviews Do You Need Before Trusting a Driver?
We subsampled 8,504 public hotel reviews to see when driver estimates stop moving. About 100 mentions of a driver settled it; location rankings need far more.
A driver estimate becomes trustworthy when enough responses in your data mention that driver, not when the file passes a magic total. In our test on 8,504 public hotel reviews, a problem settled to within about 7 points of its true effect once roughly 100 reviews in the sample named it, and a driver with a large effect showed up clearly with only about 40 mentions. So 2,000 survey responses a year is enough for the common, strong drivers and too few for rare ones or for location-by-location comparisons. For a team that wants every survey answer, review and ticket read together and each driver reported with its own count and interval, Dimension Labs does that work for you, and the method below works with any tool.
Below: the subsampling test, a rule for how many mentions a driver needs, what that means for location comparisons, and a checklist for your own data. For the driver method itself, see key driver analysis from text; for why a driver ranking can mislead even with plenty of data, see correlation vs causation in customer experience data.
How did we test how many reviews a driver needs?
We took the public file behind our Orlando hotel review analysis: 8,504 rated Google reviews of 45 Marriott-family hotels, January 2025 to July 2026. 23.1% of them are one or two stars. We flagged six problems in each review with simple keyword rules written for this post (a dirty room or smell, rude staff, something broken, fees, noise, the front desk), then estimated each problem's effect on the chance of a one- or two-star review with all six in one model, holding review length and the hotel fixed.
That gave the reference answer from all 8,504 reviews. Rude staff came out at 40.7 points, a dirty room at 31.5, fees at 17, something broken at 15.2, noise at 8.8, and the front desk at 0.9, with an interval that includes zero. These are close to the published causal estimates for the same reviews, which put rude staff at 44, a dirty room at 34 and the front desk at 2.
Then we drew 400 random samples at each of 250, 500, 1,000, 2,000 and 4,000 reviews, refit the same model in every sample, and watched how far the estimates wandered.
Dimension Labs analysis
The estimate for one driver only settles once about 100 reviews in the sample name it
Effect of a review naming something broken in the room on the chance it is one or two stars, estimated in 400 random samples of each size. Dot: median estimate. Line: the middle 90% of samples. Dashed line: the estimate from all 8,504 reviews.
All 8,504 reviews: 15.2 points
At 250 reviews, the middle 90% of estimates for "something broken" ran from -12 to 38.4 points, so the same file could tell you it mattered a lot or not at all. At 2,000 reviews the range was 8.3 to 21.9, and at 4,000 it was 10.7 to 19.4. The median barely moved. What changed was the spread, and the spread tracks how many sampled reviews mention the problem: about 12 at 250 reviews, about 95 at 2,000.
Is 2,000 survey responses a year enough to find what drives our NPS?
For the drivers that many customers mention and that move the score a lot, yes. For rare drivers and small effects, no. The table shows both cases in the hotel data.
Dimension Labs analysis
Big effects show up in 1,000 reviews; small ones need several thousand, and a null never does
Six drivers ranked by their effect in the full file. The last three columns give the share of random samples of that size in which the driver came out clearly above zero (95% interval).
| Driver | Mentions | Effect | 1,000 | 2,000 | 4,000 | |
|---|---|---|---|---|---|---|
| 1 | Rude or unprofessional staff | 321 | 40.7 | 100% | 100% | 100% |
| 2 | Dirty room or smell | 725 | 31.5 | 100% | 100% | 100% |
| 3 | Fees and charges | 454 | 17 | 73% | 96% | 100% |
| 4 | Something broken | 405 | 15.2 | 60% | 88% | 100% |
| 5 | Noise | 348 | 8.8 | 22% | 35% | 63% |
| 6 | Front desk or check-in | 1,865 | 0.9 | 4% | 3% | 3% |
Rude staff and a dirty room came out clearly above zero in every 1,000-review sample. Fees and broken items, with effects of 15 to 17 points, were found in 96% and 88% of 2,000-review samples. Noise, at about 9 points, was found in only 35% of 2,000-review samples and 63% at 4,000.
Rankings take longer than effects. The top driver was the same as in the full file in 85% of 1,000-review samples and 92% at 2,000, but the full order of all six matched in only 38% of 2,000-review samples, because fees and broken items sit about 2 points apart. Two drivers that close will swap places from one wave to the next. Report them as tied, with intervals, rather than as second and third.
A rough rule that matched the subsampling: to detect an effect of d points with the usual 80% power, you need about 7.8 × p(1 − p) / d² mentions of the driver, where p is the share of bad outcomes among them. With p around 0.25, that is about 66 mentions for a 15-point effect, 148 for 10 points and 589 for 5 points. Divide by the share of responses that mention the driver to get the total you need. With 2,000 responses a year, a driver that 5% of respondents mention gives you about 100 mentions, enough to see an effect of about 12.1 points or more; one that 2% mention gives about 40, enough only for effects near 19.2 points.
The attraction operator who asked us this had a 0.5 to 1% survey response rate and 2,000 to 6,000 responses a year. At that volume, pooling the year is fine for the main drivers; splitting by month, by attraction or by visitor type is where most driver lists stop being reliable.
How many reviews per location do I need to compare locations fairly?
More than most location rankings use. Comparing locations is a different question from finding drivers: each location's rate has its own margin of error, and two locations are only different if the gap is bigger than both margins together.
Dimension Labs analysis
With 100 reviews, a location's bad-review rate is only known to within about 8 points
Margin of error (95%) on one location's share of one- and two-star reviews, at the 23.1% rate seen across the 45 hotels.
At the hotels' 23.1% average, a location with 100 reviews has a margin of about ±8.3 points, and one with 400 reviews about ±4.1. To tell two locations apart reliably, the gap has to be about 17 points with 100 reviews each, 12 with 200 each and 8 with 400 each. The 45 hotels in this file have between 54 and 643 rated reviews each, median 158, over 18 months. A monthly ranking of these hotels would mostly be ranking noise.
For driver analysis by location, it is worse: you need about 100 mentions of a driver at each location. Few single locations have that, so the better design is to estimate drivers across all locations together, with the location held fixed, and then ask where each driver is more common. Our post on sending every location its own feedback report shows how to flag locations with too little data in a monthly report.
What is the minimum sample size for key driver analysis on customer comments?
There is no single minimum, because the answer depends on three numbers you can count before you start:
Mentions per driver. Count how many records in your file name each driver. Under about 50, treat the driver as a lead to read, not an estimate. Around 100 or more, the estimate is usually stable to within 5 to 10 points.
The size of effect you care about. If only a 10-point effect would change a decision, plan for about 150 mentions of each driver; for 5 points, about 600.
The number of cuts. Every split by month, location, product or segment divides your mentions. A file that supports one driver list rarely supports twelve monthly ones.
Then check stability directly, the way we did: refit the driver model on random halves of your data. If the top drivers change between halves, you have too little data for that ranking.
If what you need is a quick read on whether one score really moved, the CSAT drop checker runs the test on your counts, and why did our CSAT drop covers the follow-up steps.
Why would an insights team use Dimension Labs for driver analysis on small samples?
Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.
When the sample is the constraint, those four things help in specific ways:
Every channel counts toward the same driver. Survey comments, reviews, tickets, chats and calls go into one place, so a driver that few survey respondents mention can gather enough mentions once the tickets and reviews are added.
Survey scores, visit records, locations and CRM fields sit next to the text, so each driver can be tested against the outcome you care about. The join to CRM and revenue data, and effects reported with confidence intervals, are on Prove ($2,999 a month) and up.
Every record gets the same Dimensions, so a driver means the same thing in January and in June. Explore ($299) uses the pre-built Dimensions library; Dimensions written for your business start on Discover ($999).
The results come back as a deck, dashboard or report with the count behind each driver shown, so nobody presents a driver that rests on a handful of comments as if it rested on hundreds.
The NBA uses it to analyze 100,000+ open-ended fan responses within hours instead of months, with insight review cycles down from about 60 days to under 6 hours (NBA story).
You may not need Dimension Labs at all if you have one survey with a few rated attributes and no open text, a regression in a spreadsheet or your survey platform's driver chart will answer the question, and the mention counts above still tell you how far to trust it.
FAQ
How many survey responses do I need for key driver analysis?
Count mentions per driver rather than total responses. About 100 mentions of a driver gave an estimate stable to within about 7 points in our test; strong drivers showed up with 40. A total of 2,000 responses is enough for drivers that at least 5% of respondents mention and that have effects of about 12 points or more.
Can I run driver analysis on 300 responses?
You can find the one or two strongest drivers, if they are common. In our test the top driver matched the full-file answer in 60% of 250-review samples, and smaller drivers ranged from strongly positive to below zero. Read the comments, report counts, and wait for more data before ranking.
Should I compare locations every month?
Only those with enough data. With 100 reviews each, two locations need to differ by about 17 points before the gap is reliable. Pool several months, or compare each location with a group of similar locations, and mark small locations as too small to rank.
What is the best tool for driver analysis when samples are small?
For a team whose comments are spread across surveys, reviews, tickets and calls, Dimension Labs, because pooling channels raises the mention count for every driver and each driver is reported with its count and, on Prove, its interval. For one survey with rated attributes only, your survey platform's driver chart is simpler.
Export a year of survey comments or reviews with the score attached, and start with one file to see how many mentions each driver has.