Regression to the Mean: Why Your Worst Stores Improve After Any Fix
The worst-ranked locations improve the next period whether or not anything is fixed. Public data on 3,892 hospitals shows how much, and how to measure a turnaround against it.
Your worst locations usually improve after a fix because they were picked in a bad period, and part of any bad period is bad luck that does not repeat. In public US hospital survey data, the 389 hospitals with the lowest share of patients who would definitely recommend them in 2023 rose 3.1 points in 2024, while all other hospitals rose 0.4. That group was defined by its ranking alone, not by any shared program. To tell a real turnaround from that drift, compare the locations you fixed with low performers you did not fix, and choose them on a different period from the one you measure from. For an operator running a bottom-quartile program across dozens or hundreds of locations, Dimension Labs is built for this check: it reads every review, survey and conversation by location, and on Prove joins them to sales and operations data so the fixed locations are measured against untreated ones, with a range on the effect.
This page covers one trap in reading location results: picking on an extreme. Matching pilot and comparison stores before a rollout is in how to pick pilot locations before a chain-wide rollout, the full before-and-after test of a fix is in how to check whether a CX fix moved the number, and telling a weekly move from noise is in how to tell a real trend from noise.
Why did our worst locations improve after the fix?
Because a location's score in any one period is its usual level plus chance: a few unusually angry customers, a short-staffed month, a quiet week where three bad reviews count for a lot. Rank locations on that period and the bottom of the list collects the ones with bad luck as well as the ones with real problems. Next period the luck does not repeat, so the group moves back toward the average whether anyone fixed anything or not. Statisticians call this regression to the mean. Francis Galton described it in 1886 in the heights of parents and children, and Martin Bland and Douglas Altman set out how it misleads before-and-after studies in their 1994 BMJ Statistics Notes.
Medicine shows how large it can be. In an Australian trial in mild hypertension, patients were enrolled because their blood pressure was high. The placebo group's average diastolic pressure fell 6.6 mm Hg with no treatment at all, while the treated group's fell 12.2. Without the placebo group, the drug would have been credited with the whole fall. The placebo group is what made the real effect visible, and a comparison group does the same job for a store program.
The same pattern runs through a ranking of locations. We grouped 3,892 US hospitals into tenths by the share of patients who said they would definitely recommend them in 2023, using the public CMS patient survey, and looked at the same measure in 2024.
Public CMS data, analysed by Dimension Labs
The hospitals ranked worst on patient recommendations in 2023 improved most in 2024. The best fell
3,892 US hospitals grouped into tenths by the share of patients who would definitely recommend them in 2023. Bars: average change in that share in 2024, in percentage points.
The worst tenth started at 50.3% and rose 3.1 points; 64.8% of those hospitals improved. The best tenth started at 87.7% and fell 2.1 points. Some of those hospitals surely worked on their scores, and some slipped. The point is what the pattern looks like with no single program behind it: a steady slope from the bottom of the ranking to the top, which is the signature of regression to the mean.
You can estimate how much of the rise to expect from chance alone. The correlation between a hospital's 2023 and 2024 scores was 0.879, so a group that starts 19.2 points below the average should close about 12% of that gap, plus the 0.7 points every hospital moved on average: about 3.0 points. The worst tenth rose 3.1. Nearly all of the improvement at the bottom is what the ranking alone predicts. A program judged on that rise alone would be credited with an improvement the ranking produced by itself.
Why do small locations improve the most?
Because their scores carry more chance. Barnett, van der Pols and Dobson, reviewing the problem for epidemiologists in 2005, note that the effect grows with measurement error and when follow-up is limited to a group selected on its starting value. A location's score built on a few dozen reviews or surveys has a lot of measurement error, so a bad quarter there says less about the location.
Public CMS data, analysed by Dimension Labs
Among the worst-ranked hospitals, the ones with the fewest surveys bounced back the most
Hospitals in the worst tenth for 2023, by the number of completed surveys behind their 2023 score. Average change in 2024, in points.
Within the worst tenth, hospitals whose 2023 score rested on fewer than 100 surveys rose 7.7 points in 2024; those with 300 or more rose 1.9. They started at almost the same level, 49.2% and 50.9%. For a chain, the small-sample locations are the ones with few reviews a month, a low survey response rate or a new opening, and they are the ones most likely to top a worst-stores list by chance and leave it by chance. A real store ranking shows the same thing. In our public analysis of 24 Home Depot stores in metro Atlanta, the store with the highest share of one- and two-star reviews, 32.8% at 1480 Satellite Blvd, is one with under 70 reviews, and the report flags every store under about 70 reviews to read with caution.
A store like that is the likeliest to leave the bottom next quarter whatever is done there, which is why how to compare NPS across locations fairly sets a minimum count before a store is ranked at all.
How do you tell if a turnaround worked or was regression to the mean?
Road safety has the best-documented case. In Great Britain, speed cameras went to sites that had just had unusually many casualties, partly by chance and partly because the roads were dangerous. When the national four-year evaluation allowed for regression to the mean on a subsample of 216 urban sites, the measured reduction in fatal or serious collisions at mobile camera sites fell from 51.6% to 17.6%.
Published evaluation
Allowing for regression to the mean cut the measured fall in serious collisions at mobile camera sites from 51.6% to 17.6%
Reduction in collisions at 216 urban speed camera sites in Great Britain, year ending March 2004, against the rate before the cameras.
Allowing for the national trend onlyAllowing for the trend and regression to the mean
The cameras still worked. The same review concludes that they brought substantial reductions in collisions and casualties over and above the likely effects of regression to the mean. The review also warns that the subsample estimates should be read with considerable caution, because only 216 of the urban sites had the data; that is exactly why a comparison built into the program from the start beats an adjustment made years later. That is the useful lesson for an operator: allowing for regression to the mean can leave a real effect standing, only smaller. It gives the fix its true size, which is the number you need to decide whether to roll it out and what to spend on it.
Five checks, in the order they matter:
Leave some low performers untouched. Take the locations that qualify for the program and let a coin decide which half gets the fix first. The untouched half shows how much the group would have improved anyway. This is the only check that needs no assumptions.
Pick on one period, measure from another. Choose the locations on, say, the six months before last quarter, and measure the change from last quarter onward. The chance that put a location on the list is then not part of the starting score you measure from. Bland calls this a duplicate baseline.
Estimate the drift you should expect. With a few past periods you can compute how strongly one period's score predicts the next, and from that how far a group that starts this low should drift back on its own, as in the hospital estimate above. Credit the fix only with what is left over.
Do not judge on the change from baseline alone. Compare the follow-up scores of fixed and untouched locations while holding their starting scores fixed (an analysis of covariance), which Bland's review of the problem recommends over simple after-minus-before differences, citing Vickers and Altman.
Check that the complaint itself fell. If the fix targeted slow drive-through service, the share of comments about slow service should fall at fixed locations and not at untouched ones. A score can drift back; a specific complaint that disappears only where the fix went is much harder to explain away.
Here are the first two checks on the hospital data. Picked on an earlier period, October 2021 to September 2022, the worst tenth rose 1.7 points from 2023 to 2024, against 3.1 for the worst tenth picked on 2023 itself. And when we split the 2023 worst tenth at random and called one half "fixed", that half rose 2.9 points and the untouched half 3.4, so the "fixed" half did 0.6 points worse, with a 95% interval from 1.7 points worse to 0.6 points better, which is what a fix that does nothing should show.
Public CMS data, analysed by Dimension Labs
Picked on an earlier year, the worst hospitals improved about half as much. Split at random, a "fixed" half did no better than the rest
Average change in the share of patients who would definitely recommend the hospital, 2023 to 2024, in points, with 95% intervals.
no change
A real program would show its effect as the gap between those two halves. Without the untouched half, the "fixed" hospitals' rise of nearly three points would have looked like a success.
Should we pilot a fix on our lowest scoring stores?
Yes, if the fix is meant for them, with two conditions. Hold back a random share of the qualifying stores so the pilot has an untouched comparison from the same part of the ranking, and never judge the pilot against the chain average, because low stores move toward the average on their own. If the fix is meant for every store, pilot it in a spread of stores instead, matched in pairs, as how to pick pilot locations describes. That page warns against piloting in your best stores; the worst stores carry the mirror-image risk, a head start toward improvement that has nothing to do with the change. A pilot picked on a bad quarter can also look like an effect that fades, and how long to wait before judging a change covers the other reasons early results shrink.
Outside factors add to the drift. A bad quarter for a store is often a bad quarter for its weather, roadworks or local competition, and those pass too; the weather ran your last pilot makes that case for pilots in general.
Can past quarters tell you how much drift to expect?
Yes, if every location's history is kept, not just its last score. A customer world model joins what customers said at each location, what they did (visits, orders, renewals), what the business did (which locations got which fix, and when) and what the world did (weather, local events, a competitor opening). With several periods of that record, it can say how much a location's bad quarter usually carries into the next one, and so how much improvement to expect before any fix is credited. If the relationship measured on past quarters holds, the program's effect is what remains after that drift, stated as a range. Checking those ranges against what actually happened is covered in how accurate are customer behavior predictions.
Why judge a bottom-quartile program in Dimension Labs?
Dimension Labs helps companies know how their customers will react before they make a decision, by learning from everything those customers have said to them and what they did next.
For a multi-location operator, that starts with every channel where customers describe a location: Google reviews, surveys, app reviews, support chats and calls, read the same way for every store. It connects them to sales, staffing and the dates each location got a fix with little setup; the join to CRM and revenue data, and effects with confidence intervals, are on Prove ($2,999 a month) and up. It enriches every comment with Dimensions written for your business, such as "order wrong", "waited too long" or "manager fixed it", so the complaint a fix targeted can be counted at fixed and untouched stores alike, and operations, finance and the regional teams read the same labels; custom Dimensions start on Discover ($999 a month). And it turns the result into a deck for the operations review, a dashboard for regional managers, a PDF or executive report, or an interactive data story shared across the company.
Travelers uses Dimension Labs as one governed layer across millions of customer interactions, comparing performance across business units and regions, and reports $3.13M in total savings and a 600% ROI within 18 months. (Travelers story) Comparing units fairly, on the same labels and against the right baseline, is the discipline a turnaround program needs.
With a handful of locations and a fix you can roll out everywhere, something simpler is the better choice: a spreadsheet with two or three past quarters per store and a coin flip for which stores go first will cover most of this page. Dimension Labs earns its place when the program runs across many locations, the evidence sits in reviews, surveys and operational data at once, and each wave of fixes needs a measured effect before the next is funded.
FAQ
What is regression to the mean?
It is the tendency of a group picked for an extreme score to score closer to the average the next time it is measured, because part of what made the score extreme was chance. It happens with no change at all, which is why it can make any fix applied to the worst performers look like it worked.
Why did our worst locations improve after the fix?
Partly, and sometimes entirely, because they were chosen after a bad period. In public CMS data, the worst tenth of hospitals on patient recommendations rose 3.1 points the next year against 0.4 for the rest, with no shared program. Measure the fix against similar low performers that did not get it.
How do I tell if a turnaround worked or was regression to the mean?
Hold back a random share of the qualifying locations, select them on an earlier period than the one you measure from, and check that the specific complaint the fix targeted fell only where the fix went. What the fixed locations gained beyond the untouched ones is the effect.
Should we pilot a fix on our lowest scoring stores?
Only with an untouched comparison group drawn from the same low stores. Judged against the chain average, a pilot in the worst stores will look better than it is.
Running a program for your lowest-scoring locations? Export the reviews or survey comments for the stores you fixed and the ones you did not, add the date each fix went in, and start with one file. Every plan from Explore to Scale starts with a 14-day free trial.