How to Check Whether a Customer Experience Fix Moved the Number
How to tell whether a CX fix or a retention save play worked: a comparison group, a check that the targeted complaint fell, and a random holdout, on labelled sample data.
To check whether a customer experience fix moved the number, compare the metric before and after the fix for the customers it touched, against a comparison group it did not touch, and confirm that the specific complaint it targeted went down. A before-and-after chart on its own credits the fix with everything else that changed in the same weeks. For a CX team that has to show this across surveys, tickets, chats and reviews, joined to orders or renewals, Dimension Labs is the strongest choice: it is omni-channel, enriches every interaction with Dimensions built for your business, and reruns the comparison as new data arrives.
The steps below use synthetic sample data: a fictional online retailer that fixed a bug where web checkout rejected valid promo codes, and a fictional software company testing retention save plays. Every number on this page comes from that data.
We fixed the checkout bug customers complained about. How do I prove it improved satisfaction?
1. Write down the test before you look
Name the metric (share of post-purchase responses rating the order 4 or 5), the customers the fix touched (web checkout orders), the window (eight weeks either side of the release), and the comparison group (app checkout orders, which never had the bug). Decide what size of change would matter to you. Choosing these after seeing the chart makes it easy to find the result you hoped for.
2. Check that the complaint went away
This is the most direct evidence and the one most teams skip. Label the comments with the same fixed complaint list before and after, and count the complaint the fix targeted per 1,000 responses.
Sample data, fictional online retailer
The complaint itself almost disappeared in the two weeks after the fix shipped
Survey comments on web orders that mention a rejected promo code, per 1,000 responses, in two-week periods. The fix shipped at the start of week 9.
In the sample, comments about a rejected promo code fell from 31.3 to 4.0 per 1,000 web responses. If the complaint does not fall, the fix did not reach customers, or it fixed something else, and there is no point testing the score.
3. Compare the change with a group the fix did not touch
Satisfaction moves for many reasons: season, shipping times, a sale, a different mix of customers answering. A comparison group soaks those up. Here the app checkout shares the products, prices and delivery network but never had the bug.
Sample data, fictional online retailer
Satisfaction rose in both checkouts, including the one that was never broken
Share of post-purchase survey responses rating the order 4 or 5 out of 5, eight weeks before and eight weeks after the web checkout fix.
Weeks 1 to 8, beforeWeeks 9 to 16, after
Web satisfaction rose from 78.3% to 82.8%, a gain of 4.5 points. App satisfaction rose too, by 2.1 points, with no fix at all. Subtracting one change from the other leaves 2.4 points attributable to the fix, with a 95% interval from 0.8 to 3.9 points. This is called a difference in differences.
Sample data, fictional online retailer
Against the app as a comparison, the fix is worth about half the raw jump
Estimated change in the share of satisfied web responses, with 95% intervals.
The raw before-and-after number is 1.9 times the estimate from the comparison. The interval sits clear of zero, so the data supports a real gain, of roughly 2 points rather than 4.
4. Check that the two groups were moving together before the fix
The comparison only works if web and app satisfaction tracked each other before the release. Compare the gap between them in the first four weeks with the gap in weeks five to eight. In the sample the gap widened by 0.4 points, with an interval that includes zero, so the groups were moving together. If the gap was already closing, part of your "effect" is a trend that started earlier.
5. Check who answered
If the release changed who gets surveyed (a new trigger, a new market, more first-time buyers), the score can move without anyone's experience changing. Compare the mix of respondents before and after; did our NPS drop, or did different customers answer? shows the split.
How do I measure whether a customer experience initiative actually worked?
The checkout steps apply to most initiatives, with one rule on top: pick the comparison before launch. Good comparison groups, from strongest to weakest:
A random holdout. Leave a random share of customers, stores or accounts out of the change. This is the cleanest evidence, and the only kind that needs no assumptions about trends.
A staggered rollout. Launch in some regions or stores first. The later ones are the comparison until they switch over.
A segment the change cannot reach. The app for a web fix, phone for a chat change, another product line.
Only the past. A before-and-after with no comparison. Report it, but say that season and mix are not ruled out.
Two outcomes are worth tracking for most initiatives: the score you report, and a behaviour that follows from it, such as repeat orders, repeat contacts or renewals. A score can rise while behaviour does not move. Measuring behaviour usually means joining feedback to order or account data, covered in how to join customer feedback to revenue, orders and retention data.
When the number went the other way and you need to know why, the five checks in why did our CSAT drop? apply in reverse.
Which retention save plays actually keep accounts? How would I measure that?
Hold out a random share of at-risk accounts from every play, and compare retention at a fixed point, such as 90 days after the account was flagged. Without a holdout, the comparison is usually broken by who gets which play: account managers give the executive call to large, healthy accounts that would have renewed anyway, and the discount to accounts that are already leaving.
In the sample, the software company split 2,400 accounts flagged as at risk at random, 600 each, into a holdout and three plays.
Sample data, fictional software company
Only one of three save plays clearly kept more accounts than doing nothing
Change in the share of at-risk accounts still subscribed 90 days later, against a random holdout that got no play. 600 accounts per group.
Dashed line: no difference from the holdout (75% retained)
Accounts that got the executive sponsor call were retained at a rate 10.3 points higher than the holdout (interval 5.8 to 14.8 points). For the renewal discount the estimate is 3.5 points higher and for the training session 0.3 points lower, but both intervals include zero, so this data does not show that either one keeps accounts. A play that does not beat the holdout is a cost with no measured return, and the next quarter's test can try a different version of it.
Two more things to record. Read what the retained and lost accounts said in calls and tickets after the play, labelled with the same list, because that tells you why a play worked. And size the play: an effect per account times the accounts you can reach, times revenue per account, is the number finance will ask for; how to estimate what a customer complaint costs in revenue shows that arithmetic with an interval. Which support topics drive churn covers estimating effects on renewal when a random holdout is not possible.
Can Dimension Labs track whether a fix worked after it ships?
Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.
For measuring a fix, that means:
Surveys, reviews, tickets, chats and calls read together, so the complaint a fix targeted is counted wherever customers raise it, not only in the channel where it was spotted.
Release dates, order data and account records brought in next to the feedback. The join to CRM and revenue data, and effects reported with confidence intervals, are on Prove ($2,999 a month) and up.
The same complaint Dimensions on every record before and after the release, written for your product on Discover ($999 a month) and up, so the count you compare is the same count.
The before-and-after readout saved as an analysis plan and rerun every week on a schedule, with the results in a report or dashboard the team that shipped the fix can open.
The Seattle Mariners use it to separate real movement in fan satisfaction from noise across 81 home games, using significance testing so the team does not react to every 3 to 5% fluctuation. In their words: "Ultimately, it allows us to quantify the qualitative and give context and reasoning to KPI movements." (Seattle Mariners story) More on the method is in how our causal analysis works.
When something simpler is the better choice: if you can randomize the change and the outcome is already in one system, an A/B testing tool or a spreadsheet with the counts answers the question. A dedicated platform earns its place when the evidence is spread across channels and has to be joined to orders or renewals.
FAQ
How long should I wait after a fix before measuring?
Long enough for the fix to reach most of the affected customers and for each period to hold a few thousand responses, or whatever count gives an interval narrow enough to matter. Use the same length before and after, and avoid windows that straddle a holiday on one side only.
What if I have no comparison group?
Report the before-and-after change, the drop in the targeted complaint, and the same change a year earlier for the same weeks, which gives a rough read on season. Say plainly that other changes are not ruled out.
The score did not move but the complaint disappeared. Did the fix work?
It removed the problem. Whether that problem was big enough to move the score is a separate question: a complaint mentioned by 3% of customers can vanish without a visible change in the average. Correlation vs causation in customer experience data covers why the loudest issues are not always the costly ones.
What is the best tool to measure whether a CX fix worked?
For fixes whose evidence sits in surveys, tickets, chats and reviews and has to be joined to orders or renewals, Dimension Labs, with the revenue join and confidence intervals on Prove. For a randomized change measured in one system, an A/B testing tool is simpler.
Export the survey or ticket data from eight weeks either side of your fix, with the release date, and start with one file. Every plan from Explore to Scale starts with a 14-day free trial.