How to Tell Whether a Product Release Changed Your Reviews
Before-and-after comparisons credit a release with everything else that happened that week. How matched windows, time series and control groups separate the two, with a worked example on sample data.
To tell whether a product release changed your reviews, compare the reviews written after it with the reviews written before, in three increasingly strict ways: matched windows of the same length, an interrupted time series that allows for the trend you were already on, and a comparison with customers who had not yet received the release, such as the other platform or devices still on the old firmware. Then read the reviews from after the release for what they name. Only the comparison with a control group separates the release from everything else that happened that week. Dimension Labs is the best fit for a product team that wants app reviews, NPS and support tickets tied to its release log and reported after every launch: it brings every channel into one place with your release and device data, tags every comment with Dimensions written for your product, and builds the post-release report for you.
Below are the methods, a worked example on clearly labelled sample data, and a short case from public Duolingo reviews of why a version number alone can mislead.
What data do I need to test a release?
Three things on every row and one list beside them:
The date and the rating. A star rating for reviews, the 0 to 10 answer for NPS, CSAT for tickets.
The version the customer was on. App store review exports usually carry it; the Duolingo App Store file has a version on all but one of 47,705 reviews. For NPS on a device, join the firmware from your device records at the time of the response.
The platform, market and segment. Releases rarely reach everyone on the same day.
A release log. Ship date per platform, the rollout percentage by day, and any server-side flags turned on separately from the binary. A feature switched on by flag will not line up with a version number.
If you only have the rating and the date, you can still do a matched-window and time-series check, but not the control comparison.
Why is a simple before-and-after comparison misleading?
Because the release is rarely the only thing that changed. A marketing push brings in new users who rate differently. A holiday shifts who writes. A price change or a news story lands the same week. Weekly releases make it worse: when you ship every week, every drop lines up with some version. And reviews come in bursts after a bad week, so the weeks right after a dip are often better by chance alone. Before reading any release effect, check whether the mix of reviewers changed; NPS drop or mix shift shows the split.
Methods for a release question
Four ways to compare before and after, from weakest to strongest
What each comparison holds fixed, and when to use it.
| Method | What it compares | What it handles | What it misses |
|---|---|---|---|
| Plain before and after | The weeks after the release with the weeks before | Nothing beyond chance | Seasons, campaigns, other changes that week |
| Matched windows | Equal windows on each side, same weekdays, release week left out | Day-of-week patterns, the messy launch days | Anything else that changed at the same time |
| Interrupted time series | The jump at the release against the trend before it | A rating that was already drifting | A shock that lands the same week |
| Control group (difference-in-differences) | Customers who got the release against customers who had not yet | Seasons, campaigns and news that hit both groups | Differences between the groups that change at the same time |
A worked example on sample data
The data in this section is synthetic, made up for this post. A team ships version 6.2 to Android in week 9 and to iOS in week 13. In weeks 9 to 12, a seasonal wave of new users also arrives on both platforms. We built the data so that 6.2 truly raises the share of one- and two-star reviews by 5 points and the wave adds 3 points on both platforms.
Sample data, synthetic
Both platforms got worse in week 9, but only one had the new version
Share of one- and two-star reviews per week. Version 6.2 shipped on Android in week 9 and reached iOS in week 13.
iOS, still on 6.1Android
A plain before-and-after on Android, four weeks each side, says 6.2 raised the low-star share from 11.7% to 21.9%, about 10 points. An interrupted time series on Android alone says 11.9 points. Both are about double the real effect, because they credit the seasonal wave to the release.
iOS shows why. It had no new version in weeks 9 to 12, and its low-star share still rose 4.4 points, from 9.9% to 14.3%. That rise is the wave plus noise. Subtracting it is a difference-in-differences: Android's change minus iOS's change over the same weeks. It gives 5.8 points (95% interval 1.9 to 9.6). When 6.2 reached iOS in week 13, the same comparison the other way round, with Android as the group that had already updated, gives 4.3 (0.7 to 8.0).
Sample data, synthetic
Only the comparisons with a control group land near the real effect
Estimated change in the share of one- and two-star reviews caused by version 6.2, by method, with 95% intervals. The sample data was built with a true effect of 5 points.
True effect in the sample data: 5 points
Two estimates from two separate rollouts, both near the true value, are much stronger evidence than either alone. If your release reached platforms, markets or device cohorts on different dates, plan the comparison before you ship and keep the staggered rollout long enough to read it. A week of overlap with about 400 reviews per group gives a wide interval; four weeks narrows it.
Did version 6.2 cause the spike in one-star reviews?
Run five checks before you answer yes.
Timing on each platform. The low-star share should rise when 6.2 reaches each group, and not before. A rise that starts a few days early points to something else.
A control group. Compare with customers not yet on 6.2 over the same weeks. If they got worse too, the release explains only the difference.
What the reviews name. Tag every review with a fixed list of topics and count the ones that name the changed feature per 100 reviews, before and after. A release effect usually comes with a topic you can see. Our post on why a rating is dropping walks through that topic comparison step by step.
Dose. In a staged rollout, the effect should grow with the share of users who have it.
A placebo date. Run the same comparison on a week with no release. If it finds a similar jump, the method is picking up noise.
For the size of the effect in revenue rather than stars, join the reviews or tickets to subscriptions and refunds; what a customer complaint costs covers the join.
What can a version table get wrong?
In public App Store reviews of Duolingo, the share of one- and two-star reviews was 19.0% on version 7.68 and 20.9% on 7.69, then 62.7% on 7.70 and 71.6% on 7.71. Read as a version table, 7.70 broke something. But the first 7.70 review is dated April 29, 2025, the day after Duolingo announced it was going "AI-first", and the reviews from that week are about the announcement. The few reviews still written on the older build on April 28 and 29 were already worse than the two weeks before, 28.0% of 75 against 19.4% of 511, a small sample but in the same direction.
Public App Store reviews of Duolingo
A version table would blame 7.70 for a drop that began with an announcement
Share of one- and two-star App Store reviews written on each version, April 15 to May 5, 2025.
Two things about App Store data make this easy to get wrong. Reviews move to the newest build almost at once: in February and March 2026, 98.8% of the reviews in the file were written on the newest version released so far, so there is rarely an old-version group to compare with. And a weekly release schedule means every news event, price change and server-side switch shares a date with some version. The Duolingo analysis separates the announcement from the later Energy rollout by reading what the reviews say, not which build they came from.
How do I tie NPS and review trends to firmware releases?
Hardware has an advantage here: firmware updates roll out over weeks, and many devices stay on the old version for a while. That gives you a control group in the same calendar weeks.
Join each NPS response and support ticket to the device's firmware on the response date, from your device records. Reviews on retail sites will not carry it, so use them for the topic read, not the version split.
Compare respondents on the new firmware with respondents still on the old one, in the same weeks, within the same product and market.
Check the groups before the update too. Early updaters often differ (more engaged, auto-update on), so compare the change in each group's score, not the level.
Tag the comments for the feature or fault the release touched, so a score change comes with the reason customers give.
Which tool ties reviews and NPS to specific releases?
Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.
For a release question that covers four things. App Store and Google Play reviews, NPS, support tickets and chats sit in one place, so you can see whether a release shows up in reviews and the inbox in the same week. Your release log, rollout percentages and device firmware come in alongside, so every comment carries the version and the date the customer got it. Every comment is tagged with Dimensions written for your product (the feature names, the faults, the workflows a release touched), and product, support and engineering read the same labels. And the agent builds the post-release report as a deck, dashboard or PDF on a schedule after each launch. Custom Dimensions start on the Discover plan at $999 a month; Explore at $299 uses the pre-built library; effects with confidence intervals and the join to revenue are on the Prove plan at $2,999 and up.
Dropbox uses Dimension Labs to read its chatbot and support conversations at the record level. In the words of Maureen O’Sullivan, Program Manager at Dropbox: “Dimension Labs is key for us to both report outcomes to the business and prioritize where to dedicate resources to enhancing our customer journey.”
FAQ
How do I see if our last app update made reviews worse?
Compare the share of one- and two-star reviews in equal windows before and after the update, leaving out the release week, then compare that change with a group that did not have the update yet, such as the other platform. Read the low-star reviews from after the update for the feature they name.
How long should I wait after a release before judging it?
Long enough to have a few hundred reviews or responses on each side, and past the launch days, when reviews are noisiest. A busy consumer app gets there in weeks; a small product may need months. Decide the window before you look, so the result does not pick it.
What if every platform got the release on the same day?
Then there is no control group from the rollout. Use an interrupted time series against the trend, compare with a segment the change did not touch (a market or plan that does not use the feature), and lean harder on what the reviews say.
What is the best tool to tie reviews and NPS to releases?
For a product team that needs reviews, surveys and tickets joined to its release log and reported after every launch, Dimension Labs. If you only need to watch the store rating after each release and reply to reviews, an app review tool is simpler; our comparison of app review analysis tools covers them.
To test your last release, start with one review export and your release dates: the date, rating, version and text are enough for the first comparison.