Churned vs Retained Customers: How to Compare What Each Group Said
How to compare what churned and renewed customers said before their decision, worked on sample data where the topic churners raised most did not set them apart.
To compare what churned and retained customers said, build one row per customer with their outcome and their renewal or cancellation date, collect everything each customer wrote in the same window before that date, tag it with one fixed list of topics, and compare the share of customers in each group who raised each topic. Then hold fixed the things that differ between the groups anyway, such as tenure and plan, because a gap that comes from who the customers are will look like a churn reason. For a team that needs this across surveys, chats, tickets and calls, joined to renewal data from the CRM and reported to customer success and product, Dimension Labs is the strongest choice: it reads every channel, tags every interaction with Dimensions written for your business, and builds the comparison as a report your teams use.
This post is about comparing what the two groups said. Measuring which support topics change the chance of churning, with effects and intervals, is a separate step covered in which support topics drive churn; scoring accounts into a watchlist is in how to build a churn risk report.
What data do I need to compare churned and retained customers?
Two tables and a key. The first has one row per customer: an ID, the outcome (renewed or cancelled), the date of that decision, and a few facts you will want to hold fixed, such as tenure, plan and size. The second has everything customers wrote: survey comments, NPS and CSAT verbatims, chats, emails, tickets, call transcripts, each with the customer ID and a date. Join on the ID and keep only text from the window before each customer's decision date.
Keep the customers who wrote nothing. Silence is part of the comparison, and dropping silent customers changes every percentage you calculate.
How do I choose the window for each group?
Anchor both groups to their own decision date and use the same length of window, for example the 90 days before cancellation and the 90 days before renewal. Comparing churned customers' last 90 days with everything renewed customers ever wrote, or with the last 90 calendar days, mixes up the timing of the comments with the outcome. Customers close to a decision can write differently from customers in the middle of a contract, whichever way they decide.
If cancellations can happen at any time rather than at a renewal date, give each retained customer a comparison date: pick a date in the same month from the retained customer's own history, or match each cancelled customer to a retained one with similar tenure and use the same date for both.
What does the sample company look like?
The worked example uses sample data made up for this post. It does not come from a customer. A software company had 4,000 customers with a renewal date in one quarter. 705 cancelled and 3,295 renewed, a churn rate of 17.6%. Customers in their first year churned at 27.8% and the rest at 10.8%. Everything each customer wrote in the 90 days before their renewal date was tagged with seven topics, and a customer counts as having raised a topic if any of their messages did.
What did each group say?
Start with the plain comparison. In the sample, how-to questions were the most common topic among cancelled customers, raised by 37.7% of them, but 34.1% of renewed customers raised them too. The price increase was raised by 17.7% of cancelled customers and 5.3% of renewed ones. Praise for support ran the other way: 23.2% of renewed customers against 13.3% of cancelled ones.
Sample data, made up for this post
Churned customers raised most topics a little more often, and praise much less
Share of customers in each group who raised each topic anywhere in what they wrote in the 90 days before their renewal date. 705 cancelled, 3,295 renewed.
RenewedCancelled
Silence differs too. 25.8% of cancelled customers wrote nothing in the window, against 31.8% of renewed customers. Report that number next to the topics, because it tells you how much of each group the comparison covers. If you compare only customers who wrote something, every share rises and the gaps change size: among commenters, the price increase came up for 23.9% of cancelled and 7.8% of renewed customers.
Which differences are real, and which come from who the customers are?
Churned customers are not a random sample of your customers. They tend to be newer, smaller or on cheaper plans, and those customers write about different things whatever they decide. In the sample, first-year customers churn more than twice as often as established ones and also ask far more about onboarding. So onboarding confusion shows a raw gap of 6.4 points between the groups, and almost all of it is tenure: compared within the same tenure group, the gap is 1.1 points, with an interval from -1.2 to 3.4.
The fix is simple. Split customers by tenure (or plan, or size), compute the gap within each group, and average the gaps weighted by how many customers are in each group. The price increase gap grows slightly to 14.5 points (interval 11.0 to 17.9), a missing integration goes from 5.5 to 7.4 points, and how-to questions end up 0.1 points lower among cancelled customers than among similar renewed ones.
Sample data, made up for this post
The topic churned customers raise most is not the one that sets them apart
Left: share of cancelled customers who raised each topic. Right: the gap between cancelled and renewed customers, in percentage points, with tenure held fixed.
The ranking changes. Sorted by how many churned customers raised it, how-to questions come first. Sorted by how much more churned customers raised it than similar renewed customers, the price increase comes first, then a missing integration and naming a competitor, and how-to questions fall to the bottom with praise.
Sample data, made up for this post
Once tenure is held fixed, the onboarding gap mostly disappears
Gap between cancelled and renewed customers in the share raising each topic, percentage points, with 95% intervals. Raw gap below each label.
No gap
A gap describes the two groups. Customers who raise the price increase may have been on their way out for other reasons. To estimate how much a topic changes the chance of cancelling, compare customers who raised it with similar customers who did not, holding more fixed, as which support topics drive churn does with tickets.
When did churned customers start saying it?
The second question buyers ask is what customers said in the 90 days before they cancelled. Split the window into 30-day blocks and compare. In the sample, 48.0% of cancelled customers who raised the price increase first did so 31 to 60 days before their renewal date, against 28.0% of renewed customers who raised it. In a real company that window would line up with the renewal notice, which tells customer success when a conversation about price still has time to help.
Timing also catches a common mistake: counting the cancellation conversation itself. Messages sent to cancel are full of reasons, but they arrive after the decision. Exclude the cancellation request and anything after it, or analyze it separately with the exit survey; how to analyze open-text cancellation reasons covers that part.
Does what churned customers say match what makes them angry?
Often it does not, which is one more reason to compare groups rather than read complaints. In our analysis of 9,186 public Google reviews of 33 gyms in metro Chicago, front desk and staff problems did the most damage to ratings, while billing and cancellation trouble came with a stated intent to leave or warn others in half of the reviews that raised it, against 1.4% across all reviews. The full analysis is in why gym members say they will cancel. A complaint list sorted by anger would have missed the billing problem.
Where does Dimension Labs help with a churned versus retained comparison?
Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.
- Fully omni-channel: survey comments, chats, emails, tickets and calls are read together, so a customer who complained by email and stayed quiet in the survey still counts as having raised the topic.
- Easy integration with operational data: renewal dates, outcomes, tenure and plan come in from the CRM or billing system, which sets each customer's window and the groups you hold fixed. Joining to CRM and revenue data is on the Prove plan.
- Effortless enrichment, custom to your business: every message is tagged with the same Dimensions in both groups, written in plain English for your product (price increase, missing integration, competitor named). Custom Dimensions start on Discover; Explore uses the pre-built library.
- A new visual analytics layer: the agent builds the side-by-side comparison as a deck, dashboard or PDF that customer success, product and finance read, and refreshes it each quarter.
The NBA uses the same approach on fan surveys: every open-ended response is tagged, and the league can identify churn signals and upsell indicators directly from fan language, analyzing 100,000+ open-ended responses within hours instead of months.
FAQ
How do I compare survey comments from members who churned with members who stayed?
Give every member a decision date, take their comments from the same window before it, tag them with one list of topics, and compare the share of members in each group who raised each topic. Keep members who left no comment in the denominator, and compare within tenure or plan groups.
Should I compare shares of comments or shares of customers?
Shares of customers. A few customers who write a lot can dominate a comment count, and churned customers often write more in their last months.
How many churned customers do I need?
Enough that the gap's interval is narrow enough to act on. With a few hundred churned customers you can read gaps of five points or more; with a few dozen, read the comments and treat the counts as leads.
What is the best tool for comparing what churned and retained customers said?
For a team that needs every channel tagged the same way and joined to renewal data, Dimension Labs. If your feedback lives in one survey tool and the churn flag is already a field there, that tool's own text and filter features may be enough.
To run the comparison on your own customers, start with one export: a file with customer ID, outcome, decision date and the text they wrote is enough.