How to Analyze Open-Text Cancellation Reasons From Exit Surveys
How to code the free text in a cancellation survey and read it against the menu choice, worked on sample data where a third of "too expensive" answers never mention price.
To analyze open-text cancellation reasons, code every free-text answer with a fixed list of reasons that you build from the answers themselves, allow more than one reason per answer, and then cross-tabulate those codes against the reason each customer picked from the menu. The cross-tab is where most of the value is: it shows what "too expensive" or "other" actually means for your customers, and which reasons have no box at all. For a team that runs this every month, alongside the tickets and calls that came before each cancellation, Dimension Labs is the strongest choice: it codes every answer with Dimensions written for your business, reads the other channels too, and builds the monthly report for retention, product and finance.
This post is about the exit survey itself. Comparing what customers who left said earlier with what customers who stayed said is a different analysis, covered in churned vs retained customers.
Why isn't the picklist reason enough?
A cancellation menu asks a customer to pick one answer, at the moment they want to be done. Price is the easiest box to tick: it is true in some sense for almost everyone who leaves, it needs no explanation, and it does not invite a retention offer that asks for more. So the menu over-reports price and under-reports everything that takes a sentence to explain, such as a charge they did not expect or a project that ended.
The free-text box is where customers explain. It is also messy: half-sentences, several reasons at once, other languages, and answers like "n/a". Reading it needs a method, not a skim.
The sample data
The worked example uses sample data made up for this post. It does not come from a customer. A subscription business collected 2,400 cancellation survey responses. Each customer picked one reason from six options and could add a comment. 41.7% picked "Too expensive", 18.1% "Not using it enough", 11.5% "Missing features", 11.0% "Switching to another product", 6.5% "Technical problems" and 11.1% "Other".
63.6% wrote something in the comment box. 8.6% of those comments had no reason in them, leaving 1,395 usable answers. Customers who picked "Too expensive" were the least likely to write anything (57.6%), and those who picked "Other" the most likely (78.7%).
How do I code the free-text answers?
Build the list of reasons from the answers, not from the menu. Read a few hundred answers, draft a code for each distinct reason, write a definition and a boundary rule for each, and test whether two people coding the same 100 answers agree. Our post on building a codeframe for open-ended responses covers each step. Four rules matter most for exit surveys:
- Allow several codes per answer. "Too pricey for how little I used it" is both price and low usage. In the sample, 35.8% of usable answers carried more than one code.
- Separate the price itself from billing problems. "Too expensive" and "charged me after I cancelled" need different owners.
- Code the customer's situation as well as the product: a temporary pause and a budget cut are reasons, and they are not failures of the product.
- Mark answers with no reason in them as such, and keep them out of the shares, but report how many there were.
Sample codeframe, made up for this post
Five of the nine codes, with the rules that keep them apart
Each code needs a definition and a boundary rule, so that two people coding the same answer choose the same codes.
| Code | Use when the answer... | Boundary rule |
|---|---|---|
| 1 The price itself 418 answers | Says the price is too high for them, or compares it with a cheaper option. | Not for surprise charges, see the next code. |
| 2 An unexpected charge 138 answers | Describes a charge they did not expect: renewal, upgrade, charge after cancelling, a fee. | Takes priority over the price itself when both appear. |
| 3 Not using it enough 351 answers | Says they did not use it enough, forgot about it, or got less value than expected. | Not for a planned break; use a temporary pause. |
| 4 A temporary pause 221 answers | Plans to stop for a season, a project gap or a trip, and may come back. | Needs a sign of a return or a fixed end. |
| 5 Business or budget change 139 answers | Budget cut, company closed, merged, role changed, project ended. | About the customer's situation, not the product. |
Coding by hand works for a few hundred answers. Beyond that, apply the same codeframe with a language model and check its labels against a person's on a sample every month; how to tell if your AI tagging is accurate shows how.
Customers picked "too expensive" but wrote something else. How do I analyze that?
Cross-tabulate. For each menu choice, take the customers who picked it and wrote a usable answer, and count the share whose answer carries each code. In the sample, 524 customers picked "Too expensive" and wrote a usable answer. Only 40.5% of them wrote about the price and nothing else. 26.1% said they were not using the product enough, 18.3% described a charge they did not expect, and 34.5% did not complain about the price level at all.
Sample data, made up for this post
A third of the customers who picked "too expensive" never mention the price in their own words
What the free text says, among the 524 customers who picked "Too expensive" and wrote a usable answer. One answer can carry several codes.
Those are three different problems. A customer who thinks the price is too high might respond to a cheaper plan. A customer who is not using the product is telling you about onboarding or habit, and a discount only delays the cancellation. A customer surprised by a charge is telling you about billing communication, and is likely to say so in public too. In our analysis of 9,186 public Google reviews of Chicago gyms, about one in six billing complaints described a charge after cancelling or a fee the member did not expect.
Run the same cross-tab for every menu choice. Where the text mostly agrees with the menu, the menu is doing its job. In the sample, the text named a reason in the same family as the menu choice in 84.9% of usable answers, so the menu is broadly right but incomplete.
What does the text find that the picklist cannot?
Reasons with no box. In the sample, 31.3% of usable answers named a temporary pause, a business or budget change, or an unexpected charge, and none of those is on the menu. The "Other" box is mostly these: of 198 usable answers from customers who picked "Other", 66.2% described a pause and 29.8% a business change.
Sample data, made up for this post
Three reasons in the text have no box on the picklist
Share of the 1,395 usable free-text answers that carry each code, across all picklist choices.
A pause is worth its own response: a cheaper pause option instead of a cancellation, and a win-back message timed to the end of the season. If a reason keeps turning up in "Other", add it to the menu, but keep coding the text, because new reasons arrive after every price change and product release.
Which cancellation reasons should we act on first?
Size each reason by the number of customers who give it, not the number of comments, and give each one an owner. Then check two things the exit survey cannot tell you on its own:
- Whether the reason is specific to customers who leave. Customers who stay also say the price is high. Comparing the leavers' earlier comments with those of customers who renewed shows which reasons set them apart; see churned vs retained customers.
- What it costs. Join the coded reasons to revenue per customer to rank them in dollars; how to estimate what a customer complaint costs covers the method, and which support topics drive churn does the same with tickets.
To find which current customers are heading the same way, the coded reasons become inputs to a watchlist; how to build a churn risk report covers that.
Track the coded reasons monthly next to the picklist. When the two start to diverge, for example "too expensive" holds steady while unexpected charges in the text double, something changed in billing.
What should a team use to code exit survey answers every month?
Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.
For cancellation reasons that means: the exit survey sits next to the tickets, chats and calls from the same customers, so a billing complaint before the cancellation and "too expensive" in the survey are seen together; renewal, plan and revenue data come in from billing or the CRM (joining to revenue is on the Prove plan); every answer is coded with your own codeframe as Dimensions, in any language, the same way every month (custom Dimensions start on Discover; Explore uses the pre-built library); and the agent turns the result into the monthly cancellation report as a deck, dashboard or PDF for retention, product and finance.
The NBA moved its open-ended survey work this way: insight review cycles went from about 60 days to under 6 hours, and the league reports 100% coverage of its Voice of Fan feedback, structured and open-ended. Dropbox uses the same record-level tagging on support conversations; in its program manager's words, “Dimension Labs is key for us to both report outcomes to the business and prioritize where to dedicate resources to enhancing our customer journey.”
FAQ
How do I analyze the free-text answers in our cancellation survey?
Build a codeframe from a sample of answers, code every answer with it (several codes allowed), check the coding on a sample, and cross-tabulate the codes against the menu choice. Report shares of customers and keep a count of answers with no reason in them.
Should we remove the free-text box to get more responses?
No. Make it optional and keep it. The menu tells you the category; the text tells you what to fix.
Should "too expensive" trigger a discount offer?
Only for the customers whose text is about the price itself. In the sample, 40.5% of those who picked it gave price as their only reason; for those not using the product, a discount mostly delays the cancellation.
What is the best tool for analyzing open-text cancellation reasons?
For a team that wants exit surveys coded alongside tickets, calls and billing data, every month, Dimension Labs. If you only have a few hundred answers a quarter in one survey tool, coding them by hand with a codeframe, or the text features in that survey tool, may be enough.
To code your own exit survey, start with one export of cancellation responses: the menu choice and the comment are enough for a first read.