Simulated Customers Get the Direction Right and the Size Wrong
In EY's wealth test, heirs said 82% would keep their advisor, a simulation said 43%, and industry studies put it at 20 to 30%. Simulations find the direction; your own customer record gives the size.
Simulated customers are good at telling you which way a reaction will go and poor at telling you how big it will be. The public tests that check a simulation against real people keep showing the same shape: the direction is usually right, the size comes out too large, and the gaps between segments come out larger still. If you are being pitched a synthetic panel to forecast a pricing change, use it to find out what to worry about, and take the size from what your own customers did after your past changes.
Taking that size from the record is what Dimension Labs is built for, so weigh my view with that in mind; the numbers below are all published, ours included.
What happened when a simulation and a survey answered the same question?
The clearest public case comes from wealth management. EY asked Aaru, a simulation company, to rerun its 2025 Global Wealth Study of 3,600 affluent investors in more than 30 markets as a blinded simulation. On one question, whether heirs keep their parents' financial advisor, 82% of heirs in the survey said they would. Aaru's simulated heirs said 43%. The industry studies Aaru cites put the share who actually stay at 20 to 30%, according to Aaru's case study. Ned Koh, Aaru's co-founder, rounded the simulated figure to roughly 40% when he told the story to Fortune.
The 20 to 30% is an industry figure, not a follow-up of these heirs, so it is a yardstick rather than a verdict. The order of the three numbers holds either way.
The survey asked people about a future they had not lived yet, and they answered as the loyal clients they meant to be. The simulation moved the right way, down and by a lot, and still landed above the real range. On a planning base that gap matters: by my arithmetic, a firm that planned on 43% would expect 1.4 to 2.2 times as many retained heirs as the industry range suggests, and a firm that planned on the survey's 82% would expect 2.7 to 4.1 times as many. The simulation was a real improvement on asking people. You still could not have planned revenue on it.
Do simulations overstate effects at random, or in one direction?
In one direction, as far as the published work shows, and that matters because a lean does not average out. A Nature paper published in July 2026 had GPT-4 predict the results of 70 survey experiments with almost 120,000 participants. The predictions were strongly correlated with the real effects, and the authors also found that they "systematically overestimated effect sizes". Steve Bickley, an economist at QUT, put both halves in two sentences: GPT-4 "was often good at ranking the likely effects of different treatments. But it systematically estimated the effects were around twice as big as the real results."
A 2026 benchmark by Chen, Zhu and Zheng tested four language models, from 8 billion parameters to frontier systems, on two large public surveys. On a segment-targeting task the models inflated gaps between segments two to fourfold, and a larger, more capable model did not remove the failures underneath.
Published tests
In three public tests, the simulated figure came out 1.4 to 4 times the real one
How many times larger the simulated figure was than the real one; 1 means right-sized. The EY survey row shows what people said, for comparison. Dots mark the middle of each range.
Real size
My explanation for the lean: a model answering a prompt sees the change clearly and reacts to all of it. A real customer half-notices a price change, is busy, has switching costs and forgets to cancel. That friction keeps real effects small, and it is mostly missing from text about people. The studies show the lean; the reason is my reading of it.
Why segments are where a pricing plan gets hurt
A pricing change rarely lands on one average customer. The plan usually raises the price where customers will tolerate it and protects the group that will not, and that is the step simulation does worst. Chen, Zhu and Zheng found the models treat demographics as far more predictive of attitudes than they are, and that following them would send a team to the wrong segment in about half of US cases. For a strategy lead, that is a coin flip on who gets the price rise.
Sweeping across price points is the other weak spot. In Gui and Toubia's study, people's demand fell as price rose while the simulated buyers produced an inverted U, so even the direction broke. That study and the rest of the research are in are synthetic customers accurate enough for pricing decisions?
Segments are hard with real data too. In the hardware company launch we published, both totals held and three of the twelve segment ranges missed, all on the high side. The difference is what a miss teaches. Those segments were still growing in the months the model learned from, so the next range gets more data or a wider band. A simulated segment gap can only be checked against that same kind of record, which is the case for starting from it.
What do the simulation companies say their tools are for?
The better companies in this market are clear about where their tools fit, and they deserve credit for it. Listen Labs, now part of Salesforce, builds its Listen Twins from interviews with real people it recruits, positions them for everyday questions such as which campaign concept to run or what customers might pay more for, and says a full research study is still needed for higher-level decisions. Aaru describes "simulated populations grounded in real-world behavior and outcomes", and Simile says every population "starts with real people" and that businesses can bring opt-in data of their own. EY announced an alliance with Aaru in September 2026 and says the two are working with financial services firms to test marketing messages, product and pricing offers.
These are serious efforts to ground simulation in behavior, and they suit a question with no history behind it: a new market, a new category, a message nobody has seen. A price change to customers you already have is a different case, because the history exists. It sits in your billing system, your support queue and your reviews, and it is the most direct evidence there is of how those particular customers react. How twins, CDPs and a customer world model differ by starting data is in customer world model vs digital twin, CDP and context layer.
Where does a reliable size come from?
From the smaller versions of this decision you have already made. A price change in one region, a plan you retired, a fee added at some locations before others: each left a record of what customers wrote in the weeks after and what they did, in renewals, orders, refunds and cancellations. Measured against customers or locations the change did not reach, with the weather and the calendar held still, that record gives a size with a range around it, and a way to check it. Predicting how customers will react to a price increase walks through that read on public reviews.
We tried the alternative on ourselves before deciding this. Simulated customers built from one company's earlier NPS comments predicted its later scores no better than a plain average, and each buyer type's own history beat both; the test is written up in our earlier post on synthetic customers. So in Dimension Labs the numbers come from what customers did. Simulated voices can help explain why, labelled as simulated and shown next to the real comments behind them, and they never set a number.
How would I run the pricing decision?
If the synthetic-panel pitch is on your desk, you do not have to choose between the two. Give each source the question it is good at.
How I would split the work
Give simulation the questions about which way and why, and your own record the questions about how much and for whom
For a pricing change to customers you already have.
| Question | Give it to | Why |
|---|---|---|
| Which objections could this price change raise? | Simulated customers | Fast and wide; each objection becomes something to look for in real comments |
| Which way will customers move? | Simulated customers, then check | Direction is what published tests most often get right |
| How many will cancel or trade down? | Your customer record | Simulated effects ran about twice the real size in the Nature study |
| Which segment reacts hardest? | Your customer record, one segment at a time | Simulated segment gaps ran two to four times too large |
| Can I trust the estimate? | A past decision, scored blind | A method that cannot call a decision you already made should not set the next one |
Then ask the vendor the question I would want asked of us. Take one pricing change you have already made, give the tool only what you knew the day before, and compare its number with what happened. Whatever number survives that test should still arrive as a range, with "if the relationship we measured holds" next to it. How to backtest a decision we already made shows that scorecard with its misses.
Where does Dimension Labs fit in that split?
Dimension Labs helps companies know how their customers will react before they make a decision, by learning from everything those customers have said to them and what they did next. It reads tickets, chats, calls, reviews and surveys with one set of labels built for your business, joins them to orders, renewals and locations, and measures what past changes did against comparison groups, with ranges. The join to revenue data and effects with confidence intervals are on Prove ($2,999 a month) and up. That covers the size half of a pricing decision, and a simulation can still supply the hypotheses. What the full model looks like is in what is a customer world model?
FAQ
Do simulated customers get the direction of a reaction right?
Often, for the average reaction to a clearly described change: GPT-4's predictions of 70 survey experiments correlated strongly with the real results in a 2026 Nature paper. Two exceptions matter for pricing: which segment reacts most, and demand across a range of prices.
Should I use a synthetic panel to forecast a price change?
Use it to list objections and to check which way customers are likely to move. For how many will leave, and which segment, use what your own customers did after past price changes, measured against a comparison group and given as a range.
Pick the last price change you made. Export the reviews, tickets or survey comments from the weeks around it, with the date of the change, and start with one file to see how your own customers reacted.