How Do I Backtest a Decision We Already Made?
How to backtest a business decision: predict from data before it, grade each range against what happened, and check the method on placebo dates. With a real scorecard from a product launch.
A backtest is how you know which predictions to trust before you bet on one. To backtest a decision you already made, go back to the day before it and predict the outcome using only the data you had then, as a range. Then compare the range with what happened and grade it: held or missed. Run the same method on dates when nothing changed, and check whether it beats simply carrying the recent past forward; if it does not, the prediction is no better than the trend. We check our own predictions this way and publish the scorecard below, misses included, because a method you can judge on its record is one you can act on.
The worked example is real: a hardware company's February 2026 product launch, backtested on 2026-11-06, with the company unnamed. A second example shows how an honest "Unclear" answer steered a company away from a fix aimed at the wrong cause and toward a three-month trial.
What is a backtest of a business decision?
A backtest asks: if we had used this method on the day of the decision, would its prediction have held? It is a test of the method, run on a decision whose outcome you already know. That makes it different from measuring whether a change worked after the fact. Measuring a change compares the outcome with a comparison group using all the data you have now; how to check whether a CX fix moved the number covers that. A backtest hides everything after the decision date, makes a prediction, and only then opens the result.
Two things follow. You learn how far to trust the next forward prediction made the same way. And you get a record, decision by decision, of where the method held and where it missed, which is the only honest answer to "how accurate is this?" It is also the test a customer world model has to pass before it is trusted on a decision not yet made.
How do I backtest a decision step by step?
Freeze the data at the decision date. Use nothing a person could not have known that day. Recast's article on marketing-mix backtests puts the test plainly: for every input, ask whether you would really know it at the moment of the forecast, and if not, leave it out (Recast, December 2025).
Name the outcome, window and segments. For the launch: closed-won orders and revenue in the markets where the new model sold, for the 13 weeks after the announcement, overall and for six customer segments.
Run several methods on fake decision dates. Pick dates before the real one when nothing happened, and see how far each method's "no change" forecast lands from what actually came next. These are placebo dates. The launch backtest used 11 of them, weekly from late August to early November 2025.
Always include carry-forward. Carrying the recent past forward is the bar. A sophisticated method that does not beat it on placebo dates has not earned its complexity.
Pick the method by its placebo error, then build the range from those errors. The spread of a method's misses on placebo dates tells you how wide its range has to be. Check the coverage: a 90% range should contain about 90% of outcomes. Recast's version of the rule: with an 80% interval, reality should land inside about 80% of the time across many backtests.
Open the outcome and grade every range. Held or missed, for the headline and every segment. Then note which ranges held only because they were wide enough to hold almost anything.
What did a real backtest show?
On the 11 placebo dates, carry-forward had the lowest error for both orders (11.3%) and revenue (9.2%), narrowly ahead of synthetic control on orders (11.4%) and structural time series on revenue (11.3%). Difference in differences was far off on revenue (87.7%). With little history, a simple baseline is often the hardest thing to beat, and finding that out is part of the result: it is why we race every method against carry-forward before trusting any of them.
A real backtest, customer unnamed
On fake launch dates, the simplest method, carrying the recent past forward, was as good as or better than the rest
Mean error of each method's no-change forecast on 11 placebo launch dates when nothing launched, August to November 2025. Lower is better.
OrdersRevenue
From data before the launch only, the method predicted 1,591 to 2,410 orders and $3.16M to $4.63M in revenue for the 13 weeks after, if nothing changed. The company booked 2,092 orders and $4.38M. Both totals held.
A real backtest, customer unnamed
Both totals held. Three of the twelve segment ranges missed, all of them on the high side
A hardware company's February 2026 product launch, predicted from data before the launch only: no-change ranges for the 13 weeks after (90%), against what happened.
| # | Segment | Predicted range | Actual | Verdict |
|---|---|---|---|---|
| 1 | All orders in the launch markets orders | 1,591 to 2,410 | 2,092 | Held |
| 2 | All orders in the launch markets revenue | $3.16M to $4.63M | $4.38M | Held |
| 3 | Key accounts orders | 0 to 63 | 62 | Held, wide |
| 4 | Key accounts revenue | $0 to $122K | $163K | Missed, above |
| 5 | Expanding accounts orders | 0 to 135 | 98 | Held, wide |
| 6 | Expanding accounts revenue | $0 to $194K | $189K | Held, wide |
| 7 | Multi-unit first order orders | 0 to 34 | 18 | Held, wide |
| 8 | Multi-unit first order revenue | $0 to $124K | $48K | Held, wide |
| 9 | Single-unit buyers orders | 34 to 200 | 147 | Held, wide |
| 10 | Single-unit buyers revenue | $17K to $426K | $293K | Held, wide |
| 11 | Accessory or service only orders | 1 to 37 | 45 | Missed, above |
| 12 | Accessory or service only revenue | $0 to $75K | $101K | Missed, above |
| 13 | No order before orders | 1,328 to 2,225 | 1,722 | Held |
| 14 | No order before revenue | $2.66M to $4.25M | $3.58M | Held |
The totals holding is not the whole story. Read the scorecard row by row:
The mix moved more than the total. Orders at the new model's price went from 0.3 a week before the launch to 9.2 a week after, while the previous top model fell from 16.6 to 12.4 a week. The launch changed which models sold more than how many. A no-change forecast cannot predict that shift, and a scorecard that shows only totals would hide it.
Three segment ranges missed, all above. Accessory or service only orders (45 against 1 to 37) and revenue ($100,522 against $0 to $74,662), and key account revenue ($163,001 against $0 to $121,835). The likely reason is that these segments were still building up in the months the baseline learned from, so the past understated them. Seeing that tells us where to add data or widen the range before the next launch.
11 of 14 ranges held, but 7 of the 11 held only because they were wide. A range from zero to twice the expected value holds almost anything. Counted honestly: 4 ranges were tight enough to test, and all 4 held; 3 missed; 7 were too wide to prove much. Publishing that split is what gives the 4 tight hits their meaning.
One more thing the backtest exposed: the method choice depended on code. With a different synthetic control implementation, that method would have won on orders, with a range of 1,355 to 2,004, and the actual 2,092 would have fallen outside it. We report both versions rather than the one that looks better. Catching that before anyone relied on the order range is exactly the job a backtest does.
Why do ranges and placebo checks matter?
A single predicted number is always wrong by some amount, so it cannot be graded fairly: a miss of 3% and a miss of 30% both read as "wrong". A range can be graded, and a set of ranges can be checked for honesty. If 90% ranges hold far less than 90% of the time, the method is overconfident; if they hold every time because they are enormous, they say nothing. In the launch backtest, the 90% range came from the placebo errors and held on 91% of placebo dates when each was left out in turn.
Placebo dates do a second job. If a method finds a large "effect" on a date when nothing happened, it will find one on the real date too, whether or not the decision did anything. The same check works for any before-and-after question; how to tell whether a product release changed your reviews uses a placebo week for releases, and how to tell a real trend from noise in weekly feedback sets the normal range from quiet weeks.
What does an honest "Unclear" answer look like?
The same company asked a forward question: what happens if we staff website sales chat from 9am to 5pm Eastern? In October 2026, 26 of the 29 sales chats that went unanswered were already inside those hours, so the change is mostly about capacity and routing, not new hours. That finding alone kept the decision from resting on the wrong cause.
Each account's first sales chat was compared with the orders that followed over the next 45 days, answered against unanswered, with month and buyer type held fixed. If that comparison holds, staffing the hours adds $7,517 to $38,925 of closed-won revenue a month. Averaged over three months, the range runs from $5,537 lower to $54,815 higher, because a few large deals decide each quarter: 63% of answered first chats led to no order within 45 days, and the top 5% of chats carried 51% of the revenue.
A real forward question, graded Unclear
If staffing sales chat pays, the gain is real but one quarter's deals can still hide it
Extra closed-won revenue a month from answering website sales chats 9am to 5pm Eastern, if the comparison between answered and unanswered chats holds. Dot: the expected figure, 26 chats a month at $828 each.
Dashed line: no gain
The grade is Unclear, and we say exactly why:
On the later 172 chats, held out, both methods predicted worse than simply assuming no change.
One stress test failed: a placebo version of the estimate came out at more than a quarter of the real one, against a pass rule of 25%. A test on the 45 days before each chat passed, with no difference.
Two scripts on the same sample gave $706 and $812 a chat, and we have not reconciled them to the dollar.
That is the value of an honest Unclear. Instead of a confident number that could not survive its own checks, the company got a clear next step: staff the hours for about three months and judge the trial on the three-month average. Simulated from the real deal outcomes, a three-month trial comes out ahead about 9 times in 10. The decision is logged with its review date, 2027-01-20, and its result will go on the scorecard whichever way it lands.
What is on the scorecard so far?
The scorecard so far
One launch scored, one forward decision waiting for its review date
Every decision we price gets a range, a coverage target and a review date; misses stay on the card.
One backtest scored and one forward decision open. That is too few to quote a hit rate yet, so we do not. Each decision goes on this page as its review date passes, hits and misses alike, so you can judge the method on its record.
Two lessons from the work so far. The amount and shape of the data limited accuracy more than the choice of model: carry-forward won the launch backtest, and on the chat question the more flexible method lost to a plain regression on held-out chats. And simulated customers, written from each buyer type's own comments, predicted survey scores no better than a plain average of the data. We changed our method because of it: the numbers come from what customers actually did, and simulated voices only help explain the why; are synthetic customers accurate enough for pricing decisions? has the figures and the published research.
Why use Dimension Labs to check a decision?
Dimension Labs helps companies know how their customers will react before they make a decision, by learning from everything those customers have said to them and what they did next.
Our team ran the backtest above on our platform, and our team runs the backtest with you. Underneath it is what the platform does for every customer, the groundwork a trustworthy backtest needs:
It reads every customer channel together: chats, tickets, calls, surveys and reviews, so a decision's effect on what customers said is counted wherever they said it.
It joins those conversations to operational data such as orders, deals and accounts, with the dates of past decisions beside them. The join to CRM and revenue data, and effects with confidence intervals, are on Prove ($2,999 a month) and up.
It enriches every conversation into Dimensions built for your business, so the segment and topic labels are the same before and after the decision date.
It turns the result into a report, deck or dashboard that the team that made the decision can open, with the ranges and the misses on it.
The Seattle Mariners use it to separate real movement in fan satisfaction from noise. In their words: "Ultimately, it allows us to quantify the qualitative and give context and reasoning to KPI movements." (Seattle Mariners story) For testing a decision you have not made yet, see how to test a business decision on your customers before you make it.
When something simpler is the better choice: if the decision was randomized, as in an A/B test with a holdout, compare the groups directly; there is nothing to backtest. And if you only need to know whether one metric left its normal range, a control chart in your BI tool will do.
If you made a decision in the last year and kept the data from around it, you already have what a backtest needs. Dimension Labs turns that record into a scorecard, so the next prediction you act on is one you know how far to trust.
FAQ
How is a backtest different from an A/B test?
An A/B test creates a comparison group on purpose, before the change, and measures the effect. A backtest takes a method for predicting outcomes and checks it on a past decision, using only data from before it. Use A/B tests when you can randomize; use backtests to learn how far to trust predictions about the decisions you cannot.
What if the actual falls outside the range?
Record it as a miss and look for the reason, as with the three segment misses above. Do not widen the range after the fact. If misses keep landing on one side, the method is missing something that changed, and the next prediction should account for it.
How many placebo dates do I need?
Enough that the spread of errors is stable: the launch backtest used 11, each needing 8 weeks of history before it and a 13-week window after. Fewer than about ten gives a range that moves a lot when you add one more.
What is the best way to check whether a business forecast was right?
For decisions whose evidence sits in customer conversations and has to be joined to orders, deals or accounts, Dimension Labs, with the revenue join and confidence intervals on Prove. For a forecast of one series in one system, a holdout backtest in a spreadsheet or your forecasting tool, graded against a carry-forward baseline, is enough.
Find out which of your predictions to trust before you bet on the next one: export the orders or conversations from the months around one decision you already made, with its date, and start with one file. Every plan from Explore to Scale starts with a 14-day free trial.