Novelty Effects: How Long to Wait Before Judging a Change
Early lifts after a launch or remodel usually shrink. Published experiments and ballpark attendance show how fast, and a worked example shows how to judge the lasting effect.
Wait until the effect stops shrinking. Read the first weeks after a launch as their own period, estimate the effect in fixed windows against locations that did not change, and judge on the windows after the estimate levels off, which for a menu item or a store change is rarely sooner than two months. In a published Microsoft experiment, a clear early effect lost more than half its size within two weeks, and new ballparks kept their crowd bump for anywhere from two seasons to ten. When the question is how long a launch's effect will last across dozens of locations, Dimension Labs fits the job well: it reads every review, survey and conversation by location and week, and on Prove joins them to transactions so every window gets its own effect with a range.
This page is about time: how long to wait, and why early results fade. Choosing the test and comparison locations is in how to pick pilot locations before a chain-wide rollout; measuring a remodel before, during and after the works is in store remodels and new formats; and reading a weekly series for noise is in how to tell a real trend from noise.
What is the novelty effect in a test?
It is a short-lived lift that comes from a change being new rather than better. Customers try the new item, click the new button or visit the new store because it is new, and the extra activity dies away if the change is not truly useful. Its mirror image is the primacy effect: regulars are slower or less happy with a change at first, until they get used to it, so the early result looks worse than the lasting one. Ron Kohavi and colleagues at Microsoft described both in a 2012 KDD paper on puzzling experiment results, and linked novelty to the Hawthorne effect, a short-lived improvement.
A novelty effect does not mean the change failed. In the same paper, moving the Hotmail link on the MSN home page to open in a new tab had a strong novelty effect, and the long-term effect was still positive, so the change shipped. What a novelty effect changes is the size of the answer, and the size is what decides whether a change pays for its rollout.
A Microsoft team showed in 2021 how quickly a clear early result can shrink. In an experiment on about 300,000 users per group, a one-time page about an application update cut later page views by 5.07% in the first three days and 3.11% in week one; by weeks two and three the effect was 1.54% and 1.23% and no longer significant.
Published experiment
A drop that was clear in the first week shrank by more than half and was no longer significant by week two
Microsoft application experiment, about 300,000 users per group: effect of a one-time page about an update on later page views, by period.
Why did our pilot results fade after the first month?
Five causes produce the same picture, and each has its own check:
Novelty. Customers tried the change once. Check repeat behaviour: of the customers who bought the new item in the first weeks, how many bought it again, and did customers who first visited after the launch react as strongly as the regulars? Kohavi's team suggests measuring new users separately for this reason, since they are not affected by novelty or primacy.
Early noise. The first weeks rest on the least data, so they swing furthest. Kohavi's team found that in an experiment with no real effect the first day had a 67% chance of falling outside the 95% interval reached at the end, and that a chart they first read as a trend came from an A/A test, where both groups got the same thing.
Regression to the mean. If the pilot locations were picked because they just had a bad stretch, part of their first-month rise was coming anyway. Why your worst stores improve after any fix shows how large that drift is.
Launch attention ended. Extra staff, a manager watching closely, a launch promotion or local advertising lift the first weeks and then stop. List what was different about launch weeks besides the change itself.
The season or the world moved. A pilot that starts in a strong month and is read against the month before credits the change with the calendar. A comparison group in the same weeks removes this; the weather ran your last pilot makes the case.
The second cause deserves weight. Kohavi's team also could not find a single experiment where a significant result in one direction became significant in the other over time, and found it more common for a clearly positive effect to grow more positive than to reverse. So a fade after a weak first month is usually noise settling, and a fade after a strong one is usually a real but smaller effect. Both are answered the same way: by enough weeks against a comparison.
How long should we wait before judging a new menu item or remodel?
Longer than in an app test, for a plain reason: store customers come back less often. An app user can meet a change several times a day; a restaurant guest may meet it once in several weeks. The novelty has not worn off until most regulars have seen the change a few times, and that takes weeks or months in a store. The published cases show the range.
Published evidence
Early effects faded in every one of these cases. How long they took ranged from days to years
What happened to an early effect in five published cases.
| Case | Early reading | Later reading |
|---|---|---|
| Microsoft app, update page (reported 2021) | Page views 5.07% lower, first 3 days | 1.23% lower in week 3, not significant |
| Microsoft News home page, Mail button (Dmitriev and others 2017, reported 2021) | Button clicks 28% higher | Gap fell day by day; test stopped early |
| MSN home page, Hotmail in a new tab (reported 2012) | Strong novelty effect | Smaller but positive after it wore off; shipped |
| New MLB ballparks, 1950 to 2002 (published 2005) | Attendance 32% to 37% higher in year one | Back to baseline after 2 seasons (1960s parks) or 6 to 10 (newer parks) |
| 24 consumer price increases (Dimension Labs study, 2026) | Price mentions in reviews up sharply | A quarter of the early rise by weeks 7 to 14 |
New ballparks are the clearest physical case. Using Major League Baseball attendance from 1950 to 2002, Christopher Clapp and Jahn Hakes found that a new stadium raised attendance 32% to 37% in its opening year after separating the effect of the team's play, and that attendance stayed above its earlier level for only two seasons at multipurpose stadiums built from 1960 to 1974, against 6 to 10 seasons at newer ballparks. At the newer parks the bump lasted three to five times as long (6 to 10 seasons against 2, our arithmetic). A remodel that adds what customers use can hold its lift the same way; one that only looks new is likelier to fade.
Customer complaints fade too. In our study of 24 consumer price increases, the median share of reviews saying a price went up rose from 0.28% to 2.12% after the new price applied, and by seven to fourteen weeks later the rise was a quarter of the first six weeks. Judged on its first six weeks of reviews, the typical increase looked about four times as contentious as it did two months later.
A rule that works for most location changes:
Never judge on less than a full cycle. Kohavi's team treats a week as the minimum, to cover day-of-week effects. For a store, use at least a full month, so every pay cycle and weekend pattern is in both groups.
Estimate in fixed windows (weeks 1 to 2, 3 to 4, 5 to 8, 9 to 12) rather than one cumulative number, which hides a fade behind the early weeks.
Judge on the windows after the estimate stops falling. When two windows in a row agree within their ranges, that level is the answer. If it is still falling at week 12, keep reading.
Set the read date before launch, and write down which window decides, so the early weeks cannot become the result by default.
How do you read a launch window by window?
Here is the method on synthetic sample data: a fictional regional chain of 40 restaurants adds a new menu item at 20 of them, chosen by a coin flip within matched pairs, and keeps the old menu at the other 20. The sample has a lasting lift of 1.5% built in, plus a novelty bump that starts near 7% and halves about every two weeks. Each window compares the launch restaurants' change in weekly transactions with the comparison restaurants' change over the same weeks.
Worked example, synthetic sample data
The first two weeks read as a 6% lift. From week 9 on, the same launch is worth about 2%
Synthetic sample: a fictional 40-restaurant chain adds a menu item at 20 restaurants. Extra weekly transactions against the 20 that did not, by window, with 90% intervals.
no effect
In the synthetic sample, weeks 1 to 2 read as a lift of 6.1% (4.1% to 8.0%), a range that sits entirely above the lasting effect. Weeks 9 to 12 and 13 to 16 agree with each other, and together, weeks 9 to 16 give 2.3% (1.0% to 3.6%), which contains the 1.5% built into the sample. A single number over all 16 weeks, 3.7%, would still overstate it. The early windows are not wasted: their size and how fast they fall tell you how much of the launch was curiosity, which is what to plan staffing and stock around for the next launch.
Two additions make the reading sharper:
Split first-time from returning customers. With loyalty or order data, compare customers who first visited after the launch with regulars. New customers meet the change as normal, so their response is close to the lasting one; it is the same idea Kohavi's team applies to new users.
Read what customers say, window by window. Label comments for "tried it because it was new", "came back for it", "miss the old one" and "still getting used to it". A rising share of "came back for it" in weeks 5 to 8 is early evidence of a lasting effect; a rising share of "miss the old one" warns of a primacy dip that may or may not recover.
When the launch is a product rather than a store change, launching or cutting a product covers what the launch takes from other items, and how to check whether a CX fix moved the number covers the comparison-group test in full.
How does a customer world model plan for the fade?
By remembering how past launches faded. A customer world model keeps four layers joined: what customers said, what they did, what the business did and when, and what the world did in the same weeks. Each past launch then leaves a curve behind: how big the first weeks were, how fast they fell and where they settled, by location type and customer group. If the relationship measured on past launches holds, the next launch can be read against that curve from its second week, as a range, instead of waiting blind for month three. Checking that kind of range against what happened is covered in how accurate are customer behavior predictions.
Why read a rollout window by window in Dimension Labs?
Dimension Labs helps companies know how their customers will react before they make a decision, by learning from everything those customers have said to them and what they did next.
Around a launch, that means every place customers react: reviews, post-visit surveys, app reviews, support chats and calls, read the same way at every location and dated to the week. Transactions, loyalty data and each location's launch date connect to those comments with little setup, and on Prove ($2,999 a month) and up they join to CRM and revenue data so each window gets an effect with a confidence interval. Custom Dimensions, from Discover ($999 a month), label every comment for your launch, such as "tried it because it was new", "came back for it" or "miss the old menu", in terms marketing, operations and finance all read the same way. And it turns each window's result into a deck, a dashboard, a PDF or executive report, or an interactive data story that can be rerun on a schedule as new weeks arrive, with each run saved to the Final Reports folder.
Speed matters when the answer changes every few weeks. The NBA uses Dimension Labs to read more than 100,000 open-ended fan survey responses within hours and to track week-over-week shifts in product sentiment, and cut its insight review cycles from about 60 days to under 6 hours. (NBA story) A two-month review cycle would deliver the first window's answer after the novelty has already gone.
If you changed one location and can wait a quarter, a weekly sales export and a nearby location that did not change, read month by month, will answer most of this, and is the better choice. Dimension Labs earns its place when launches roll out across many locations in waves, the reaction arrives in reviews, surveys and sales at once, and the next wave's go or no-go depends on knowing the lasting effect early.
FAQ
What is the novelty effect?
A short-lived lift that comes from a change being new rather than better. It fades as customers get used to the change. The opposite, a primacy effect, is an early dip from customers who preferred the old way and have not adjusted yet.
How long should we wait before judging a new menu item or remodel?
Until the window-by-window effect stops falling, and never less than a full month. For store changes that is rarely under two months; new ballparks kept a crowd bump for two to ten seasons.
Why did our pilot results fade after the first month?
Usually novelty, early noise in a small first sample, pilot locations chosen after a bad stretch, or launch support that ended. A comparison group over the same weeks and a split of new and returning customers tell them apart.
Does a fading effect mean the change failed?
No. Many changes keep a smaller lasting effect after the novelty goes; Microsoft shipped the Hotmail change whose early novelty effect faded because the long-term effect was still positive. Judge the level it settles at.
Launched something in the last few months? Export the reviews or survey comments for the locations that changed and a few that did not, add the launch dates, and start with one file. Every plan from Explore to Scale starts with a 14-day free trial.