The Weather Ran Your Last Pilot

On February 3, 2026, Chipotle's chief executive told analysts that the company's new protein menu, launched in late December, had trends accelerating "all the way up until the storm hit." Its CFO, answering the same question, said the first-quarter outlook carried about 100 basis points of comparable sales lost to restaurants the storm had closed. The CEO's summary of where that left the trend: "a lot of noise in the underlying trend at present." One storm was sitting on top of a menu launch, and the company said so.

That was an honest answer, and most pilot readouts never give it. The typical deck says the change worked and shows a number. My view, from running these analyses with operators: most of those numbers are partly weather, partly the calendar and partly foot traffic, and a pilot result means very little until those three are held still.

The readout on the CFO's desk

Say the readout in front of you says a new store format lifted sales 8% at the pilot locations. The question that decides whether 8% is real is the one the deck rarely answers: what else changed that week?

The season alone can produce a number like that. In U.S. Census data for restaurants and bars, January sits at 90.8 on a seasonal index where a typical month is 100, and May at 106.7, a gap of nearly 16 points with no decision behind it. A pilot that starts in winter and is read in spring has the calendar in it before the format does anything. The storm runs the other way: Brinker, which owns Chili's, put Winter Storm Fern at about $20.0 million of lost revenue in its January 28 release. A pilot that launched the week before that storm looks like a failure on any before-and-after.

Chipotle and Brinker named the storm because they had a quarter to report to investors. A pilot readout inside a company has no such audience, so the weather usually goes unmentioned and gets counted as part of the decision.

One change, read four ways

We built a sample with a known answer to see how far off the usual readouts are. A fictional 40-store chain cuts its menu at 16 pilot stores in Dallas, Fort Worth, Memphis and Nashville on January 20, 2026, with a true effect of 3% fewer visits built into synthetic sales. The weather is each city's real daily record, so the pilot meets Winter Storm Fern three days after launch. The full build is in was it the decision or the weather?; here I only want what it says about a readout.

Every simple comparison was wrong in the same direction. The before-and-after came out at 3.2 times the true effect, and even the comparison with stores that kept the old menu came out at about twice, because the pilot cities got more ice. That second number is the one that worries me most, since a comparison group is what most teams offer as proof that a readout is careful. Only with each store's own weather held fixed did the estimate land on 2.8%, with a range that contains the true 3%.

For a CFO the gap is the decision itself. A 9.5% fall gets a format rolled back. A 2.8% fall gets weighed against the simpler kitchen and faster service the change was meant to buy, which was the real question all along.

Why the error usually flatters the pilot

The sample is a drop. The readout on a CFO's desk is more often a lift, and there the error tends to run in the pilot's favour, because people choose where and when pilots run. A team that believes in a new format launches it in stores with strong managers, in markets that are growing, in a season when traffic is rising. Each of those choices adds to a before-and-after, and none of them is the format. A good spring weekend moves a readout the same way a storm does, with the sign flipped, and nobody writes a footnote about good weather.

So my default reading of an 8% lift with no comparison stores and no weather is that some of it is real and the readout cannot say how much. That is good news for a format that works. A real effect survives the questions below, and a number that has survived them is one finance can roll out on.

What else belongs in the room

Weather is the confounder people already accept, and it is one of several. The rule I hold our own models to is that an outside source earns its place if it is a plausible confounder for an outcome the business already cares about.

For a business with locations the list is short: weather, the season and the calendar, foot traffic, local competition and the makeup of each store's trade area. The last two matter most for the selection problem above. If the pilot stores were picked because they sit in strong markets, trade area and competition belong in the comparison as much as the storm does.

Industry benchmarks do not make the list. Knowing that the category's comps fell tells you about the market. It does not tell you why your pilot stores moved, which is the only thing the readout is for.

The second witness: what customers wrote

Sales say how many people came. What they wrote says whether the ones who came liked the change, and a storm moves the first without touching the second. In the sample, reviews naming the menu change rose from 1.0% to 15.3% at the pilot stores and stayed under 1% elsewhere, so that part of the drop belonged to the decision. On a lift the same test runs the other way. If sales rose 8% and almost nobody mentions the new format, something else brought those people in, and the readout should say what.

Three questions before you accept a pilot number

First, what else changed that week, at these stores and not at the others? Ask for the weather days, the calendar and any local openings or closures, by store.

Second, what did comparable stores that did not make the change do over the same days, with each store's own weather held fixed? A comparison with last year, or with the chain average, carries a different year's weather and a different set of stores.

Third, does the method find nothing when nothing happened? Run the same model on a date with no launch. In the sample it returns 0.5% fewer visits, with a range that includes zero. When our team picks a forecasting method, it has to earn the job the same way, by its error on 11 earlier placebo launches. Ask for the answer as a range, because a range is what an honest result looks like when a storm sits inside the window, and it narrows as each store's weather, traffic and comments are added.

A readout that comes back with all three is one you can fund. The full method, with the Census seasonality and the weather data, is in was it the decision or the weather?, and the general version for any decision is how to test a business decision before you make it.

Why measure pilots with Dimension Labs?

Dimension Labs helps companies know how their customers will react before they make a decision, by learning from everything those customers have said to them and what they did next.

For a pilot, that means reading every review, survey answer, ticket and chat for every store with the same Dimensions, so "names the new format" is counted the same way at each location; joining those fields to store sales, and to outside context such as daily weather, on the Prove plan and up; and reporting each effect with a confidence interval in a deck or executive report finance can open. Holding the season, the weather and what customers said in one place, next to what they did, is what we mean by a customer world model. It also makes the readout fast: the NBA cut its insight review cycles from about 60 days to under 6 hours.

FAQ

How do I know if a pilot result is real or just the weather?

Compare the pilot stores with stores that did not change over the same days, hold each store's daily weather fixed, and run the same model on a date when nothing launched. If the effect shrinks once weather is in, the weather was carrying part of it.

Is comparing with the same weeks last year enough?

It removes the normal season but adds last year's weather and events. In the sample above it still put the effect at 8.3% against a true 3%.

Which outside data should a pilot analysis include?

Only sources that could plausibly move both where the pilot ran and the result: weather, the calendar, foot traffic, nearby competition and trade-area makeup. Industry benchmarks usually do not qualify, because they move with the whole market.

Upload one pilot's sales and reviews by store, with the launch date, and see how much of the lift survives the weather: start with one file. Plans start at $299 a month with a 14-day free trial; the join to sales and outside data and measured effects are on Prove and up.