Your MMM is probably wrong: what we learned calibrating with geolifts

We measured a retailer's affiliate channel with three tools over the same period. GA4 credited it with 50% of sales. The MMM gave it 30%. A geo holdout, where we switched affiliates off in regions comparable to the rest of the country, measured 9% incremental.

Affiliates were about 40% of that account's spend. Three answers that far apart on the biggest channel are not a technical detail. They are the difference between raising its budget and cutting it.

With real data there is no way to know which of the three to believe, because nobody knows the true ROAS. So we built a lab: simulated businesses where we set the answer, to see how close each MMM gets and what brings it closer. What we found, in one sentence: in every world we tested, running experiments mattered more than choosing the model.

A lab where we know the answer

An MMM can fit sales well, converge without trouble, and still be badly wrong on every channel's ROAS. The usual diagnostics tell you whether the model converged, not whether the ROAS is right. Before you trust an instrument, you test it where you already know the result.

Every simulated business goes through four steps, and the model only sees the third:

1. Build the business. Margins from 10% to 80%, a fixed ROAS per channel (from channels that never pay back their cost up to 4× breakeven), saturation and carryover.
2. Simulate the decisions. Planners raise spend when demand is coming, as they do in real life. There is price, promotion and seasonality.
3. Observe with noise. Weekly sales and spend with 5% measurement error. That is all the model gets.
4. Grade against the truth. We compare each MMM's ROAS and budget plan against the real ones.

We ran 7 worlds with 20 businesses each, more than 4,500 fits in total. What separates the three main worlds is how closely spend follows demand. In clean room it does not. In real world it does, and there are demand shocks the model cannot see. In adversarial spend almost copies demand. Four more worlds (dominant pricing, channel interactions, only 39 weeks of data, and a regime change halfway through the series) were held back for the final test, to catch overfitted models.

The obvious objection is that we built the benchmark for the models to fail. We did not: a model that knows the exact shape of the business recovers the truth in clean room, with a bias of +0.08. The data can be learned.

Three metrics

• ROAS bias: estimated ROAS minus true ROAS, in points. A bias of +7.5 means a channel with a true ROAS of 2 shows up as 9.5. To pass, bias has to land within ±0.3.
• Coverage: how often the 80% interval contains the true ROAS. We ask for at least 0.60. A narrow, wrong interval is worse than a wide, honest one.
• Regret: the one that matters for decisions. With the model's ROAS we build a plan that moves each channel's budget by up to ±30%. Because we know the real business, we know what that plan sells and what the optimal plan would sell. Regret is what was left on the table divided by the gain available. If the optimal plan adds $100 and the model's plan captures $79, regret is 0.21. The limit is 0.5.

A regret above 1 is worse than leaving money on the table: following the model sells less than changing nothing.

Popular MMMs fail, even well configured

On default settings, no popular MMM passed in real world. Then we did what a good analyst would: priors centred on breakeven from the margin, longer chains, convergence checks. Bias fell by about a third, and still none passed.

Median ROAS bias in points, real world, 20 businesses per model, second round of experiments. The pass limit is ±0.3.

In money: with a bias of +7.5, a channel that returns $200 for every $100 spent shows up returning $950. On that read you raise the budget of a channel that, at a 50% margin, barely pays for itself.

Meridian got worse when we tuned it. Its default prior was working as shrinkage by accident, and centring it on the business's economics took that brake away.

Robyn came out with the bias closest to zero (between −0.4 and −1.0), and it is easy to read that as working better. The reason is mechanical: its ridge regression shrinks every effect towards zero, and that offsets the upward bias confounding puts in. On average it lands close; channel by channel it is still wrong, it gives no intervals, and its budget plan was the worst of all, with a regret of 1.18. Following it sold less than changing nothing.

Two suspects: structure and confounding

An additive model adds the media effect on top of the base: the same spend sells the same in January as in December. A multiplicative one scales it with demand: the same spend sells more when demand is already high. If the business is multiplicative and the model assumes additive, the model has nowhere to put the extra peak-season sales, so it hands them to media, which also goes up in those weeks.

Switching only the likelihood from additive to multiplicative cut clean-room bias from +3.4 to +1.3. We built our own model on that idea. Then we pre-registered a test in worlds with a different structure, with the criterion that our model should not be punished more than PyMC outside its own shape. It failed: its bias worsened by +1.56 against +1.10 for PyMC. So we downgraded the claim to “specific to this benchmark”.

The second suspect has no fix inside the model. The correctly specified model, whose forward model is identical to the data generator, has a bias of +0.08 in clean room and +2.6 in real world. When spend follows demand, not even the perfect model finds the true ROAS. What it needs is evidence from outside the data: an experiment.

Calibrating with geolifts fixes most of the bias

With five geolifts, PyMC's bias in real world fell from +7.6 to +0.88. The simulated lifts are noisy on purpose (±30% at 2σ), like real ones, with a 4-week window and a 4-week cooldown.

Median ROAS bias, real world, 20 businesses. The lifts cover the highest-spend channels, in order.

The other models improve too, though not equally. NumPyro went from +5.3 to +1.5 with five lifts. Meridian went from +6.1 to +4.3, because its calibration enters as a per-channel ROI prior rather than as a likelihood: the calibration mechanism matters. In clean room, with five lifts, PyMC and Meridian land inside the limit (+0.14 and +0.12).

The balloon effect

With a single lift, the tested channel is measured well and the bias moves to the channels you did not test. Squeeze one side of the balloon and the other side swells. With three lifts the curve bends and regret drops below the limit for NumPyro and PyMC. With five, the highest-spend channels have an anchor and the bias has nowhere left to go. This is exactly what happened with affiliates in the real case at the top.

The pre-registered test

For the final test we froze the criteria before running, used a new seed, and added the four worlds no model had seen: 7 scenarios, 13 arms, 1,820 fits and 0 errors. Our registered bet was that our model would win. It did not.

No model passed uncalibrated in any scenario. With five lifts, bias fell between 70% and 90% for every model family. Our model was the best uncalibrated and the most consistent in the new worlds, but it did not beat a default PyMC with five lifts.

Two caveats. In adversarial, calibration helps the decision but worsens the bias on untested channels. In the dominant-pricing world nobody passes, not even the perfect model.

We also tested calibration in worlds built to prove us wrong: one additive, one with noisy response curves. With five lifts, regret came in at 0.032 and 0.084 (limit 0.25). The claim about structure did not survive that test; the one about calibration did.

How you run the geolift matters almost as much as running it

A well-measured geolift has three stages:

1. Warm-up. You switch spend on (or off, in a holdout) and the incremental effect takes time to build because of carryover.
2. Convergence. The effect reaches its plateau. That level is the lift you want to measure, so you wait to see it before you read.
3. Cooldown. You end the intervention and measure how long the effect takes to fade.

With a carryover of 0.6 per week, if you stop during warm-up you see about 67% of the real effect and the channel looks worse than it is. In simulation, short reads underestimated the lift by 31% (median).

The cooldown also measures adstock. If each week without spend keeps about 60% of the previous week's effect, carryover is 0.6, and that number goes into the MMM as a prior instead of being guessed from the historical series.

Even so, an imperfect lift is much better than none. In an exploratory test with 6 businesses per world, real-world regret came out like this:

With badly read lifts, regret still came in 2× to 3.5× lower than with no lifts at all. The sample is small, so this shows the direction and is less solid than the pre-registered tests.

Which channel to test first

We built a sophisticated selector, Next Best Geolift (NBG), to choose the next test. It combines four signals: how uncertain it is whether the channel sits above or below breakeven, the probability that two channels are in the wrong order, which channels the model cannot separate from each other, and a power model that says how many weeks the test needs.

We compared it against a simple rule: spend × the width of the ROAS interval. In other words, test first the channel with the most money at stake where the model could be most wrong. The protocol, frozen before running, required NBG to tie or beat the simple rule in both worlds after one lift.

Pre-registered, 10 businesses per world, 140 fits. NBG lost in both worlds and the random channel came last, so the test does discriminate.

One business from the validation shows why. The simple rule picked a channel with 16% of spend and a very wide ROAS range: the model saw it at 5.0 and it actually returned 9.35, the best in the portfolio. The test fixed the most expensive mistake and regret fell from 0.56 to 0.29. NBG picked the highest-spend channel, whose true ROAS (2.1) was already inside the model's interval. The test confirmed what the model already knew and regret stayed at 0.56. When every channel is ambiguous, NBG ends up choosing by spend and ignores how wide the uncertainty is, which is exactly what the simple rule uses.

We kept the simple rule. From NBG we kept what does work: how many weeks the test needs and which channel it is tangled up with.

What real geolifts show

On real data, an uncalibrated MMM is good for ranking channels, not for measuring their ROAS. We ran it uncalibrated on six real clients. Three never reached a stable answer because they had too little data. The three that converged had 126 weeks or more, and their ranking of contribution to sales made sense, probably because the contribution the model assigns tracks spend itself closely.

What you can measure, you measure with experiments.

Retailer C: back to affiliates

The affiliate holdout measured a drop of between 8.7% and 11.3% in sales when the channel was off, depending on the control pool. We use 9%, the conservative read. Affiliates do work, but the MMM gave them more than three times what the experiment measured, and GA4 more than five.

In ROAS, with affiliates at about 40% of spend: GA4 implied 1.25× the account's average ROAS, the MMM 0.75×, and the holdout 0.23×. The channel returned less than a third of what the MMM said.

The MMM was calibrated, but on other channels. Affiliates had no anchor and soaked up the bias: the balloon effect, in a real account. The inefficiency we found is worth several hundred million pesos.

Four rules for calibrating an MMM

1. Run at least three lifts before you trust any absolute ROAS. With only one, the bias moves to the channels you did not test.
2. Calibrate first where the spend and the uncertainty are. That simple rule beat our sophisticated selector in a pre-registered test, and it is the one that would have caught the affiliate mistake.
3. Fix the duration before you start, and read with a cooldown window. Reading during warm-up underestimates the lift and costs you the adstock measurement.
4. Run a recurring lift every quarter. Channels and demand change, and calibration has to keep up.

We built our own model expecting structure to be the answer. In the test we designed ourselves, a default PyMC with five lifts beat it. In every world we tested, running experiments mattered more than choosing the model.

This article summarises the talk Entropy Labs gave at MeasureCamp 2026.