Novelty Effect in A/B Testing: Why Early Lifts Decay
The novelty effect is a temporary lift (or drop) in an A/B test metric that comes from users noticing and reacting to the change itself, not from the change being genuinely better. It's biggest in the first days of a test and fades as regular users get used to the new version, which is why a test stopped early can report a win that mostly evaporates by week three.
Quick answer: Novelty effect happens because regular users react to something being different, and that reaction fades once the change stops feeling new. A test can show a +13% blended lift in week 1 and only +4% by week 4 with zero change in the true effect, because returning users' reaction decays while new users (who have no "before" to compare against) hold steady the whole time. Guard against it by splitting your results into new vs. returning user cohorts and running the test long enough for the returning-user trend to flatten, not just until the overall number crosses significance.
What is the novelty effect in A/B testing?
It's a short-term behavior change driven by the change being new, separate from any change in the underlying value of the feature. A redesigned button might get more clicks purely because it looks different and draws the eye, not because it's a better button, and that curiosity click rate drops back down once the new layout becomes the default mental model.
Why do early results look better than they really are?
Because your test population is a mix of new users (who have never seen either version, so there's nothing for them to react to as "new") and returning users (who have a baseline to compare against and notice the change immediately). The returning-user reaction inflates the blended metric early on, then fades, while the new-user number stays roughly flat throughout — so the overall lift trends downward even though nothing about the treatment changed.
Here's a worked example. Control conversion is 4.0% for new users and 5.0% for returning users, and new users make up 30% of test traffic. The true effect is a steady +8% relative lift for new users the whole time. Returning users start with a +15% novelty-driven lift in week 1 that decays to +2% by week 4:
| Cohort | Control rate | Week 1 rate | Week 1 lift | Week 4 rate | Week 4 lift |
|---|---|---|---|---|---|
| New users (30% of traffic) | 4.00% | 4.32% | +8.0% | 4.32% | +8.0% |
| Returning users (70% of traffic) | 5.00% | 5.75% | +15.0% | 5.10% | +2.0% |
| Blended (what you'd actually report) | 4.70% | 5.32% | +13.2% | 4.87% | +3.5% |
If you'd stopped after week 1, you'd have reported a +13.2% win. The number that actually reflects steady-state value is closer to the week 4 figure, and even that hasn't fully separated from the flat +8% true new-user effect yet.
How long should I run a test to rule out novelty?
Long enough for the returning-user trend line to flatten out, not just long enough to hit your sample-size target. As a starting rule of thumb, plan for at least two to four weeks for consumer-facing UI changes, since that covers multiple return visits for most usage patterns and gives the initial curiosity reaction time to fade.
The reliable check isn't a fixed calendar duration though, it's the shape of the trend: plot lift by week (or by day for high-frequency products) and keep the test running until the line stops declining. If week 3 and week 4 report roughly the same lift, you've likely found the steady-state effect; if it's still dropping, the number isn't stable yet.
How do I check for novelty effect in my data?
Split results by cohort: new users acquired during the test vs. users who already existed before it started. New users have no novelty to react to, so their lift is your best early read on the true effect. If the returning-user lift is well above the new-user lift and trending down over time, you're looking at novelty, not a durable win.
# Pandas: split lift by cohort and week
df['week'] = ((df['test_day'] - 1) // 7) + 1
df['cohort'] = df['first_seen_before_test'].map({True: 'returning', False: 'new'})
weekly = df.groupby(['cohort', 'week', 'variant'])['converted'].mean().unstack('variant')
weekly['lift_pct'] = (weekly['treatment'] / weekly['control'] - 1) * 100
print(weekly)What is the primacy effect?
The primacy effect is the mirror image of novelty: a change that requires relearning a habit or workflow initially looks worse than it really is, because regular users are momentarily slower or more error-prone with the unfamiliar version, and the metric recovers as they adjust. It shows up most with navigation redesigns, keyboard-shortcut changes, or anything that disrupts muscle memory built up under the old version.
The practical implication is the same as novelty, just flipped: if you stop a test early because week 1 looks like a loss, you might be killing a change that would have been a clear win by week 4 once users relearned the new pattern. Segmenting new vs. returning users catches this too, since new users never had the old habit to unlearn.
Common mistakes with novelty effect
- Stopping the moment overall lift crosses significance. Early significance is often driven disproportionately by the returning-user novelty spike, which is exactly the part of the effect that won't persist.
- Never segmenting new vs. returning users. The blended number alone can't tell you whether you're looking at a durable effect or a decaying one.
- Assuming novelty always means the true effect is smaller. Sometimes it's the reverse (primacy) — check the direction of the trend rather than assuming decay.
- Confusing novelty decay with regression to the mean. Novelty is a real behavioral reaction that fades over calendar time; regression to the mean is a statistical artifact of remeasuring an extreme group. They can co-occur, but the fixes differ.
Pro Tip: Build the new-vs-returning split into your experiment dashboard by default, not as a follow-up query you run when a result looks suspicious. Teams that only check cohorts after the fact tend to have already shipped several "wins" that were mostly novelty.
← Back to Data Analysis Tips