Designing a holdout that can actually detect ARPU lift
Most ARPU experiments are not inconclusive. They were underpowered on the day they were designed, and nobody computed it.
4.6%
A 200,000-customer base with a 10% holdout cannot reliably detect anything below a 4.6% ARPU lift. Most programmes are chasing 2%.
Key takeaways
- 01Revenue is heavy-tailed, so conversion-rate sample-size rules understate what an ARPU experiment needs, usually by several times.
- 02At CV 2.2, detecting a 2% ARPU lift at 80% power needs roughly a million customers with a 10% holdout. At 200,000 the floor is 4.6%.
- 03Twenty interim looks turn a nominal 5% false positive rate into 26.8%. Weekly peeking is the single most damaging habit in growth measurement.
- 04Holdout returns flatten above 20%: going from 10% to 20% improves the MDE by half a point, and going to 50% adds almost nothing.
- 05CUPED is the cheapest power you can buy. At a pre-period correlation of 0.7 it roughly doubles the effective sample with no extra customers.
- 06Measure your own CV before designing anything. It moves the answer more than any other input on this page.
Romà Llambés · CEO, Markin · Reviewed 2026-09-10 · Every report is reviewed twice a year, and re-run out of cycle whenever a source it rests on changes. · Method and sources
The question
How large does a holdout have to be before it can actually detect the ARPU lift you are hoping for?
Revenue is heavy-tailed, so the sample size a conversion-rate calculator gives you is wrong by a wide margin. At a coefficient of variation of 2.2, detecting a 2% ARPU lift at 80% power needs roughly a million customers with a 10% holdout. At 200,000 the minimum detectable effect is 4.6%, and a real 2% lift is found only about a quarter of the time.
- Sample
- No customer data. Two components: a closed-form power calculation for the difference in arm means, and a Monte Carlo on simulated revenue, 2,000 replications per cell, 800 for the peeking table and 200 for the CUPED table. Revenue per customer is drawn from a zero-inflated lognormal, 35% zero spenders, calibrated to a mean of 12 currency units and to coefficients of variation of 1.4, 2.2 and 3.5. The grid covers base sizes of 50,000 to 5,000,000 and holdout shares of 2% to 50%.
- Period
- Simulation run on 14 September 2026, seed 20260914. The arithmetic is time-independent; the plausible CV range is the part that ages.
- Reviewed
- 2026-09-10Reviewed twice a year
- 4.6%
- At a coefficient of variation of 2.2, a 200,000-customer base with a 10% holdout has a minimum detectable effect of 4.6%. A real 2% lift is found 25.1% of the time.
- 26.8%
- Twenty interim looks raise the false positive rate from 4.9% to 26.8% under a true null. Four looks already take it to 13.6%.
- 2.05%
- Holdout returns flatten fast. On a million-customer base at CV 2.2, moving from 2% to 10% holdout improves the MDE from 4.40% to 2.05%; moving from 20% to 50% improves it only from 1.54% to 1.23%.
Findings
What we found
- 014.6%Derived in this report
At a coefficient of variation of 2.2, a 200,000-customer base with a 10% holdout has a minimum detectable effect of 4.6%. A real 2% lift is found 25.1% of the time.
This is the central result and it is uncomfortable, because 2% to 3% is exactly the range most next-best-action programmes are aiming at, and 200,000 customers feels like a large base. The design is not marginal, it is short by a factor of five: a 2% lift needs roughly a million customers at the same CV and holdout share. Teams read a p-value of 0.4 as evidence the programme did nothing, when it is evidence the experiment could not have seen anything.
Do thisCompute the MDE before launch, not the p-value after. If the expected lift is below the MDE, redesign the experiment or pool the window rather than running it.
- 0226.8%Derived in this report
Twenty interim looks raise the false positive rate from 4.9% to 26.8% under a true null. Four looks already take it to 13.6%.
Simulated by running the same null experiment with equally spaced interim analyses and stopping at the first p below 0.05. The damage starts immediately: two looks cost 7.5%, four cost 13.6%. In a dashboard culture where anyone can open the experiment daily, the effective number of looks is not a design choice anybody made. The remedies are well established, a single pre-registered read, an alpha-spending boundary, or an always-valid confidence sequence, and all of them require deciding before launch.
Do thisWrite the read date into the experiment brief and either hold to it or switch to a sequential procedure. Do not leave a live significance number on a dashboard.
- 032.05%Derived in this report
Holdout returns flatten fast. On a million-customer base at CV 2.2, moving from 2% to 10% holdout improves the MDE from 4.40% to 2.05%; moving from 20% to 50% improves it only from 1.54% to 1.23%.
The precision comes from the smaller arm, so starving the control is the most expensive saving available. The cost of the holdout is also smaller than it feels: at $12 ARPU, 70% margin and a 3% lift, a 10% holdout on a million customers forgoes roughly $75,600 of margin over three months, against the cost of scaling a programme that does nothing across the entire base. Ten percent is the right default, twenty where the base allows, and below five only when the base is very large.
Do thisDefault to a 10% holdout. Price it explicitly with the formula in the model block so the conversation is about a number rather than about principle.
- 041.96xDerived in this report
CUPED buys more power than any sample increase available to most teams: at a pre-period correlation of 0.7, the effective sample roughly doubles.
Simulated variance ratios follow 1 / (1 - rho^2) closely: 1.10x at rho 0.3, 1.33x at 0.5, 1.96x at 0.7 and 5.29x at 0.9. Subscription businesses with stable monthly revenue sit at the high end, which is precisely where the ARPU measurement problem is otherwise hardest. It removes variance, not bias, so it is safe in a way that outlier trimming is not.
Do thisCompute the correlation between pre-period and in-period revenue per customer. Above 0.5, apply CUPED before adding a single customer to the experiment.
- 054.5%Derived in this report
The false positive rate of the naive test on heavy-tailed revenue is 4.5% against a nominal 5%. The distribution is not the problem.
Worth stating plainly, because the usual reaction to skewed revenue is to reach for a rank test or a log transform, both of which change the estimand from mean revenue to something a finance team cannot use. At these base sizes the central limit theorem is doing its job on the arm means. The problem with revenue experiments is power and stopping rules, not validity.
Do thisKeep testing the mean, which is the quantity the business cares about. Spend the effort on sample size, holdout share and stopping rules instead.
Charts and tables
The data
- 50,0009.19%
- 200,0004.59%
- 1,000,0002.05%
- 5,000,0000.92%
The highlighted bar is the base size at which most teams believe they are running a well-powered revenue experiment.
| Base size | CV 1.4 | CV 2.2 | CV 3.5 |
|---|---|---|---|
| 50,000 | 5.85% | 9.19% | 14.62% |
| 200,000 | 2.92% | 4.59% | 7.31% |
| 1,000,000 | 1.31% | 2.05% | 3.27% |
| 5,000,000 | 0.58% | 0.92% | 1.46% |
| Holdout share | Minimum detectable effect | Margin forgone over the window |
|---|---|---|
| 2% | 4.40% | $15,120 |
| 5% | 2.83% | $37,800 |
| 10% | 2.05% | $75,600 |
| 20% | 1.54% | $151,200 |
| 50% | 1.23% | $378,000 |
| True lift | Share of runs declared significant |
|---|---|
| 0% (null) | 4.5% |
| 1% | 10.5% |
| 2% | 25.1% |
| 3% | 46.3% |
| 5% | 84.9% |
| Interim looks | False positive rate |
|---|---|
| 1 (single read) | 4.9% |
| 2 | 7.5% |
| 4 | 13.6% |
| 8 | 16.1% |
| 20 (weekly over five months) | 26.8% |
| Pre-period correlation | Variance ratio | Effective sample multiplier |
|---|---|---|
| 0.3 | 0.909 | 1.10x |
| 0.5 | 0.750 | 1.33x |
| 0.7 | 0.510 | 1.96x |
| 0.9 | 0.189 | 5.29x |
Run it on your numbers
What can your base actually detect?
Enter your base size, holdout share and measured CV. Everything runs in your browser and nothing is sent anywhere.
Measure the coefficient of variation on your own base: standard deviation of per-customer revenue over a window the length of the experiment, including customers who spent nothing, divided by the mean.
Minimum detectable effect
4.59%
The lift you expect is below the floor this design can see. A null result would tell you nothing, so redesign before launching.
- Power at the lift you expect
- 44.8%
- Treated
- 180,000
- Control
- 20,000
Two-sided alpha 0.05, 80% power. Assumes one read at a fixed date.
Implications
What this means for the person who has to act
Growth lead
Put the minimum detectable effect in the experiment brief next to the expected lift. If the expected lift is below it, the experiment has already been decided and running it only buys a false negative.
Analytics
Measure the coefficient of variation of per-customer revenue over your actual window, and the correlation with the pre-period. Those two numbers change experiment design more than any modelling choice available to you.
Finance
Price the holdout explicitly rather than arguing about it. On the reference design it is about $75,600 of margin over a quarter, against the cost of scaling a programme that does nothing across the full base.
Executive sponsor
Ask for the read date before launch and refuse to look before it. Twenty casual looks turn a 5% error rate into 27%, and no analysis after the fact can undo it.
Method
Definitions that change the answer
- Coefficient of variation (CV)
- Standard deviation of per-customer revenue over the measurement window, divided by its mean. This single number, not the base size, is what makes revenue experiments hard. Consumer subscription bases sit near 1.4 when everyone pays a similar price, and above 3 wherever a small share of customers carries a large share of revenue.
- Minimum detectable effect (MDE)
- The smallest relative ARPU lift the design would find, with 80% power, at a two-sided 5% significance level. A result below the MDE is not evidence of no effect; it is evidence that the experiment could not see.
- Power
- The probability of declaring a win when the lift is genuinely there. Power below 50% means a coin toss decides whether a real programme survives its own measurement.
- Peeking
- Evaluating significance repeatedly while the experiment runs and stopping at the first significant result. It is not a small sin: it inflates the false positive rate far above the nominal 5%.
- CUPED
- Controlled-experiment Using Pre-Experiment Data. Subtracting the part of in-period revenue predicted by the same customer's pre-period revenue. It removes variance rather than bias, so it buys sample size for free.
Estimator
The model, written out
The relative lift in mean revenue per customer, treated against randomised control, over a fixed window: delta = (mean_T - mean_C) / mean_C.
Test statistic
t = (mean_T - mean_C) / sqrt(var_T/n_T + var_C/n_C)
A Welch two-sample comparison of arm means. Almost every growth team uses this estimator, whether or not it names it.
Minimum detectable effect
MDE = (z(1 - a/2) + z(1 - b)) x CV x sqrt(1/n_T + 1/n_C)
At a 5% two-sided level and 80% power the leading constant is 2.80. The CV term is why revenue experiments need far more customers than conversion experiments.
Cost of the holdout
C = N x h x w x ARPU x m x delta
Holdout share h, window w in months, margin m. It is only a cost when the programme actually works, which is the thing the holdout exists to find out.
CUPED adjustment
Y* = Y - theta x (X - mean(X)), var(Y*) = var(Y) x (1 - rho^2)
X is pre-period revenue for the same customer, theta the least-squares coefficient. Effective sample size multiplies by 1 / (1 - rho^2).
What has to hold
- Customers are randomised individually and independently, and the assignment holds for the whole window.
- Arm means are approximately normal by the central limit theorem. At the base sizes here that holds even for a heavy-tailed individual distribution, which the simulated false positive rate of 4.5% against a nominal 5% confirms.
- The treatment affects only treated customers. No spillover through shared households, referrals or word of mouth.
- The measurement window is fixed before launch and the same for both arms.
What breaks it
- Extreme skew with a small holdout. A handful of very large spenders landing in one arm dominates the difference, and the normal approximation degrades exactly when the control arm is smallest.
- Repeated significance testing, which is the difference between a nominal 5% false positive rate and the 27% measured after twenty looks.
- Correlated customers, which makes the effective sample smaller than the headcount and every number in this report optimistic.
- Revenue outliers handled by ad hoc trimming after the result is seen. Trimming rules belong in the pre-registration.
Method
How to reproduce this
- 01
Run scripts/research-holdout-sim.ts in this repository. Seed 20260914. Every table below is printed by that script.
- 02
Measure your own CV first: take per-customer revenue over a window the same length as your planned experiment, including customers who spent nothing, and divide the standard deviation by the mean.
- 03
Compute the MDE for your base and holdout share: MDE = (z(1 - alpha/2) + z(power)) x CV x sqrt(1/n_treated + 1/n_control).
- 04
Compare that MDE against the lift the programme is actually expected to produce. If the expected lift is below the MDE, the experiment is already decided and the decision is to redesign it, not to run it.
- 05
Fix the read date before launch and write it down. Then either hold to one read, or adopt a sequential procedure that controls error across looks.
Uncertainty
How wrong each figure could be
Two sources. Power figures are closed-form and exact given CV. Simulated figures carry Monte Carlo error: at 2,000 per cell (800 for the peeking table, 200 for CUPED), a rate near 5% has a standard error of about 0.5 percentage points and a rate near 50% about 1.1 points. Figures are reported to one decimal because that is what the replication count supports, not because they are precise to one decimal.
MDE of 4.6% at n = 200,000
Across CV values of 1.4 to 3.5. Measuring your own CV is therefore worth more than any other input refinement.
2.9% to 7.3%
Power of 46.3% at a 3% true lift
The closed-form value is 44.8%, which is inside simulation error of the simulated result. The two methods agreeing is the point of running both.
+/- 1.1 points of Monte Carlo error
False positive rate of 4.5% under the null
Statistically indistinguishable from the nominal 5%. Heavy tails do not break the test at these base sizes; peeking does.
+/- 0.5 points
CUPED effective sample multiplier
Follows 1 / (1 - rho^2) closely in simulation. Your own rho is an afternoon of SQL, and it decides whether CUPED is a footnote or the largest single win available.
1.1x at rho 0.3 to 5.3x at rho 0.9
Robustness
What survives a different assumption
Each row changes one design or modelling choice and re-runs the calculation at the reference design: 200,000 customers, 10% holdout, CV 2.2.
| Change one assumption | What changes | Result | Conclusion |
|---|---|---|---|
| CV measured at 1.4 rather than assumed at 2.2 | Lower dispersion | MDE falls from 4.6% to 2.9%. The same base becomes adequate for a 3% programme. | Flips |
| Holdout raised from 10% to 20% | More control, less treated revenue | MDE improves from 2.05% to 1.54% on a million-customer base. Returns diminish sharply beyond 20%. | Holds |
| Holdout cut from 10% to 2% | Minimising forgone uplift | MDE worsens from 2.05% to 4.40%. The saving is real and the experiment stops working. | Flips |
| Window doubled, revenue per customer doubles, CV unchanged | Longer measurement | MDE unchanged. Time helps only by accumulating customers or by lowering CV, never by itself. | Holds |
| Weekly interim reads over five months | Twenty looks at the same data | False positive rate rises from 4.9% to 26.8%. One in four dead programmes is declared a winner. | Flips |
| CUPED applied with pre-period correlation of 0.7 | Variance reduction | Effective sample roughly doubles; the MDE falls by about 30%. No extra customers required. | Holds |
Three design choices decide whether an ARPU experiment can see anything: the CV you measured rather than assumed, the holdout share, and the number of times you look. The first two are settled before launch and the third is settled by discipline. None of them is a modelling question, which is why underpowered revenue experiments are a process failure rather than a statistical one.
Limits
What this cannot tell you
- The simulated revenue distribution is a stylised zero-inflated lognormal, not your base. It reproduces the two features that matter, a large mass at zero and a long right tail, and nothing else about your business.
- The power calculation assumes independent customers. Household sharing, referral effects and shared accounts create correlation that inflates the true variance beyond what CV alone captures.
- Peeking is simulated only in its naive form, stopping at the first p below 0.05. The report names the corrections but does not quantify their operating characteristics.
- CUPED gains depend on the correlation between pre-period and in-period revenue in your own base. The simulated ratios show the shape of the relationship, not the gain you will get.
- Nothing here addresses bias. A perfectly powered experiment on a contaminated control still measures the wrong thing.
Falsifiability
What would change our mind
- 01
Evidence that per-customer revenue in large B2C bases is substantially less dispersed than the CV range of 1.4 to 3.5 used here. Every requirement in the report would fall proportionally.
- 02
A variance-reduction technique that reliably beats CUPED on revenue outcomes without changing the estimand.
- 03
A demonstration that correlation between customers, through households or referrals, is large enough that the independence assumption, not the CV, is the binding constraint.
- 04
A sequential procedure whose simulated operating characteristics match a single fixed read while permitting continuous monitoring in practice rather than in principle.
Provenance
Sources, and what each one was used for
- Derived in this reportMarkin Monte Carlo, scripts/research-holdout-sim.ts, seed 20260914
Every simulated figure: size, power, peeking and CUPED tables. The script is published in full so the tables can be regenerated rather than trusted.
- Derived in this reportClosed-form power calculation for a difference in means
The MDE tables and the holdout-share comparison.
- Independent researchBCG, How Measurement Is Evolving in Next-Best Action (2026)
The recommendation to read results over a full window rather than in the first weeks, which is the practical form of the peeking result here.
- Anonymised Markin deploymentsMarkin deployments
The plausible CV range of 1.4 to 3.5 and the observation that first experiments are commonly designed against a 2% to 3% expected lift.
Questions this report answers
- How big does a holdout need to be to measure ARPU lift?
- How do I calculate sample size for a revenue experiment?
- Why are my ARPU experiments always inconclusive?
- What is the minimum detectable effect for an A/B test on revenue?
- Is it bad to check experiment results early?
- What is CUPED and when should I use it?
Frequently asked
- How big does my holdout need to be to measure ARPU lift?
- It depends on your coefficient of variation, not on your base size alone. At CV 2.2, detecting a 2% lift at 80% power takes roughly a million customers with a 10% holdout; at 200,000 the floor is 4.6%. Measure your own CV first: standard deviation of per-customer revenue over the window, divided by the mean.
- Why are my revenue experiments always inconclusive?
- Almost always because they are underpowered rather than because the treatment did nothing. Revenue is heavy-tailed, so sample-size intuition borrowed from conversion tests understates the requirement several times over. Compute the minimum detectable effect before launch and compare it with the lift you expect.
- Is it really that bad to check results early?
- Yes. Simulated on a true null, stopping at the first significant interim read gives a 7.5% false positive rate after two looks, 13.6% after four and 26.8% after twenty. Either fix a single read date before launch or use an alpha-spending boundary or an always-valid confidence sequence.
- What does a holdout cost?
- N x holdout share x months x ARPU x margin x true lift. On a million customers with a 10% holdout, $12 ARPU, 70% margin and a 3% lift over three months, roughly $75,600 of margin. It is a cost only if the programme works, which is the thing the holdout exists to establish.
- Should I use a rank test or log-transform revenue because it is skewed?
- Usually no. The simulated false positive rate of the ordinary test on heavy-tailed revenue is 4.5% against a nominal 5%, so validity is not the problem. A rank test or a log transform changes the estimand to something other than mean revenue, which is the number the business decides on.
- What is CUPED and is it worth doing?
- CUPED subtracts the part of in-period revenue predicted by the same customer's pre-period revenue, removing variance without introducing bias. At a pre-period correlation of 0.7 the effective sample roughly doubles. For a subscription base with stable monthly revenue, it is the cheapest power available.
Act on this
- PlaybookRun a retention offer before renewal without paying customers who were staying anywayPre-renewal save offers on Stripe, sized by risk instead of by list membership.
- ResearchFrom campaigns to per-customer decisionsThe blocker is almost never the model. It is identity, latency, write-back, a margin number and the willingness to withhold.
- ResearchThe retention discount lineMost save offers are evaluated on how many customers stayed. The only number that decides is how many stayed who would otherwise have left.
- GuideHow Markin measures incrementalityThe holdout design, the measurement window and the stopping rules Markin uses before any result is allowed to scale. Written to be checked.
- GuideThe evidence standard behind every number we publishHow Markin decides what it is allowed to claim in public: the claim register, mandatory qualifiers, review dates, and the claims we refuse to make.