---
title: Designing a holdout that can actually detect ARPU lift
url: https://markin.ai/research/designing-a-holdout-that-can-actually-detect-arpu-lift
updated: 2026-09-10
author: Romà Llambés, CEO, Markin
cadence: Every report is reviewed twice a year, and re-run out of cycle whenever a source it rests on changes.
description: Sample size, minimum detectable effect and peeking risk for revenue experiments, with a published Monte Carlo: why heavy-tailed ARPU destroys naive sample-size rules.
---

# Designing a holdout that can actually detect ARPU lift

> Most ARPU experiments are not inconclusive. They were underpowered on the day they were designed, and nobody computed it.

## Headline

**4.6%.** A 200,000-customer base with a 10% holdout cannot reliably detect anything below a 4.6% ARPU lift. Most programmes are chasing 2%.

## The question

How large does a holdout have to be before it can actually detect the ARPU lift you are hoping for?

## Short answer

Revenue is heavy-tailed, so the sample size a conversion-rate calculator gives you is wrong by a wide margin. At a coefficient of variation of 2.2, detecting a 2% ARPU lift at 80% power needs roughly a million customers with a 10% holdout. At 200,000 the minimum detectable effect is 4.6%, and a real 2% lift is found only about a quarter of the time.

## Key takeaways

- Revenue is heavy-tailed, so conversion-rate sample-size rules understate what an ARPU experiment needs, usually by several times.
- At CV 2.2, detecting a 2% ARPU lift at 80% power needs roughly a million customers with a 10% holdout. At 200,000 the floor is 4.6%.
- Twenty interim looks turn a nominal 5% false positive rate into 26.8%. Weekly peeking is the single most damaging habit in growth measurement.
- Holdout returns flatten above 20%: going from 10% to 20% improves the MDE by half a point, and going to 50% adds almost nothing.
- CUPED is the cheapest power you can buy. At a pre-period correlation of 0.7 it roughly doubles the effective sample with no extra customers.
- Measure your own CV before designing anything. It moves the answer more than any other input on this page.

## Method

**Sample.** No customer data. Two components: a closed-form power calculation for the difference in arm means, and a Monte Carlo on simulated revenue, 2,000 replications per cell, 800 for the peeking table and 200 for the CUPED table. Revenue per customer is drawn from a zero-inflated lognormal, 35% zero spenders, calibrated to a mean of 12 currency units and to coefficients of variation of 1.4, 2.2 and 3.5. The grid covers base sizes of 50,000 to 5,000,000 and holdout shares of 2% to 50%.

**Period.** Simulation run on 14 September 2026, seed 20260914. The arithmetic is time-independent; the plausible CV range is the part that ages.

### Definitions that change the answer

- **Coefficient of variation (CV).** Standard deviation of per-customer revenue over the measurement window, divided by its mean. This single number, not the base size, is what makes revenue experiments hard. Consumer subscription bases sit near 1.4 when everyone pays a similar price, and above 3 wherever a small share of customers carries a large share of revenue.
- **Minimum detectable effect (MDE).** The smallest relative ARPU lift the design would find, with 80% power, at a two-sided 5% significance level. A result below the MDE is not evidence of no effect; it is evidence that the experiment could not see.
- **Power.** The probability of declaring a win when the lift is genuinely there. Power below 50% means a coin toss decides whether a real programme survives its own measurement.
- **Peeking.** Evaluating significance repeatedly while the experiment runs and stopping at the first significant result. It is not a small sin: it inflates the false positive rate far above the nominal 5%.
- **CUPED.** Controlled-experiment Using Pre-Experiment Data. Subtracting the part of in-period revenue predicted by the same customer's pre-period revenue. It removes variance rather than bias, so it buys sample size for free.

### How to reproduce this

1. Run scripts/research-holdout-sim.ts in this repository. Seed 20260914. Every table below is printed by that script.
2. Measure your own CV first: take per-customer revenue over a window the same length as your planned experiment, including customers who spent nothing, and divide the standard deviation by the mean.
3. Compute the MDE for your base and holdout share: MDE = (z(1 - alpha/2) + z(power)) x CV x sqrt(1/n_treated + 1/n_control).
4. Compare that MDE against the lift the programme is actually expected to produce. If the expected lift is below the MDE, the experiment is already decided and the decision is to redesign it, not to run it.
5. Fix the read date before launch and write it down. Then either hold to one read, or adopt a sequential procedure that controls error across looks.

### What this cannot tell you

- The simulated revenue distribution is a stylised zero-inflated lognormal, not your base. It reproduces the two features that matter, a large mass at zero and a long right tail, and nothing else about your business.
- The power calculation assumes independent customers. Household sharing, referral effects and shared accounts create correlation that inflates the true variance beyond what CV alone captures.
- Peeking is simulated only in its naive form, stopping at the first p below 0.05. The report names the corrections but does not quantify their operating characteristics.
- CUPED gains depend on the correlation between pre-period and in-period revenue in your own base. The simulated ratios show the shape of the relationship, not the gain you will get.
- Nothing here addresses bias. A perfectly powered experiment on a contaminated control still measures the wrong thing.

## The model

The relative lift in mean revenue per customer, treated against randomised control, over a fixed window: delta = (mean_T - mean_C) / mean_C.

- **Test statistic.** `t = (mean_T - mean_C) / sqrt(var_T/n_T + var_C/n_C)` A Welch two-sample comparison of arm means. Almost every growth team uses this estimator, whether or not it names it.
- **Minimum detectable effect.** `MDE = (z(1 - a/2) + z(1 - b)) x CV x sqrt(1/n_T + 1/n_C)` At a 5% two-sided level and 80% power the leading constant is 2.80. The CV term is why revenue experiments need far more customers than conversion experiments.
- **Cost of the holdout.** `C = N x h x w x ARPU x m x delta` Holdout share h, window w in months, margin m. It is only a cost when the programme actually works, which is the thing the holdout exists to find out.
- **CUPED adjustment.** `Y* = Y - theta x (X - mean(X)),  var(Y*) = var(Y) x (1 - rho^2)` X is pre-period revenue for the same customer, theta the least-squares coefficient. Effective sample size multiplies by 1 / (1 - rho^2).

### What has to hold

- Customers are randomised individually and independently, and the assignment holds for the whole window.
- Arm means are approximately normal by the central limit theorem. At the base sizes here that holds even for a heavy-tailed individual distribution, which the simulated false positive rate of 4.5% against a nominal 5% confirms.
- The treatment affects only treated customers. No spillover through shared households, referrals or word of mouth.
- The measurement window is fixed before launch and the same for both arms.

### What breaks it

- Extreme skew with a small holdout. A handful of very large spenders landing in one arm dominates the difference, and the normal approximation degrades exactly when the control arm is smallest.
- Repeated significance testing, which is the difference between a nominal 5% false positive rate and the 27% measured after twenty looks.
- Correlated customers, which makes the effective sample smaller than the headcount and every number in this report optimistic.
- Revenue outliers handled by ad hoc trimming after the result is seen. Trimming rules belong in the pre-registration.

## Uncertainty

Two sources. Power figures are closed-form and exact given CV. Simulated figures carry Monte Carlo error: at 2,000 per cell (800 for the peeking table, 200 for CUPED), a rate near 5% has a standard error of about 0.5 percentage points and a rate near 50% about 1.1 points. Figures are reported to one decimal because that is what the replication count supports, not because they are precise to one decimal.

- **MDE of 4.6% at n = 200,000:** 2.9% to 7.3%. Across CV values of 1.4 to 3.5. Measuring your own CV is therefore worth more than any other input refinement.
- **Power of 46.3% at a 3% true lift:** +/- 1.1 points of Monte Carlo error. The closed-form value is 44.8%, which is inside simulation error of the simulated result. The two methods agreeing is the point of running both.
- **False positive rate of 4.5% under the null:** +/- 0.5 points. Statistically indistinguishable from the nominal 5%. Heavy tails do not break the test at these base sizes; peeking does.
- **CUPED effective sample multiplier:** 1.1x at rho 0.3 to 5.3x at rho 0.9. Follows 1 / (1 - rho^2) closely in simulation. Your own rho is an afternoon of SQL, and it decides whether CUPED is a footnote or the largest single win available.

## Robustness

Each row changes one design or modelling choice and re-runs the calculation at the reference design: 200,000 customers, 10% holdout, CV 2.2.

| Change one assumption | What changes | Result | Conclusion |
| --- | --- | --- | --- |
| CV measured at 1.4 rather than assumed at 2.2 | Lower dispersion | MDE falls from 4.6% to 2.9%. The same base becomes adequate for a 3% programme. | Flips |
| Holdout raised from 10% to 20% | More control, less treated revenue | MDE improves from 2.05% to 1.54% on a million-customer base. Returns diminish sharply beyond 20%. | Holds |
| Holdout cut from 10% to 2% | Minimising forgone uplift | MDE worsens from 2.05% to 4.40%. The saving is real and the experiment stops working. | Flips |
| Window doubled, revenue per customer doubles, CV unchanged | Longer measurement | MDE unchanged. Time helps only by accumulating customers or by lowering CV, never by itself. | Holds |
| Weekly interim reads over five months | Twenty looks at the same data | False positive rate rises from 4.9% to 26.8%. One in four dead programmes is declared a winner. | Flips |
| CUPED applied with pre-period correlation of 0.7 | Variance reduction | Effective sample roughly doubles; the MDE falls by about 30%. No extra customers required. | Holds |

Three design choices decide whether an ARPU experiment can see anything: the CV you measured rather than assumed, the holdout share, and the number of times you look. The first two are settled before launch and the third is settled by discipline. None of them is a modelling question, which is why underpowered revenue experiments are a process failure rather than a statistical one.

## Findings

1. **4.6%.** At a coefficient of variation of 2.2, a 200,000-customer base with a 10% holdout has a minimum detectable effect of 4.6%. A real 2% lift is found 25.1% of the time. (Derived in this report) This is the central result and it is uncomfortable, because 2% to 3% is exactly the range most next-best-action programmes are aiming at, and 200,000 customers feels like a large base. The design is not marginal, it is short by a factor of five: a 2% lift needs roughly a million customers at the same CV and holdout share. Teams read a p-value of 0.4 as evidence the programme did nothing, when it is evidence the experiment could not have seen anything. **Do this.** Compute the MDE before launch, not the p-value after. If the expected lift is below the MDE, redesign the experiment or pool the window rather than running it.
2. **26.8%.** Twenty interim looks raise the false positive rate from 4.9% to 26.8% under a true null. Four looks already take it to 13.6%. (Derived in this report) Simulated by running the same null experiment with equally spaced interim analyses and stopping at the first p below 0.05. The damage starts immediately: two looks cost 7.5%, four cost 13.6%. In a dashboard culture where anyone can open the experiment daily, the effective number of looks is not a design choice anybody made. The remedies are well established, a single pre-registered read, an alpha-spending boundary, or an always-valid confidence sequence, and all of them require deciding before launch. **Do this.** Write the read date into the experiment brief and either hold to it or switch to a sequential procedure. Do not leave a live significance number on a dashboard.
3. **2.05%.** Holdout returns flatten fast. On a million-customer base at CV 2.2, moving from 2% to 10% holdout improves the MDE from 4.40% to 2.05%; moving from 20% to 50% improves it only from 1.54% to 1.23%. (Derived in this report) The precision comes from the smaller arm, so starving the control is the most expensive saving available. The cost of the holdout is also smaller than it feels: at $12 ARPU, 70% margin and a 3% lift, a 10% holdout on a million customers forgoes roughly $75,600 of margin over three months, against the cost of scaling a programme that does nothing across the entire base. Ten percent is the right default, twenty where the base allows, and below five only when the base is very large. **Do this.** Default to a 10% holdout. Price it explicitly with the formula in the model block so the conversation is about a number rather than about principle.
4. **1.96x.** CUPED buys more power than any sample increase available to most teams: at a pre-period correlation of 0.7, the effective sample roughly doubles. (Derived in this report) Simulated variance ratios follow 1 / (1 - rho^2) closely: 1.10x at rho 0.3, 1.33x at 0.5, 1.96x at 0.7 and 5.29x at 0.9. Subscription businesses with stable monthly revenue sit at the high end, which is precisely where the ARPU measurement problem is otherwise hardest. It removes variance, not bias, so it is safe in a way that outlier trimming is not. **Do this.** Compute the correlation between pre-period and in-period revenue per customer. Above 0.5, apply CUPED before adding a single customer to the experiment.
5. **4.5%.** The false positive rate of the naive test on heavy-tailed revenue is 4.5% against a nominal 5%. The distribution is not the problem. (Derived in this report) Worth stating plainly, because the usual reaction to skewed revenue is to reach for a rank test or a log transform, both of which change the estimand from mean revenue to something a finance team cannot use. At these base sizes the central limit theorem is doing its job on the arm means. The problem with revenue experiments is power and stopping rules, not validity. **Do this.** Keep testing the mean, which is the quantity the business cares about. Spend the effort on sample size, holdout share and stopping rules instead.

## Calculator

What can your base actually detect?. Enter your base size, holdout share and measured CV. Everything runs in your browser and nothing is sent anywhere. Available on the web version of this report at https://markin.ai/research/designing-a-holdout-that-can-actually-detect-arpu-lift#calculator.

## Data

### Minimum detectable relative ARPU lift, by base size and coefficient of variation. 10% holdout, two-sided alpha 0.05, 80% power.

| Base size | CV 1.4 | CV 2.2 | CV 3.5 |
| --- | --- | --- | --- |
| 50,000 | 5.85% | 9.19% | 14.62% |
| 200,000 | 2.92% | 4.59% | 7.31% |
| 1,000,000 | 1.31% | 2.05% | 3.27% |
| 5,000,000 | 0.58% | 0.92% | 1.46% |

Closed-form, not simulated. Find your own CV in the header row before reading a cell.

### What a holdout buys and what it costs. One million customers, CV 2.2, $12 monthly ARPU, 70% margin, a true 3% lift, three-month window.

| Holdout share | Minimum detectable effect | Margin forgone over the window |
| --- | --- | --- |
| 2% | 4.40% | $15,120 |
| 5% | 2.83% | $37,800 |
| 10% | 2.05% | $75,600 |
| 20% | 1.54% | $151,200 |
| 50% | 1.23% | $378,000 |

The cost column exists only if the programme works. If it does not, the holdout is what stops you scaling it, and the saving runs the other way.

### Simulated size and power. 200,000 customers, 10% holdout, CV 2.2, 2,000 replications per row, seed 20260914.

| True lift | Share of runs declared significant |
| --- | --- |
| 0% (null) | 4.5% |
| 1% | 10.5% |
| 2% | 25.1% |
| 3% | 46.3% |
| 5% | 84.9% |

The null row is the validity check: 4.5% against a nominal 5%. The rest is the power problem.

### The cost of looking. False positive rate under a true null, stopping at the first significant interim read. 200,000 customers, CV 2.2, 800 replications per row.

| Interim looks | False positive rate |
| --- | --- |
| 1 (single read) | 4.9% |
| 2 | 7.5% |
| 4 | 13.6% |
| 8 | 16.1% |
| 20 (weekly over five months) | 26.8% |

Naive peeking only. Alpha-spending boundaries and always-valid confidence sequences control this, and both must be chosen before launch.

### CUPED. Variance ratio and effective sample multiplier by pre-period correlation. 20,000 customers, CV 2.2, 200 replications per row.

| Pre-period correlation | Variance ratio | Effective sample multiplier |
| --- | --- | --- |
| 0.3 | 0.909 | 1.10x |
| 0.5 | 0.750 | 1.33x |
| 0.7 | 0.510 | 1.96x |
| 0.9 | 0.189 | 5.29x |

Simulated values track the analytic 1 / (1 - rho^2) closely, which is the check that the implementation is doing what the theory says.

## What this means for the person who has to act

- **Growth lead.** Put the minimum detectable effect in the experiment brief next to the expected lift. If the expected lift is below it, the experiment has already been decided and running it only buys a false negative.
- **Analytics.** Measure the coefficient of variation of per-customer revenue over your actual window, and the correlation with the pre-period. Those two numbers change experiment design more than any modelling choice available to you.
- **Finance.** Price the holdout explicitly rather than arguing about it. On the reference design it is about $75,600 of margin over a quarter, against the cost of scaling a programme that does nothing across the full base.
- **Executive sponsor.** Ask for the read date before launch and refuse to look before it. Twenty casual looks turn a 5% error rate into 27%, and no analysis after the fact can undo it.

## What would change our mind

- Evidence that per-customer revenue in large D2C bases is substantially less dispersed than the CV range of 1.4 to 3.5 used here. Every requirement in the report would fall proportionally.
- A variance-reduction technique that reliably beats CUPED on revenue outcomes without changing the estimand.
- A demonstration that correlation between customers, through households or referrals, is large enough that the independence assumption, not the CV, is the binding constraint.
- A sequential procedure whose simulated operating characteristics match a single fixed read while permitting continuous monitoring in practice rather than in principle.

## Sources, and what each one was used for

- [Derived in this report] Markin Monte Carlo, scripts/research-holdout-sim.ts, seed 20260914: Every simulated figure: size, power, peeking and CUPED tables. The script is published in full so the tables can be regenerated rather than trusted.
- [Derived in this report] Closed-form power calculation for a difference in means: The MDE tables and the holdout-share comparison.
- [Independent research] [BCG, How Measurement Is Evolving in Next-Best Action (2026)](https://www.bcg.com/publications/2026/measuring-incrementality-in-next-best-action-programs): The recommendation to read results over a full window rather than in the first weeks, which is the practical form of the peeking result here.
- [Anonymised Markin deployments] Markin deployments: The plausible CV range of 1.4 to 3.5 and the observation that first experiments are commonly designed against a 2% to 3% expected lift.

## Questions this report answers

- How big does a holdout need to be to measure ARPU lift?
- How do I calculate sample size for a revenue experiment?
- Why are my ARPU experiments always inconclusive?
- What is the minimum detectable effect for an A/B test on revenue?
- Is it bad to check experiment results early?
- What is CUPED and when should I use it?

## Act on this

- [Playbook](https://markin.ai/playbooks/retention-offer-before-renewal)
- [Guide: How Markin measures incrementality](https://markin.ai/methodology/incrementality-measurement)
- [Guide: The evidence standard behind every number we publish](https://markin.ai/trust/evidence-standards)

## FAQ

**How big does my holdout need to be to measure ARPU lift?**

It depends on your coefficient of variation, not on your base size alone. At CV 2.2, detecting a 2% lift at 80% power takes roughly a million customers with a 10% holdout; at 200,000 the floor is 4.6%. Measure your own CV first: standard deviation of per-customer revenue over the window, divided by the mean.

**Why are my revenue experiments always inconclusive?**

Almost always because they are underpowered rather than because the treatment did nothing. Revenue is heavy-tailed, so sample-size intuition borrowed from conversion tests understates the requirement several times over. Compute the minimum detectable effect before launch and compare it with the lift you expect.

**Is it really that bad to check results early?**

Yes. Simulated on a true null, stopping at the first significant interim read gives a 7.5% false positive rate after two looks, 13.6% after four and 26.8% after twenty. Either fix a single read date before launch or use an alpha-spending boundary or an always-valid confidence sequence.

**What does a holdout cost?**

N x holdout share x months x ARPU x margin x true lift. On a million customers with a 10% holdout, $12 ARPU, 70% margin and a 3% lift over three months, roughly $75,600 of margin. It is a cost only if the programme works, which is the thing the holdout exists to establish.

**Should I use a rank test or log-transform revenue because it is skewed?**

Usually no. The simulated false positive rate of the ordinary test on heavy-tailed revenue is 4.5% against a nominal 5%, so validity is not the problem. A rank test or a log transform changes the estimand to something other than mean revenue, which is the number the business decides on.

**What is CUPED and is it worth doing?**

CUPED subtracts the part of in-period revenue predicted by the same customer's pre-period revenue, removing variance without introducing bias. At a pre-period correlation of 0.7 the effective sample roughly doubles. For a subscription base with stable monthly revenue, it is the cheapest power available.

Source: https://markin.ai/research/designing-a-holdout-that-can-actually-detect-arpu-lift