Experiment
An experiment is a controlled release of a candidate action against a randomised holdout, sized in advance so the result can distinguish a real effect from noise. It has a pre-registered metric, a stopping rule and guardrails, and it produces an estimate of incremental revenue rather than a rate.
Why it matters for ARPU
Reported uplift that was never held out is usually seasonality or selection. Experiments are the only mechanism that converts activity into defensible ARPU growth.
Related terms
Holdout groupA holdout is a randomly selected set of customers deliberately excluded from an action, kept as the counterfactual.IncrementalityIncrementality is the portion of an outcome that would not have happened without the action.Statistical powerStatistical power is the probability that a test detects an effect of a given size when that effect is real.Guardrail metricA guardrail metric is a measure an experiment must not damage even if the primary metric improves: unsubscribe rate, complaint volume, margin, support contacts, app uninstalls.