A/B test
Also called: Split test, Randomised controlled trial
An A/B test randomly assigns customers to two or more variants and compares a pre-declared metric between them. Randomisation is what licenses a causal claim: any pre-existing difference between groups is distributed by chance rather than by selection.
Why it matters for ARPU
It remains the cheapest instrument for establishing that a growth action worked, and the pre-declared metric is what stops a null result from being retold as a success.
Related terms
Holdout groupA holdout is a randomly selected set of customers deliberately excluded from an action, kept as the counterfactual.Statistical powerStatistical power is the probability that a test detects an effect of a given size when that effect is real.Sequential testingSequential testing allows results to be monitored continuously and stopped early without inflating false positives, using always-valid confidence intervals instead of a single fixed-horizon check.Guardrail metricA guardrail metric is a measure an experiment must not damage even if the primary metric improves: unsubscribe rate, complaint volume, margin, support contacts, app uninstalls.