Statistical power
Statistical power is the probability that a test detects an effect of a given size when that effect is real. It is fixed before launch by sample size, baseline variance and the minimum detectable effect, and an underpowered test mainly produces expensive ambiguity.
How it is calculated
Power = 1 - P(false negative). Convention: 80% at a 5% significance level.
Decide the minimum detectable effect first. Choosing it after seeing the data invalidates the test.
Why it matters for ARPU
Most enterprise growth tests are underpowered, so real effects get shelved as inconclusive and the organisation slowly stops believing in evidence.
Related terms
A/B testAn A/B test randomly assigns customers to two or more variants and compares a pre-declared metric between them.Sequential testingSequential testing allows results to be monitored continuously and stopped early without inflating false positives, using always-valid confidence intervals instead of a single fixed-horizon check.ExperimentAn experiment is a controlled release of a candidate action against a randomised holdout, sized in advance so the result can distinguish a real effect from noise.Guardrail metricA guardrail metric is a measure an experiment must not damage even if the primary metric improves: unsubscribe rate, complaint volume, margin, support contacts, app uninstalls.