Sequential testing
Also called: Always-valid inference
Sequential testing allows results to be monitored continuously and stopped early without inflating false positives, using always-valid confidence intervals instead of a single fixed-horizon check. It formalises the peeking that teams do anyway.
Why it matters for ARPU
It is what makes continuous decisioning statistically legitimate: decisions can be made as soon as the evidence supports them, not only on a scheduled read-out date.
Related terms
A/B testAn A/B test randomly assigns customers to two or more variants and compares a pre-declared metric between them.Statistical powerStatistical power is the probability that a test detects an effect of a given size when that effect is real.Multi-armed banditA multi-armed bandit allocates traffic adaptively toward variants that are performing well while continuing to sample the others.ExperimentAn experiment is a controlled release of a candidate action against a randomised holdout, sized in advance so the result can distinguish a real effect from noise.