---
title: What is A/B test?
url: https://markin.ai/glossary/ab-test
category: Experimentation and causality
aliases: Split test, Randomised controlled trial
---

# A/B test

> An A/B test randomly assigns customers to two or more variants and compares a pre-declared metric between them. Randomisation is what licenses a causal claim: any pre-existing difference between groups is distributed by chance rather than by selection.

## Why it matters for ARPU

It remains the cheapest instrument for establishing that a growth action worked, and the pre-declared metric is what stops a null result from being retold as a success.

## Related terms

- [Holdout group](https://markin.ai/glossary/holdout-group), A holdout is a randomly selected set of customers deliberately excluded from an action, kept as the counterfactual.
- [Statistical power](https://markin.ai/glossary/statistical-power), Statistical power is the probability that a test detects an effect of a given size when that effect is real.
- [Sequential testing](https://markin.ai/glossary/sequential-testing), Sequential testing allows results to be monitored continuously and stopped early without inflating false positives, using always-valid confidence intervals instead of a single fixed-horizon check.
- [Guardrail metric](https://markin.ai/glossary/guardrail-metric), A guardrail metric is a measure an experiment must not damage even if the primary metric improves: unsubscribe rate, complaint volume, margin, support contacts, app uninstalls.

## Go deeper

- [Experimentation vs continuous decisioning](https://markin.ai/compare/experimentation-vs-continuous-decisioning), Tests versus a running loop.
- [Growth optimization](https://markin.ai/solutions/growth-optimization), Experimentation at scale.

Source: https://markin.ai/glossary/ab-test