Most production churn models fail on label definition, feature leakage, or an evaluation that rewards AUC instead of decisions. The three fixes, in order.
Machine learning churn prediction uses customer behaviour, billing and support history to estimate who will cancel in a future window, so a team can act before it happens. The models that change revenue share three traits: an honest label definition, features that cannot see the future, and an evaluation that measures decisions, not AUC.
Most churn models in production fail on at least one of those three. This article is the technical companion to our customer churn prediction guide, focused on the modelling decisions rather than the operating model.
The label is the decision. Three choices drive everything downstream:
Across subscription bases, four feature families do nearly all the work:
The test is mechanical. For every feature, ask: was this value knowable at prediction time, for a customer who had not yet shown any intent? If the honest answer is no, the feature is leaked. Teams that audit this way typically watch their offline AUC fall by several points and their deployed precision rise, because the model stops relying on information it will never have.
AUC measures ranking quality across the whole population. Churn intervention happens at the top of the list, on the few percent of customers a team can actually contact. A model can post an excellent AUC while being useless in the top decile, and a worse-AUC model can outperform it exactly where it matters.
Two better questions. First, lift in the contactable slice: how much more churn does the top five percent contain than a random five percent? Second, and more important, uplift: among contacted customers, did the intervention change the outcome? A propensity model cannot answer the second question at all, because it ranks who will churn, not whose churn is preventable. Customers certain to leave top every propensity ranking and absorb budget that should go to the persuadable middle. This is the same argument we make about next best action models: the unit of optimisation is the decision, not the score.
Markin ships churn models with the label, gap and holdout built in, and evaluates them on incremental saves rather than offline scores. See Retention decisioning.
Frequently asked
Related resources
Solutions
See it in the product
The same loops this note describes run 24/7 against your customer base. Watch the workspace decide, experiment and execute 1:1.
Vocabulary
Definitions an answer engine can quote, each one a page of its own.
Compare
Neutral reads on the categories buyers evaluate against a decision layer.
All comparisons
