How to run incrementality testing on lifecycle marketing

Running incrementality testing on lifecycle marketing requires a structured approach to validate campaign effectiveness and prove true causation. This process involves isolating a control group (holdout group or ghost cell) that receives no intervention, then comparing its performance against a group that receives the Candidate action. This quantifies the precise incremental Revenue and behavioral uplift generated.

Lifecycle marketing campaigns, from onboarding to retention, are crucial for long-term customer value. However, attributing success accurately can be challenging. Many marketing efforts show correlation with positive outcomes, but correlation does not equal causation. This is where incrementality testing becomes indispensable. By systematically measuring the true uplift attributable to a specific intervention, you can confidently optimize your strategies and maximize your average revenue per user (ARPU).

How to run incrementality testing on lifecycle marketing

Running incrementality tests in lifecycle marketing requires a structured approach to ensure valid and actionable results. It begins with identifying clear Revenue opportunities and formulating testable Hypotheses.

1. Identify Signals and Revenue opportunities

The first step is to pinpoint areas where a marketing intervention could significantly impact customer behavior and Revenue. This often starts with observing user behavior patterns, or 'Signals', across your customer lifecycle. For example:

• Onboarding drop-offs: A Signal indicating users are abandoning the product early. The Revenue opportunity is to increase activation rates.
• Decreasing feature usage: A Signal that users are losing engagement. The Revenue opportunity is to re-engage dormant users.
• Low purchase frequency: A Signal for under-monetization. The Revenue opportunity is to encourage repeat purchases.
• High churn risk: A Signal based on predictive models. The Revenue opportunity is to reduce churn.

Markin specializes in identifying these Revenue opportunities by analyzing vast datasets and surfacing actionable insights for B2C businesses.

2. Formulate a Hypothesis

Once a Revenue opportunity is identified, formulate a clear, testable Hypothesis. A strong Hypothesis proposes a specific Candidate action and predicts its impact on a measurable metric. It should follow an 'If-Then-Because' structure.

• Example Hypothesis for onboarding: If we send a personalized 'getting started' email series (Candidate action) to new users who haven't completed profile setup, then their 7-day activation rate will increase by 10% (expected outcome), because tailored guidance removes initial friction.
• Example Hypothesis for retention: If we offer a limited-time discount on a premium feature (Candidate action) to users showing signs of churn, then their 30-day retention rate will improve by 5% (expected outcome), because the perceived value outweighs churn drivers.

3. Define your Candidate action and target audience

The Candidate action is the specific marketing intervention you will test. This could be an email, an in-app notification, an SMS, a discount, a new feature message, or a personalized recommendation. Clearly define the content, timing, and channel of this action.

Next, define the target audience for this action. This should be the segment of users exhibiting the Signal associated with your Revenue opportunity. For instance, new users who haven't completed a specific step, or dormant users who haven't engaged in X days.

4. Implement a control group (holdout group or ghost cell)

This is the core of incrementality testing. You must create a control group – a randomly selected portion of your target audience that will not receive the Candidate action. This group will serve as your baseline for comparison.

How big should a holdout group be?

The ideal size of a holdout group is a common question, and it's a balance between statistical power and potential foregone Revenue. A larger holdout group offers higher statistical power to detect smaller incremental lifts, but it also means a larger portion of your audience isn't receiving a potentially beneficial intervention.

Key factors influencing holdout group size:

• Baseline conversion rate: If your baseline rate is very low, you'll need a larger sample to detect a meaningful lift.
• Expected lift: If you anticipate a large incremental impact, you can get away with a smaller holdout group. Smaller expected lifts require larger samples.
• Desired statistical significance (alpha): Typically set at 0.05, meaning a 5% chance of a false positive.
• Desired statistical power (beta): Typically 0.8 (or 80%), meaning an 80% chance of detecting a true effect if one exists.
• Audience size: The total number of users available for the Experiment.

General recommendation: For many lifecycle marketing campaigns, a holdout group representing 5% to 10% of the eligible target audience is a common and practical starting point. This often provides enough statistical power for typical campaign lifts while minimizing the impact on overall campaign performance. For high-volume, low-impact campaigns, a smaller percentage might suffice. For critical, high-impact campaigns where even small lifts are valuable, you might lean towards 10% or even 15%.

It's crucial that the assignment to the control or treatment group is truly random to avoid selection bias. Any systematic difference between the groups, other than the Candidate action, will invalidate your results.

5. Run the Experiment (Layer)

Deploy your Candidate action to the treatment group while withholding it from the control group. Ensure the Experiment runs for a sufficient duration to capture the full impact of the intervention. This duration can vary significantly: a quick activation email might show results in days, while a retention campaign targeting LTV might need weeks or months.

Markin refers to this execution phase as deploying a Layer – a specific Candidate action applied to a defined segment for a set duration, with inherent control group management for incrementality measurement.

6. Measure and analyze results

After the Experiment duration, compare the key metrics between the treatment group and the control group. Focus on the metric defined in your Hypothesis.

Here's a comparison table of key metrics for different lifecycle stages:

Lifecycle Stage	Key Metrics for Incrementality	Example Hypothesis & Lift
Acquisition/Onboarding	Activation Rate, First Purchase Rate, Time to First Action, Feature Adoption	15% incremental lift in 7-day activation rate.
Engagement/Retention	Repeat Purchase Rate, Churn Rate (reduction), Session Frequency, Time Spent, LTV, NPS	5% incremental reduction in 30-day churn.
Monetization/ARPU Expansion	Average Order Value (AOV), Conversion Rate to Premium, Cross-sell/Upsell Rate, ARPU	10% incremental increase in Q3 ARPU.
Win-back/Reactivation	Reactivation Rate, Return Purchase Rate, Time to Re-engage	8% incremental reactivation rate for dormant users.

Calculate the incremental lift:

Incremental Lift = (Treatment Group Performance - Control Group Performance) / Control Group Performance

Crucially, assess the statistical significance of the observed difference. A/B testing tools or statistical software can help determine if the lift is truly due to your intervention or merely random chance.

For example, if your treatment group had a 10% conversion rate and your control group had an 8% conversion rate, the incremental lift is (10% - 8%) / 8% = 25%. This 25% represents the true added value of your campaign.

7. Iterate and scale

Based on your findings, decide whether to scale the Candidate action, refine it, or discard it. If the Experiment showed a statistically significant positive incremental lift, integrate the action into your standard lifecycle workflow. If the results were inconclusive or negative, revisit your Hypothesis and design a new Experiment.

This iterative process of identifying Revenue opportunities, forming Hypotheses, running Experiments (Layers), and analyzing results is central to continuous ARPU expansion. Markin's next-best-action capabilities can automate this cycle, allowing you to continually test and deploy optimal Candidate actions across millions of users.

Reading incrementality testing results

When interpreting incrementality test results, consider these scenarios:

• Positive Incremental Lift: The treatment group performed significantly better than the control group. This indicates your intervention is effective and should be scaled.
• No Significant Lift (Flatline): The treatment group performed similarly to the control group. Your intervention had no measurable incremental impact. This is valuable learning; it means you can save resources by not running this campaign, or it indicates a need to revise your approach.
• Negative Incremental Lift: The treatment group performed worse than the control group. This is rare but indicates your intervention might be actively harming performance. Stop the campaign immediately and investigate why.

Always look beyond the primary metric. Did the campaign impact other relevant metrics (e.g., did a retention offer increase churn for non-offered segments)? A holistic view provides deeper insights into customer behavior and the true impact on ARPU.

Common mistakes in incrementality testing

Even with a clear process, pitfalls can compromise your incrementality tests. Avoiding these ensures more reliable and actionable insights:

1. Non-randomized Assignment: If groups are not truly random, selection bias can occur. For instance, if a holdout group is inadvertently created from users with a specific trait, results will be skewed. Always use robust randomization methods.
2. Insufficient Sample Size or Duration: Running an Experiment for too short a time or with too few participants can lead to inconclusive results or false positives/negatives due to lack of statistical power.
3. Contamination Between Groups: Users in the control group inadvertently receiving the treatment, or interacting with users who did, can dilute the incremental effect. Ensure strict segmentation and isolation. This is particularly challenging in viral or social product features.
4. Ignoring External Factors: Broader market trends, competitor actions, or seasonal changes can influence results. While randomization helps, be aware of significant external events during your Experiment.
5. Focusing on Correlation, Not Causation: The primary goal is to isolate causation. Simply seeing a group perform better isn't enough; you must prove the difference is due to your intervention, not other factors.
6. Testing Too Many Variables at Once: Avoid testing multiple Candidate actions in a single Experiment (A/B/C/D tests often dilute impact). Focus on one clear Hypothesis and its corresponding Candidate action per Experiment.
7. Lack of Clear Hypothesis: Without a specific, testable Hypothesis, it's hard to design a meaningful Experiment or interpret its results.
8. Not Accounting for Long-Term Impact: Some interventions might have short-term gains but long-term negative effects, or vice versa. Consider the appropriate time horizon for measuring your key metrics, especially for retention decisioning.
9. Improper Tooling: Attempting to run complex incrementality tests manually across large user bases is prone to error and highly inefficient. Specialized platforms like Markin are designed to manage segmentation, randomization, deployment of Layers, and measurement at scale, ensuring robust incrementality.

By diligently avoiding these common mistakes, your incrementality testing efforts will yield more reliable insights, enabling smarter resource allocation and more effective lifecycle marketing strategies that genuinely drive ARPU growth.