---
title: Markin + Optimizely: from winning experiments to deciding what is worth testing
url: https://markin.ai/compare/markin-and-optimizely
kind: stack
description: Optimizely proves which variation wins. Markin decides which revenue opportunity is worth testing and acts on every customer continuously, against a holdout.
updated: 2026-09-03
---

# Markin + Optimizely: from winning experiments to deciding what is worth testing

> Optimizely is an experimentation and feature-flagging platform: it runs A/B tests and bandits on variations you define and reports which wins on a metric you chose. Markin sits above it and decides which commercial opportunity deserves a test in the first place, sizes the revenue behind it, and acts continuously on customers who never enter an experiment. The learning happens in Optimizely; the decision of what to learn and who to act on happens in Markin.

## In short

- Optimizely: Which variation performs better on this metric, and how confidently? Output: A statistical read per variation: lift, confidence interval, significance.
- Markin, the decision + execution layer: Which opportunity deserves an experiment, and which action should this customer receive now? Output: A ranked, sized decision per customer,  including hold, that can feed an Optimizely flag or act directly in other channels.
- An experiment only covers customers who hit the flagged surface; the rest of the base receives no decision at all.
- It decides which revenue opportunity is worth a test, and acts on the customers a flag never reaches.

## What each layer is

**Optimizely**, An experimentation and feature-flagging platform that deploys code behind flags, runs A/B/n tests and multi-armed bandits, and reports which variation performs on a defined success metric with valid statistics.

Answers: Which variation performs better on this metric, and how confidently?

**Markin, the decision + execution layer**, A layer that discovers and sizes revenue opportunities per customer, selects the treatment with the highest expected incremental value, and assigns a control group, deciding what is worth testing and who to act on.

Answers: Which opportunity deserves an experiment, and which action should this customer receive now?

## Side by side

|  | Optimizely | Markin, the decision + execution layer |
| --- | --- | --- |
| Question it answers | Which variation performs better on this metric, and how confidently? | Which opportunity deserves an experiment, and which action should this customer receive now? |
| Primary input | Variations, a success metric, traffic allocation, audience targeting. | Customer context, outcomes, margins, costs, contact history, past experiment results. |
| Primary output | A statistical read per variation: lift, confidence interval, significance. | A ranked, sized decision per customer,  including hold, that can feed an Optimizely flag or act directly in other channels. |
| Usual owner | Product, engineering and experimentation. | Growth, data science and revenue leadership. |
| How it's measured | Lift on the chosen metric with significance and confidence interval. | Incremental revenue and ARPU against a holdout. |

## What stays unsolved when experimentation is running well

A mature Optimizely setup tests whatever the team writes into it, at the cadence the team can ship. Experimentation throughput is bounded by the hypotheses a human decides to formalise,  and the customers a flag is wired into, rather than by what the data is signalling across the base.

- An experiment only covers customers who hit the flagged surface; the rest of the base receives no decision at all.
- Stats Accelerator finds the best variation inside a test; it does not decide whether the test itself is the highest-value use of that contact.
- Each experiment reports on one metric. Nothing sums the effect of every experiment a customer saw into one revenue number.
- The backlog of hypotheses arrives at the speed the team can write specs, not at the speed the data changes.

## How the two run together

1. **Context in.** Markin reads customer context where it already lives,  warehouse, CDP, product and billing systems, plus outcome history. Optimizely experiment results and flag exposure can be part of that context, so learning feeds the next decision.
2. **Decision.** Markin generates and sizes revenue opportunities, ranks them per customer, chooses a treatment and assigns a control group. Some decisions become Optimizely experiments; most become direct actions in other channels.
3. **Activation back into Optimizely.** Where a hypothesis needs a controlled test, the decision is written back as flag targeting or audience criteria so Optimizely runs it with its own stats engine. Flag logic, variation code and significance reporting remain in Optimizely.

## Optimizely's own decisioning layer

Products: Optimizely Feature Experimentation, Optimizely Web Experimentation, Optimizely Personalization, Stats Engine, Stats Accelerator

Optimizely is an experimentation and feature-flagging platform. It deploys code behind flags, runs A/B tests and bandits on variations you define, and proves which variation wins on a success metric you chose. It does not decide which commercial opportunity is worth testing, size the revenue behind it, or act on customers who never enter an experiment.

**What it optimises**

- Optimizely Feature Experimentation is documented as a feature-flagging and experimentation platform for deploying code behind flags, running A/B tests and targeted rollouts across web, mobile and connected devices. [Vendor docs: Optimizely docs, Introduction to Feature Experimentation](https://docs.developers.optimizely.com/feature-experimentation/docs/welcome)
- Optimizely offers Frequentist (fixed-horizon), Bayesian and Sequential (Stats Engine) statistical methods, with sequential testing that keeps results valid whenever they are viewed and ends experiments early on average with fewer observations. [Vendor docs: Optimizely Support, Statistical analysis methods overview](https://support.optimizely.com/hc/en-us/articles/39714777161229-Statistical-analysis-methods-overview)
- Stats Accelerator is documented as a multi-armed bandit algorithm that manipulates traffic allocation to shorten time to statistical significance and optimise rewards during a test. [Vendor docs: Optimizely Support, Stats accelerator overview](https://support.optimizely.com/hc/en-us/articles/4410283570701-Stats-accelerator-overview)

**Documented constraints**

- The documented unit of optimisation is the variation on a defined success metric, with traffic distributed across variations. Choosing what to test, and which metric defines success, remains a human decision outside the platform. [Vendor docs: Optimizely Support, Experimentation distribution methods](https://support.optimizely.com/hc/en-us/articles/4410289050893-Experimentation-distribution-methods)
- Stats Accelerator and multi-armed bandits allocate traffic within a single experiment's variations; they do not arbitrate which of many possible commercial opportunities across the base deserves attention. [Vendor docs: Optimizely Support, Experimentation distribution methods](https://support.optimizely.com/hc/en-us/articles/4410289050893-Experimentation-distribution-methods)

**Evidence**

- Optimizely's documented performance claims concern statistical validity and speed-to-significance (Stats Engine, Stats Accelerator), not verified incremental revenue. No independent benchmark of Optimizely's impact on ARPU or revenue is published. [Vendor docs: Optimizely Support, Statistical analysis methods overview](https://support.optimizely.com/hc/en-us/articles/39714777161229-Statistical-analysis-methods-overview)

**Where Markin differs**

- **Decides what to test, not which variation wins.** Optimizely answers 'which variation performs better on this metric'. Markin answers 'which opportunity is worth running an experiment on at all', and surfaces the ones no one thought to test.
- **Optimises revenue across the base, not a metric on one test.** Stats Accelerator reallocates traffic inside an experiment. Markin reallocates contact and treatment across the whole customer estate, on expected incremental revenue net of margin and cost.
- **Holds across the estate, by default.** An Optimizely holdout is a traffic split within one experiment. Markin attaches a control group to every decision, so the reported number is incremental ARPU against holdout rather than a significance read on a flag.

## What Markin does not replace

To be explicit about scope, because procurement will ask:

- Markin does not run A/B tests, feature flags or bandits. Experimentation stays in Optimizely.
- Markin does not replace Stats Engine, Stats Accelerator or Optimizely's statistical reporting.
- Markin is not an analytics or release-management tool; it decides what is worth testing and who to act on.
- Markin does not manage variation code, rollouts or flag environments.

## Where Markin fits

Markin does not run experiments or replace flags. It decides which revenue opportunity is worth a test, and acts on the customers a flag never reaches. Experimentation keeps its stats engine, its bandits and its release safety; what changes is the input that decides what gets tested and who gets acted on.

- **The experiment queue becomes a decision, not a backlog.** Instead of a product roadmap of tests, the highest-value hypotheses are surfaced and sized from the data, so experimentation capacity is spent where the revenue is.
- **Coverage extends beyond the flagged surface.** Customers who never enter an experiment still receive a decision,  a save, an attach, a hold, measured against control in the channels they do use.
- **Learning and acting share one record.** Optimizely results feed Markin, and Markin's holdout feeds back, so the next round of hypotheses improves on proven numbers rather than drifting.

Next: [Customer Decisioning](https://markin.ai/solutions/customer-decisioning)

## When Markin is not the right answer

- You run a handful of experiments a quarter and the team can still reason about which to prioritise in a meeting.
- Every customer decision already flows through a flagged surface, so there is no unaddressed base to act on.
- Your success metric cannot be tied to revenue, so neither experiments nor decisions can be valued.

## FAQ

**Doesn't Optimizely already decide things with bandits?**

Stats Accelerator and multi-armed bandits reallocate traffic between variations inside one experiment to reach significance faster or maximise reward during the test. They optimise the test you already wrote; they do not decide which opportunity is worth testing, size the revenue behind it, or act on customers who never enter the experiment.

**Do we have to replace Optimizely?**

No. Optimizely remains the experimentation and feature-flagging layer. Markin decides which hypothesis deserves a test and passes that decision into Optimizely as flag targeting, while acting on the rest of the base directly in other channels.

**How does the decision reach Optimizely?**

As flag targeting rules or audience criteria on a flag, so an existing experiment picks it up. The integration surface is the same one your team already uses for any other upstream signal.

**Isn't running experiments all the time the same as decisioning?**

No. Experimentation tests a few hypotheses at human cadence on the surfaces you flagged. Continuous decisioning acts on every customer, every cycle, with a control group by default, including customers no experiment reaches. Experiments prove what works; decisioning decides who gets it and who gets held.

**Who owns the output, product or growth?**

Both. Product and engineering own the experiments and the flags; growth and data science own the revenue decision. The decision layer is the shared contract between them.

**What does the first ninety days look like?**

One revenue theme, one channel, a real holdout. The point of the first quarter is a defensible incremental number, not full coverage of the experiment backlog.

**How is Markin different from the decisioning or AI already inside Optimizely?**

A decisioning engine ranks actions a human already defined, inside the campaign surface it was given. Markin forms the hypotheses itself,  marketing, product, pricing or a technical anomaly holding growth back, sizes them, executes them inside Optimizely and your product surfaces, and reads each one against a randomised holdout. It behaves like a data science and growth team, not like an optimiser.

**Does Markin only test messages and offers?**

No. Anything a human growth scientist would investigate is in scope: onboarding friction, feature adoption, pricing and packaging, dunning, and technical health issues such as a checkout error rate or a broken deeplink quietly killing conversion. Marketing is one of four hypothesis domains, not the boundary.

**What is the business case for adding Markin on top of Optimizely?**

On the assumptions preloaded above, 3.0M customers at 22 a month, a small move in ARPU is a large number in absolute terms, because it applies to the whole installed base every month rather than to a campaign. Across Markin deployments the verified range on treated cohorts is +17% to +35% ARPU against a randomised holdout. The point is not more messages: it is finding the highest-value action per customer, launching it, and proving it against control before it scales.

**How long before it pays for itself?**

First sized opportunities are in test within six weeks and the first holdout-verified result lands inside 90 days. Payback depends on your base, margin and programme cost, the calculator on this page computes it from your own numbers, after applying the 20% to 40% haircut BCG finds when next-best-action programmes are incrementality-tested.

Source: https://markin.ai/compare/markin-and-optimizely