A fixed-horizon A/B test asks a static question: split traffic evenly, run until you hit a pre-computed sample size, then read the result. A multi-armed bandit asks a moving one: which arm looks best right now, and how much traffic should it get this hour? Both are legitimate ways to run an experiment, but they optimize for different things, and conflating them is why teams end up with bandit output they try — and fail — to read like a standard A/B result. This guide is about picking the right tool and knowing what each one costs you.

Two Machines, Two Different Bets

A fixed-horizon test is built to answer "which variant is better, and by how much, with a stated error rate?" It holds allocation constant (usually 50/50) so that every unit of traffic contributes equally to a single, clean effect estimate at the end. The cost of a wrong-looking variant during the test is fully absorbed — you don't get to react to it until the test is over.

A multi-armed bandit is built to answer "how do I maximize outcomes while I'm still learning?" It shifts traffic toward whichever arm currently looks best, using algorithms like epsilon-greedy, Thompson sampling, or UCB (upper confidence bound) to balance trying new information against cashing in on what it already believes. The cost of a wrong-looking variant is partially avoided in real time, because the bandit routes fewer users to it as evidence accumulates.

Neither is "more rigorous" in the abstract. A fixed-horizon test is more rigorous for producing a comparable, decision-grade effect size. A bandit is more efficient for minimizing the traffic spent on a bad variant while you're finding the good one.

The Exploration-Exploitation Tradeoff, Concretely

Every bandit algorithm is a running answer to one question: spend this next visitor on exploration (learning more about an arm you're unsure of) or exploitation (routing to the arm you currently believe is best)?

  • Epsilon-greedy exploits the current best arm most of the time and explores at random a small fixed fraction of visits (commonly single-digit percentages). Simple, but it keeps exploring at the same rate even after the answer is obvious.
  • Thompson sampling draws a plausible effect from each arm's posterior distribution and routes the visitor to whichever arm "wins" that draw. Exploration shrinks naturally as posteriors narrow, without a hand-tuned schedule.
  • UCB-based methods route to the arm with the highest plausible upper bound on its true rate, which explicitly favors arms with wide uncertainty — new arms get a boost until they've earned or lost it.

The practical effect across all three: adaptive allocation moves traffic away from underperforming arms before you have enough data to say precisely how much worse they were. That's the point of a bandit, but it's also why bandits are a poor source for the kind of effect-size number you'd want to compare against a StatFacts insight card.

Where Adaptive Allocation Quietly Costs You Statistical Power

The traffic a bandit saves you from a losing arm is traffic it also withholds from measuring that arm precisely. Three consequences follow directly:

  • Unequal, time-varying sample sizes. By the end of a bandit run, arms rarely have comparable sample sizes, and the ratio was different at every point in the run. Standard confidence-interval math assumes fixed allocation; applying it to bandit data understates uncertainty.
  • Estimates are conditioned on their own history. An arm that looked good early gets more traffic, which makes its later estimate look even more confident — a mild feedback loop that biases naive point estimates optimistically. Specialized adaptive-inference methods correct for this; a plain conversion-rate calculation does not.
  • Small or late-breaking effects can get starved out. If a genuinely better arm looks mediocre in its first few hundred visits — plausible for a modest true effect — a well-tuned bandit will already be diverting traffic away from it before enough data accumulates to reverse that impression.

None of this makes bandits wrong. It means the number a bandit reports as "arm B's rate" is a decision-support figure, not a substitute for the kind of effect estimate you'd log against a benchmark.

A Side-by-Side Comparison

Dimension Fixed-horizon A/B test Multi-armed bandit
Primary goal Precise, comparable effect estimate Maximize outcomes during the test itself
Traffic allocation Fixed (e.g., 50/50) for the full run Shifts continuously toward the current leader
Cost of a bad variant Fully paid, absorbed as "cost of learning" Partially avoided, shrinks as evidence grows
Statistical inference Standard CI / p-value math applies cleanly Requires adaptive-inference correction
Best fit High-stakes, one-shot decisions; benchmark comparisons Many-armed, short-lived tests (headlines, creative, ranking); ongoing optimization
Failure mode Wastes traffic on a clearly losing arm until the horizon Can starve a slow-to-reveal winner or a low-traffic arm

Reading StatFacts Benchmarks When Your Test Wasn't Fixed-Horizon

StatFacts insight cards report an effect range, a confidence label, and sample_context that describe results largely drawn from fixed-allocation designs. If your own result comes from a bandit, don't drop the raw "final arm rate" straight into that comparison — three adjustments keep the comparison honest:

  1. Recompute the effect from adaptive-inference-corrected estimates, not the bandit's live dashboard number, before comparing it to a benchmark's effect range. The live number is a decision aid; it's biased for measurement purposes in exactly the way described above.
  2. Treat sample_context as a check on comparability, not just size. A bandit run and a benchmark's underlying fixed-horizon test can reach the same total sample size while representing very different amounts of information about the losing arm, since the bandit spent fewer visits confirming it.
  3. Use the confidence label as a floor, not a match. If a benchmark's range comes from tightly controlled fixed-horizon tests, a bandit-derived estimate landing inside that range is a weaker form of agreement than an equivalent fixed-horizon result would be — the uncertainty behind the bandit number is usually larger than a naive interval suggests.

Setup Notes Before You Switch

  • Reserve bandits for situations with many candidate arms, short arm lifespans, and low per-decision stakes — headline variants, ranking tweaks, promotional creative. Reserve fixed-horizon tests for decisions you'll defend later or compare against a published range: pricing, checkout, core funnel changes.
  • If you need both — real-time optimization and a clean effect size — consider a short fixed-horizon phase to establish a defensible estimate, followed by a bandit phase once the decision is no longer in question and you're purely optimizing delivery.
  • Whichever design you run, decide up front which adaptive-inference method (if any) you'll use to de-bias the bandit's estimates, the same way you'd pre-register a stopping rule for a fixed-horizon test — choosing the correction after seeing which arm won defeats the purpose of using one.

Related Resources from StatFacts: