Ivan's wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
# A/B testing
**A/B testing** is a method of comparing two variants of a feature by exposing each to a different subset of users simultaneously and measuring the effect on a target metric. Variant A is typically the control (current behaviour); variant B is the treatment (the change being evaluated).

```
users → routing layer → [variant A: button is blue]   (50%)
                         [variant B: button is green]  (50%)

after N days:
conversion rate A: 3.2%
conversion rate B: 3.8%  ← statistically significant improvement
→ ship variant B to 100%
```

The purpose is to replace subjective decisions with measured outcomes. "Which colour converts better?" and "does the new onboarding flow retain more users?" are questions that intuition answers poorly. A/B tests answer them with data.

## Statistical validity

An A/B test is only meaningful if the result is statistically significant. Two requirements: the sample size must be large enough that the observed difference is not plausibly due to chance, and the test must run long enough to cover any weekly seasonality in user behaviour (typically at least one or two full weeks).

Running too many tests simultaneously is a validity risk: if you test 20 things at once and one shows a positive result at 95% confidence, that one positive result is likely a false positive. Multiple comparisons require either a stricter significance threshold or a family-wise error rate correction.

## In software systems

A/B testing is closely related to [[feature-flag|feature flags]]: the routing logic that sends users to variant A or B is usually implemented on top of a feature flag system that supports percentage rollouts and user cohort assignment. The difference is that a [[canary-release]] aims to verify correctness; an A/B test aims to measure user behaviour.