A/B testing is comparing two variants of a feature by exposing each to different subsets of users simultaneously and measuring the effect on a metric. Variant A is the control (current behaviour); variant B is the treatment being evaluated.

Intuition answers “which button colour converts better?” poorly. A Feature flag system routes users to variant A or B, collects metrics, and after enough time compares conversion rates. The purpose is replacing subjective decisions with measured outcomes.

Results must be statistically significant: large enough sample size and long enough duration (at least one or two full weeks to cover seasonality). Running too many tests simultaneously creates false positives through multiple comparisons. A/B tests differ from Canary release which aims to verify correctness; A/B tests measure user behaviour and product impact.

The example shows two variants with measured conversion rates; the red button wins with a statistically significant 0.6% improvement.

Users:
50% → Variant A (blue button)  | Conversion: 3.2%
50% → Variant B (red button)   | Conversion: 3.8%

Statistical test (2 weeks of data):
p-value: 0.02 (< 0.05, significant)
Conclusion: Red button converts better, ship it