Sequential Testing

Sequential testing is an experiment-analysis approach that allows teams to inspect results over time and stop early when evidence is strong while controlling error rates.

Key points

• Statsig frames ordinary peeking as risky because repeated looks can inflate false discoveries, but also acknowledges that teams realistically want to monitor experiments before full maturity ^[src-031].
• The article names mSPRT as a method that allows early stopping when evidence is overwhelming while keeping family-wise error rate under control ^[src-031].
• Sequential testing solves a different problem from final-readout multiple-metric correction. Statsig recommends Benjamini-Hochberg for controlling false discovery rate across many reported metrics, but says it does not handle repeated looks ^[src-031].
• The practical recommendation is to combine sequential testing for repeated monitoring with multiple-comparison correction for broad metric dashboards ^[src-031].
• The article also presents Bayesian readouts as a narrative framing that changes interpretation, not the underlying data, and should not be treated as a magic shortcut ^[src-031].
• Statsig’s significance guide reinforces the complementary risk: when the problem is many simultaneous hypotheses rather than repeated looks, teams need Multiple Testing Correction methods such as Bonferroni or Benjamini-Hochberg ^[src-035].

Related entities

• Statsig

Related concepts

• A/B Test Acceleration
• Experiment Statistical Power
• Experiment Variance Reduction
• A/B Testing Mindset
• Statistical Significance Testing
• Multiple Testing Correction
• P-Value Interpretation

Source references

• ^[src-031] Yuzheng Sun — “Speeding up A/B tests with discipline” (2025-06-24)
• ^[src-035] Jack Virag — “How to accurately test statistical significance” (2025-04-12)