P-Value Interpretation
P-value interpretation is the discipline of reading a p-value as evidence about the observed data under a null hypothesis, not as a direct probability that the hypothesis is true or false.
Key points
- Statsig defines a p-value as the probability of observing results as extreme as the measured data, assuming the null hypothesis is true [src-035].
- A p-value at or below the chosen alpha threshold supports rejecting the null hypothesis and treating the result as statistically significant [src-035].
- The article warns against a common mistake: a p-value does not state the probability that the null hypothesis itself is true or false [src-035].
- Another warning is practical: a statistically significant result can still be too small to matter in product or business terms [src-035].
- Failing to reject the null hypothesis does not prove the null; it only means the observed data did not provide enough evidence against it [src-035].
- P-values should be interpreted alongside study design, sample size, possible confounders, effect size, and the consequences of acting on the result [src-035].
Related entities
Related concepts
- Statistical Significance Testing
- Experiment Statistical Power
- Multiple Testing Correction
- A/B Testing Mindset
Source references
- [src-035] Jack Virag — “How to accurately test statistical significance” (2025-04-12)