P-Value Interpretation

P-value interpretation is the discipline of reading a p-value as evidence about the observed data under a null hypothesis, not as a direct probability that the hypothesis is true or false.

Key points

• Statsig defines a p-value as the probability of observing results as extreme as the measured data, assuming the null hypothesis is true ^[src-035].
• A p-value at or below the chosen alpha threshold supports rejecting the null hypothesis and treating the result as statistically significant ^[src-035].
• The article warns against a common mistake: a p-value does not state the probability that the null hypothesis itself is true or false ^[src-035].
• Another warning is practical: a statistically significant result can still be too small to matter in product or business terms ^[src-035].
• Failing to reject the null hypothesis does not prove the null; it only means the observed data did not provide enough evidence against it ^[src-035].
• P-values should be interpreted alongside study design, sample size, possible confounders, effect size, and the consequences of acting on the result ^[src-035].

Related entities

• Statsig

Related concepts

• Statistical Significance Testing
• Experiment Statistical Power
• Multiple Testing Correction
• A/B Testing Mindset

Source references

• ^[src-035] Jack Virag — “How to accurately test statistical significance” (2025-04-12)