A significance level, p-value, and confidence level answer different questions. None is the probability that a scientific claim is true.

QuantityDefinitionWrong interpretation
Significance level (α)A prespecified bound on rejection probability for a valid test when the null model holds“The positive result has α probability of being false”
p-valueUnder the model, probability of a result at least as incompatible with the null as observed“The null has probability p”
Confidence levelLong-run coverage target of an interval procedure under its assumptions“This interval has a 95% probability of containing the fixed parameter”

Errors and power

A Type I error rejects a true null. A Type II error fails to reject under a specified alternative. Power is rejection probability under that alternative. It depends on effect, sample size, variability, allocation, design, missingness, analysis, and alpha—not alpha alone.

A corrected coin example

For a prespecified two-sided exact binomial test of H0: p = 0.5, 55 heads in 100 flips gives a p-value about 0.368, not rejection at 0.05. An exact 95% interval is approximately 0.447 to 0.650.

from scipy.stats import binomtest
r = binomtest(55, 100, p=0.5, alternative="two-sided")
ci = r.proportion_ci(confidence_level=0.95, method="exact")

The calculation does not establish independent, identical, correctly recorded, or representative flips.

When tests and intervals correspond

A two-sided level-α test and a two-sided 100(1−α)% confidence interval give equivalent decisions when the interval inverts the same test family. One-sided questions, discreteness, approximations, transformations, nuisance parameters, multiplicity, and different interval constructions can break a simple equivalence.

See statistical significance and confidence intervals and the companion confidence-level guide.

Choose alpha through design

  1. Define the decision and consequences of false-positive and false-negative actions.
  2. Define the smallest effect that changes the decision.
  3. Specify estimands, repeated looks, endpoints, subgroups, and multiplicity.
  4. Evaluate operating characteristics under plausible null and alternative scenarios.
  5. Choose sample size, alpha allocation, stopping, and analysis before outcomes are examined.

A conventional threshold is not universal evidence. Field-specific scientific and regulatory guidance governs the actual design.

Multiplicity changes the question

With 20 independent tests at α=0.05 and every null true, probability of at least one rejection is:

1 - (1 - 0.05)^20 ≈ 0.6415

The value assumes independence and a complete null. Bonferroni controls family-wise error without requiring independence but can be conservative. Benjamini–Hochberg targets false discovery rate under stated dependence conditions. Define the hypothesis family and desired error criterion first; feature searches can create the same selection problem discussed in feature selection.

Report estimates, uncertainty, and limitations

A small p-value does not measure effect size or practical importance. A large p-value does not prove no effect. Report the estimate in meaningful units, interval method and level, exact p-value when relevant, smallest effect of interest, multiplicity handling, assumptions, deviations, and design limitations. Distinguish statistical uncertainty from bias, confounding, measurement error, and lack of generalizability.

Preregistration can separate confirmatory from exploratory work, but it does not remove invalid models, missing data, deviations, or publication bias. Disclose all prespecified outcomes and outcome-dependent choices.

Reviewed and substantially updated September 4, 2026. Original publication date preserved.