Hypothesis Testing

IMS, Ch. 11

Benjamin S. Baumer

Smith College

Mar 13, 2026

Hypothesis Testing

\(H_0\): Null hypothesis

  • the state of the world that you assume is true
  • Typically, this is a world of “no effect”
  • \(H_0\) is a specific mathematical statement about the value of a population parameter

Under the null hypothesis

Sampling distributions are easy to calculate or simulate

\(H_A\): Alternative hypothesis

  • a state of the world different from the \(H_0\)
  • Typically, this is a world in which the effect you are testing for is real

Test statistic

  • a sample statistic computed from data
    • to be understood in the context of the null distribution
    • often, the sample mean

Null Distribution

The sampling distribution of the test statistic under the null hypothesis

Thresholds

  • \(p\)-value: probability of observing something as strange or stranger as what you observed, under the assumption that \(H_0\) is true

  • \(\alpha\)-level: a threshold beyond which you begin to doubt \(H_0\)

    • a line in the sand that you draw to purposefully gauge your dubiousness

Two possible outcomes

  • Reject \(H_0\) at \(\alpha\)-significance level
    • observations are so unlikely under \(H_0\), that \(H_0\) is likely false
  • Fail to reject \(H_0\) at \(\alpha\)-significance level
    • observations are reasonably likely under \(H_0\), so can’t rule out possibility that it is true
  • Important: we never confirm the null hypothesis
    • we only fail to reject!
  • Always report \(p\)-values and a confidence interval

Testing Outcomes

For a defendant on trial:

  • \(H_0\): innocent until proven guilty
  • Type I error: \(\Pr(\text{reject} | H_0)\)
    • convicting an innocent person
  • Type II error: \(\Pr(\text{fail to reject} | H_0^c) = 1- Power\)
    • letting a guilty person go free
  • Which is worse?

Table of error types

Controversy about NHST

https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf

Comparison of NHST to CI

NHST CI
null distribution sampling distribution
test statistic point estimate
\(\alpha\) \(\alpha\)
p-value interval
is the test statistic compatible with \(H_0\)? does the CI capture the true mean?