Randomization Test

IMS, Ch. 11

Benjamin S. Baumer

Smith College

Apr 6, 2026

Review

Inference for a single proportion

• Hypothesis testing
  • parametric bootstrap (Ch. 16.1)
  • probability theory
  • normal approximation (Ch. 16.2)
• Confidence intervals
  • bootstrap (Ch. 12)
  • normal approximation (Ch. 16.2)

Inference for two proportions

• Hypothesis testing
  • randomization test (Ch. 11, 17.1)
  • normal approximation (Ch. 17.3)
• Confidence intervals
  • bootstrap (Ch. 17.2)
  • normal approximation (Ch. 17.3)

Day one

Is yawning contagious?

• Recall our first day together

• Recall the yawning data

Intuitive randomization test

library(tidyverse)
library(openintro)
library(infer)
null_dist <- yawn |>
  specify(result ~ group, success = "yawn") |>
  hypothesize(null = "independence") |> 
  generate(reps = 1000) |>
  summarize(
    trmt_yawners = sum(result == "yawn" & group == "trmt"),
    .groups = "drop"
  )

Situate observed statistic in null dist

null_dist |>
  ggplot(aes(x = trmt_yawners)) +
  geom_histogram() + 
  geom_vline(xintercept = 10, color = "red", type = 3)

Difference in proportions

library(janitor)
yawn |>
  tabyl(result, group) |>
  adorn_totals(c("row", "col"))

   result ctrl trmt Total
 not yawn   12   24    36
     yawn    4   10    14
    Total   16   34    50

two_way <- yawn |>
  group_by(group) |>
  summarize(n = n(), prop = mean(result == "yawn"))
two_way

# A tibble: 2 × 3
  group     n  prop
  <fct> <int> <dbl>
1 ctrl     16 0.25 
2 trmt     34 0.294

dp_hat <- two_way |>
  pull(prop) |>
  diff()
dp_hat

[1] 0.04411765

Setup

• \(H_0\): \(p_1 - p_2 = 0\) (independence)
• \(H_A\): \(p_1 - p_2 \neq 0\)
• \(\alpha = 0.05\)

null_dist <- yawn |>
  specify(result ~ group, success = "yawn") |>
  hypothesize(null = "independence") |> 
  generate(reps = 1000, type = "permute") |>
  calculate("diff in props", order = c("trmt", "ctrl"))

Null distribution

null_dist |>
  ggplot(aes(x = stat)) +
  geom_density(fill = "darkgray") + 
  geom_vline(xintercept = 0, linetype = 3) + 
  geom_vline(xintercept = dp_hat, color = "red", linetype = 2)

Decision

dp_hat

[1] 0.04411765

null_dist |>
  get_p_value(obs_stat = dp_hat, direction = "two-sided")

# A tibble: 1 × 1
  p_value
    <dbl>
1   0.994

Your turn

See Randomization Test handout