12-Metrics for Model Fit

SDS 291

Prof. Baumer

November 12, 2025

Overfitting

https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff

Need to balance two competing sources of prediction error/poor explanatory performance:

  • Bias: error introduced by approximating the “truth” with a model that is too simple
  • Variance: amount by which our predictions would change if we estimated the model on a different sample. Generally, the closer the model fits the observed data, the more variable it will be

Ex: first year GPA

  1. Making predictions: Can we predict a student’s first-year GPA as accurately as possible?
  2. Quantifying an association: What is the relationship between first-year GPA and the proportion of one’s high school that intends to go to college?
head(gpa)
  StudentID  GPA HSGPA SATV SATM Male   HU   SS FirstGen White CollegeBound
1    100001 3.06  3.83  680  770    1  3.0  9.0        1     1            1
2    100002 4.15  4.00  740  720    0  9.0  3.0        0     1            1
3    100003 3.41  3.70  640  570    0 16.0 13.0        0     0            1
4    100004 3.21  3.51  740  700    0 22.0  0.0        0     1            1
5    100005 3.48  3.83  610  610    0 30.5  1.5        0     1            1
6    100006 2.95  3.25  600  570    0 18.0  3.0        0     1            1

Ex: first year GPA

Developing a model to predict a student’s GPA after their first year of college

library(Stat2Data)
# First year GPA data
data("FirstYearGPA")

Note

Spend 2–3 minutes exploring the data: what combination of explanatory variables produces a model with the highest \(R^2\)?

Finding the “best” model

Need to balance multiple competing goals:

  1. Want model to be simple as possible to avoid multicollinearity and overfitting (and to improve the interpretability)
  2. Want model to do a good job of meeting its stated objective

An Important Model Building Observation

There is no one “best” model for any particular objective! However, some models do a better job of navigating this trade-off than others…

Metrics for model comparison

To compare two models for first year GPA that:

  • are fit to the same data set (with the same \(n\))
  • have the same response variable (with the same transformation)

Use model metrics:

  • (Nested models only!) \(F\) statistic (and \(F\)-tests)
  • AIC \(\iff\) Mallow’s C\(_\text{p}\)
  • BIC
  • Adjusted \(R^2\)

Use broom::glance()

lm(GPA ~ HSGPA + SATM + Male, data = gpa) |> 
  glance() 
# A tibble: 1 × 12
  r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
      <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>
1     0.218         0.207 0.414      20.0 1.79e-11     3  -116.  242.  259.
# ℹ 3 more variables: deviance <dbl>, df.residual <int>, nobs <int>

Nested \(F\)-tests

  • Compare the relative fits (\(SSE\)) of the two models with the difference in their complexities (\(p, k\))

\[ F = \frac{(SSE_\text{reduced} - SSE_\text{full}) / p}{SSE_\text{full} / (n - k - 1)} \sim F(p, n - k - 1) \]

  • We only prefer the full model if the improvement in fit is “worth” the additional complexity!

Tip

“Better” models are determined by a formal hypothesis test

Adjusted \(R^2\)

  • \(R^2\) will always increase as variables are added to model
  • Adjusted \(R^2\) “fixes” this by adding a slight penalty that scales with the number of terms in the model:

\[ \text{Adjusted } R^2 = 1 - \frac{SSE / (n - k - 1)}{SST/(n-1)} = 1 - \frac{MSE}{MST} \]

Note

Adjusted \(R^2\) is no longer interpretable as the proportion of variability explained!

Tip

“Better” models have larger adjusted \(R^2\) values

Akaike’s Information Criterion (AIC)

  • Navigates the trade-off between model complexity (\(k\)) and model fit (\(SSE\))

\[ AIC \propto n \log{\left( \frac{SSE}{n} \right)} + 2 (k + 1) \]

As more terms are added to the model: \(SSE \downarrow\) and penalty \(\uparrow\)

Tip

“Better” models have smaller AICs

Schwarz’s Bayesian Information Criterion (BIC)

  • Similar to the AIC, but places a stronger penalty on the number of terms in the model (\(k\)) whenever the sample size \(n > 7\)

\[ BIC \propto n \log{ \left( \frac{SSE}{n} \right) } + (k + 1) \log{n} \]

  • Favors models with fewer explanatory vars (relative to AIC)

Tip

“Better” models have smaller BICs