How Does Adjusted R Squared Avoid Model Overfitting (in Linear Regression)

$R_{adj}^2$ Formula

• $R_{adj}^2$ is a statistical measure that represents the proportion of the variance for a dependent variable ($Y$) that’s explained by an independent variable ($X$) in a regression model given by $$E(Y|X=x)={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2+...+{\beta }_p{x}^p$$ Moreover, $$y-E(Y|X=x)=ϵ$$ for each $x$ value where $ϵ$ is the error caused by randomness assumed to be constant variance ${\sigma }^2$ across all $x$ values.
• The formula is given by :$$R_{adj}^2=1-\frac{RSS/(n-p-1)}{TSS/n-1}$$
• The denominator is the sample variance of the dependent variable $$TSS/n-1=s^2=\frac{1}{n-1}\sum (y_i-\bar{y}{)}^2$$ and therefore measures the variance in the dependent variable.
  • This is a constant for a given dataset { ( x 1 , y 1 ) , . . , ( x n , y n ) } \{(x_1,y_1),..,(x_n,y_n)\} {(x1​,y1​),..,(xn​,yn​)}.
• The numerator is the residual sum of squares divided by its degree of freedom, which is the estimated variance for the random error $ϵ$. Specifically, $$RSS/(n-p-1)=MSE={\hat{\sigma }}^2$$ where $$RSS=\sum (\hat{f}(x_i)-y_i{)}^2$$ is the sum of residual squares. Since the sum of squared residuals measures the deviation of the fitted $y$ values from the observed $y$ values, the denominator measures the part in the variance of the dependent variable which gets explained by the regression model we proposed.
• Thus, the ratio of the two gives the proportion of the variance in the data that can be explained by the model, which makes sense of what $R_{adj}^2$ intends to measure.
• A side note regarding the degrees of freedom: $n-p-1$ is the degree of freedom for $RSS$. Here, $n$ is the sample size, $p$ is the number of coefficients in the model ($\widehat{{\beta }_i},i=1,..,p$), subtracted because each of these $p$ coefficients is an estimate of the corresponding parameter (i.e. the “truth”), and the $1$ is subtracted due to the fact that $\widehat{{\beta }_0}$, the estimate of intercept, is given by $$\widehat{{\beta }_0}=\bar{y}-\widehat{{\beta }_1}\bar{x}-\widehat{{\beta }_2}\overline{x^2}-...-\widehat{{\beta }_p}\overline{x^p}$$ where $\bar{y}$ is another estimate, reducing $RSS$ degree of freedom by another $1$. (Everytime you estimate, you lose one $df$).

Comparing Different Linear Models

• Suppose we fit $10$ linear models, $m_1,...,m_{10}$ to the same data where
  • $m_1$ is a first order polynomial model $$E(Y|X=x)={\beta }_0+{\beta }_{1}x$$
  • $m_2$ is a second order polynomial model $$E(Y|X=x)={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2$$
  • …
  • $m_{10}$ is a $10$th order polynomial model $$E(Y|X=x)={\beta }_0+{\beta }_{1}x+...+{\beta }_{10}{x}^{10}$$
• When we compare these model’s fit to the data, keep in mind that a regression task’s goal is to capture the “truth” as much and as accurately as possible. This means inaddition to examining how capable the model is in reducing $RSS$(i.e. how well the model explains the total variation in the data), we also want to avoid overfitting so that our model honestly reflects the real trend in the data. Comparing $R_{adj}^2$ allows us to compare both criterion.
• To see why, we first notice that the denominator of $R_{adj}^2$ is a constant, so the magnitude of $R_{adj}^2$ only depends on the numerator.
• Fact: $RSS$ will always decrease as the degree of your polynomial model gets higher or as you add more predictors because added terms or predictors always explain more variation in the dataset. Due to this, simply using $1-\frac{RSS}{SST}=R^2$ is obviously not enough to fulfil the goal of model comparison because more complex polynomial models will always win, but obviously too complex models cause the problem of overfitting.
• This is exactly why we divide $RSS$ by its degree of freedom in $R_{adj}^2$ so that we take into accoun the model complexity (by the$p$ in $df$). This normalization of $RSS$ will nullify the model complexity advantage on $RSS$ when $RSS$ does not reduce by an enough amount when $p$ is increased. Therefore, if $RS{S}_{m_{10}}=497,d{f}_{m_{10}}=89$ while $RS{S}_{m_{10}}=500,d{f}_{m_1}=98$, ${R_{adj}^2}_{m_{10}}$ is obviously smaller than ${R_{adj}^2}_{m_1}$, leading us to prefer $m_1$ in the two because it captures the truth as much as we want yet with much less complexity.
• Ultimately, when we are comparing $m_1,...,m_{10}$, we want to plot the $R_{adj}^2$ against the polynomial degrees, find the part of the graph where the $R_{adj}^2$ does not seem to increase any longer with the degrees, and the corresponding model might be the one we want.