What Will Happen When Adding a New Variable to a Multiple Linear Regression Model

What Will Happen When Adding a New Variable to a Multiple Linear Regression Model

• Conclusions:
  • This new (different from existing ones) variable will ALWAYS reduce Residual Sum of Squares ($RSS$)
  • However, a large $p$ for the partial t-test for this new variable leads us to conclude that this variable is not a useful or valid one and thus should not be included in the model.
• Explanations:
  • Formula: $RSS={\sum }_i(y_i-\widehat{y_i}{)}^2$, $y_i$ as observed values of the response variable at $x_i$, and $\widehat{y_i}$ is the estimated mean value of the unobservable random variable $Y$ at $x_i$ estimated/calculated by the regression model fitted. Notice that we assume no error in the observed values of the explanatory variable(s) as the predictor(s) i.e. we treat the predictors’ values as fixed all the time.
  • From the formula, we can see that $RSS$ measures how many variability in the observed values of the response variable (i.e. the dataset used) is NOT explained by the regression model fitted.
  • Adding a new variable and calculating the model parameters by minimizing $RSS$ will always make the regression model explain a larger proportion of the variations in the observed $y$ values (see https://stats.stackexchange.com/questions/179244/is-rss-decreasing-or-non-increasing).
  • This means,regardless of whether a newly added variable makes sense in the model, $RSS$ will always decrease, which cause $R^2$ to increase. This leads to the caveat that using $R^2$ to test whether to add a new variable in the model is not appropriate .
  • The proper way to test whether the new variable is really statistically significant is through the $p$ value produced by the partial t-test on this new variable (or any equivalences like the partial $F$ test on the new variable). The null hypothesis is always the original model and the alternative hypothesis the original model plus the new variable (the new model). If $p$ is large, then we fail to reject the null hypothesis and conclude that the model with the new variable is not statistically significant against the original model, so we still use the “old model”.