Basic Mathematics for Linear Regression and Least Square Method

最新推荐文章于 2022-01-31 22:44:55 发布

Eric2i

最新推荐文章于 2022-01-31 22:44:55 发布

阅读量303

点赞数

文章标签： statistics

本文链接：https://blog.csdn.net/Eric2i/article/details/113786614

版权

Basic Mathematics for Linear Regression and Least Square Method

Content

- Basic Mathematics for Linear Regression and Least Square Method
Background
Mathematical Approach
Least Squares
- Background & Founding
- Problem Statement
Computer Programming
- Matrix manipulation
- Package sklearn.linear_model.LinearRegression
Application

Background

In statistics, linear regression is a linear approach to modeling the relationship between a scalar response and one or more explanatory variables. The case of one explanatory variable is call simple linear regression; for more than one, the process is called multiple linear regression. In linear regression, the relationship are modeled using linear predictor functions whose unknown model parameters are estimated from the data. Such models are called linear models. Most, commonly, the conditional mean of the response given the values of the explanatory variables is assumed to be an affine function of those values. Like all forms of regression analysis, linear regression focuses on the conditional probability distribution of the response given the values of the predictors.
Linear regression was the first type of regression analysis to be studied rigorously, and to be used extensively in practical applications. This is because models which depend linearly on their unknown parameters are easier to fit than models which are non-linearly related to their parameters and because the statistical properties of the resulting estimators are easier to determine.

Mathematical Approach

Linear regression models are often fitted using the least squares approach, but they may also be fitted in other ways, such as by minimizing the “lack of fit” in some other norm (as with least absolute deviations regression), or by minimizing a penalized version of the least squares cost function as in ridge regression (L2-norm penalty) and lasso (L1-norm penalty). Conversely, the least squares approach can be used to fit models that are not linear models. Thus, although the terms “least squares” and “linear model” are closely linked, they are not synonymous.

Least Squares

Background & Founding

The method of least squares grew out of the fields of astronomy and geodesy, as scientists and mathematicians sought to provide solutions to the challenges of navigating the Earth’s oceans during the Age of Exploration. The accurate description of the behavior of celestial bodies was the key to enabling ships to sail in open seas, where sailors could no longer rely on land sightings for navigation.

The first clear and concise exposition of the method of least squares was published by Legendre in 1805.[8] The technique is described as an algebraic procedure for fitting linear equations to data and Legendre demonstrates the new method by analyzing the same data as Laplace for the shape of the earth. The value of Legendre’s method of least squares was immediately recognized by leading astronomers and geodesists of the time.[citation needed]

In 1809 Carl Friedrich Gauss published his method of calculating the orbits of celestial bodies. In that work he claimed to have been in possession of the method of least squares since 1795. This naturally led to a priority dispute with Legendre. However, to Gauss’s credit, he went beyond Legendre and succeeded in connecting the method of least squares with the principles of probability and to the normal distribution. He had managed to complete Laplace’s program of specifying a mathematical form of the probability density for the observations, depending on a finite number of unknown parameters, and define a method of estimation that minimizes the error of estimation. Gauss showed that the arithmetic mean is indeed the best estimate of the location parameter by changing both the probability density and the method of estimation. He then turned the problem around by asking what form the density should have and what method of estimation should be used to get the arithmetic mean as estimate of the location parameter. In this attempt, he invented the normal distribution.

An early demonstration of the strength of Gauss’s method came when it was used to predict the future location of the newly discovered asteroid Ceres. On 1 January 1801, the Italian astronomer Giuseppe Piazzi discovered Ceres and was able to track its path for 40 days before it was lost in the glare of the sun. Based on these data, astronomers desired to determine the location of Ceres after it emerged from behind the sun without solving Kepler’s complicated nonlinear equations of planetary motion. The only predictions that successfully allowed Hungarian astronomer Franz Xaver von Zach to relocate Ceres were those performed by the 24-year-old Gauss using least-squares analysis.

In 1810, after reading Gauss’s work, Laplace, after proving the central limit theorem, used it to give a large sample justification for the method of least squares and the normal distribution. In 1822, Gauss was able to state that the least-squares approach to regression analysis is optimal in the sense that in a linear model where the errors have a mean of zero, are uncorrelated, and have equal variances, the best linear unbiased estimator of the coefficients is the least-squares estimator. This result is known as the Gauss–Markov theorem.

The idea of least-squares analysis was also independently formulated by the American Robert Adrain in 1808. In the next two centuries workers in the theory of errors and in statistics found many different ways of implementing least squares.

Problem Statement

The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists of a point(data pairs) ( $x_i,y_i$ ), $i = 1, . . ., n$ , where $x_i$ is an independent variable and $y_i$ is a dependent variable whose value is found by observation. The model function has the form $\beta )$ , where $m$ adjustable parameters are held in the vector $\beta$ . The goal is to find the parameter values for the model that “best” fits the data. The fit of a model to a data point is measured by its residual, defined as the difference between the actual value of the dependent variable and the value predicted by the model: $r_i = y_i - f(x_i,\beta)$ . The lest-squares motheod find the optimal parameter values by minimizing the sum, $S$ , of squared residuals:
$S=\sum^{i}_{i=1}r_i^{2}$
An example of a model in two dimensions in that of the straight line. Denoting the y-intercept as $\beta_0$ and the slope as $\beta_1$ , the model functions is given by $f(x,\beta) = \beta_0 + \beta_1x$ . A data point may consist of more than one independent variable. In most general case there may be one or more independent variables and one or more dependent variables at each data point.

If a residual plot illustrating random fluctuations about $r_i = 0$ , indicating that a linear mode( $Y_i = \alpha + \beta x_i + U_i$ ) is appropriate. $U_i$ is an independent, random variable. If the residual plot had a parabolic shape, a parabolic model ( $Y_i = \alpha + \beta x_i + \gamma x_i^{2} + U_i$ ) would be appropriate model for the data.

In the model model above, the least squares approach to solving the problem is to try to make the sum of the squares of these residuals as small as possible; that is, to find the minimum of the function:
$S(\beta_0, \beta_1) = \sum r_i^{2}$
The minimum is determined by calculating the partial derivatives of $S(\beta_0, \beta_1)$ with respect to $\beta_0$ and $\beta_1$ and setting them to zero:
$\frac{\partial S}{\partial \beta_0} =2n\beta_0 + \sum 2(\beta_1x_i-y_i) \\ \frac{\partial S}{\partial \beta_1} = \sum (\beta_0 x_i + \beta_1 x_i^2 - y_ix_i) = 0$
thus,
$\beta_1 = \frac{\sum_{i}^{n}y_ix_i - n\bar{x}\bar{y}}{\sum_{i}^{n}x_i^2 - n\bar{x}^2}\\ \beta_0 = n\bar{y} - \beta_1 \bar{x}$
Some notations could be imported to simplify the form:
$S_{xy} = \sum _i^n x_iy_i\\ \beta_1 = \frac{S_{xy} - n\bar{x}\bar{y}}{S_{xx} - n\bar{x}^2}\\ \beta_0 = n\bar{y} - \beta_1 \bar{x}$

For application to higher order polynomial $\sum^{m-1}_{j=0} a_jx^j$
, we could consider matrix as follow:
$\begin{bmatrix} x_1^0 &x_1^1 &x_1^2 &\cdots &x_1^{m-1}\\ x_2^0 &x_2^1 &x_2^2 &\cdots &x_2^{m-1}\\ \vdots &\vdots &\vdots &\ddots &\vdots\\ x_n^0 &x_n^1 &x_n^2 &\cdots &x_n^{m-1}\\ \end{bmatrix} \begin{bmatrix} \beta_0\\ \beta_1\\ \vdots\\ \beta_n \end{bmatrix}= \begin{bmatrix} f(x_1)\\ f(x_2)\\ \vdots\\ f(x_n) \end{bmatrix}$
denotes as $X\vec{\Beta} = \vec{F}$ ,

$\begin{aligned} min \hspace{1mm} \varepsilon &= ||X\vec{\Beta} - \vec{Y}||^2 \\ &= (\vec{\Beta}^T X^T- \vec{Y}^T)(\vec{\Beta} X - \vec{Y})\\ &= \vec{\Beta}^TA^TA\vec{\Beta} - \vec{\Beta}^TA^T\vec{Y}-\vec{Y}^TA\vec{\Beta} + \vec{Y}^T\vec{Y}\\ &=\vec{\Beta}^TA^TA\vec{\Beta} - 2\vec{\Beta}^TA^T\vec{Y}+ \vec{Y}^T\vec{Y} \end{aligned}$
$\begin{aligned} \frac{\partial(\varepsilon)}{\partial \vec{\Beta}} &= \frac{\vec{\Beta}^TX^TX\vec{\Beta} - 2\vec{\Beta}^TX^T\vec{Y}+ \vec{Y}^T\vec{Y}}{\partial \vec{\Beta}} \\&=\frac{\partial(\vec{\Beta}^TX^TX\vec{\Beta})}{\partial \vec{\Beta}} - 2X^T\vec{Y}\\ &=2X^TX\vec{\Beta} - 2X^T\vec{Y}. \end{aligned}$
Therefore: $\vec{\Beta} = (X^TX)^{-1}X^T\vec{Y}$

Computer Programming

Matrix manipulation

import numpy as np 
from numpy.linalg import inv

dimension = int(input())

if dimension > 1:
    entries_x = list(map(int, input().split())) 
    entries_y = list(map(int, input().split()))

    X = np.array(entries_x).reshape(dimension,1)
    one_column = []
    X_0 = np.ones((dimension,1))
    X = np.hstack((X_0, X))
    
    Y = np.array(entries_y).reshape(dimension,1)
    
    X_trans = X.transpose()

    B = np.matmul(np.matmul(inv(np.matmul(X_trans, X)),X_trans), Y)

    if(abs(B[0][0]) > 1e-6): print("y = %-.3f"% B[0][0], "%+.3f" % B[1][0], "x")
    else: print("y =","%+.3f" % B[1][0], "x")

else:
    a = int(input())
    b = int(input())
    print("y = %.3f" % (a/b), "x")

Package sklearn.linear_model.LinearRegression


import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array(list(map(int,input().split()))).reshape(-1,1)

y = np.array(list(map(int,input().split())))

reg = LinearRegression().fit(X, y)
reg.score(X, y)

print("%.3f" % reg.intercept_ ) #beta_0
print("%.3f" % reg.coef_) #beta_1

Application

If the goal is prediction, forecasting, or error reduction, linear regression can be used to fit a predictive model to an observed data set of values of the response and explanatory variables. After developing such a model, if additional values of the explanatory variables are collected without an accompanying response value, the fitted model can be used to make a prediction of the response.
If the goal is to explain variation in the response variable that can be attributed to variation in the explanatory variables, linear regression analysis can be applied to quantify the strength of the relationship between the response and the explanatory variables, and in particular to determine whether some explanatory variables may have no linear relationship with the response at all, or to identify which subsets of explanatory variables may contain redundant information about the response.