DataWhale

1 SOME CONCEPTS

1.1 Supervised learning

There are input $x$ and label $y$.
What the supervised learning needs to do is evaluate $x$ to label $y^{{}^{\prime }}$ with clear purpose.
$$(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),(x^{(3)},y^{(3)})$$

1.2 Unsupervised learning

In unsupervised learning, there are only input $x$ without label $y$.
The method itself can classify the x to different categories named cluster.
$$(x^{(1)}),(x^{(2)}),(x^{(3)})$$

1.3 Overfitting & Underfitting

The curve fitting is far away from data as the left picture called underfitting.
The curve fitting too much to get the rule of the data like the right picture called overfitting.

1.4 Generalization

In machine learning, fitting a set of training data is not enough, you have to make the model work well to the set of test data which is named generalization.

1.5 Cross-validation

All the data can be divided some data sets. We choice one set to be valid-set, one to be test set, and others to be the training set each time.
The different choices of the valid-set from the data sets is called cross-validation.

2 LINEAR REGRESSION

2.1 Principle

For a data set $(X,Y)$, we use a linear function to fit the data.
The purpose is to predict the value of $y$ according to the given $x$ with optimal parameters.

2.2 Loss function, cost function and objective function

The loss function computes the error for a single training example.
The cost function is the average of the loss functions of the entire training set.
In linear regression, you have to minimum the structural risk or empirical risk function named objective function to fit just right.
$$|y_i-f(x_i)|$$ $$\frac{1}{N}\sum _{i=1}^N|y_i-f(x_i)|$$ $$min(\frac{1}{N}\sum _{i=1}^N|y_i-f(x_i)|+\lambda J(f))$$

2.3 Optimization

1. gradient descent
   In a linear regression model, compute each parameter’s gradient, and change the parameter towards the direction of gradient descent.
2. Newton method
   Input: objective function $f(x)$, gradient $g(x)=▽f(x)$, Hessian matrix $H(x)$, precision $ϵ$.
   Output: the minimum point of $f(x)$.
   steps:
   1. select the initial point randomly $x_0$, number of iteration $k=0$ ;
   2. computes the gradient $g(x_k)$ and Hessian matrix $H(x_k)$ of objective at point $x_k$, if $||g(x_k)||<ϵ$, stop it and the approximate solution is $x^{\ast }=x_k$ ;
   3. update the value of $x_{k+1}$ according to the equation: $$x_{k+1}=x_k-H^{-}(x_k)▽f(x_k)$$
3. Quasi-Newton method
   The basic idea of quasi-Newton method is replace $H^{-}(x_k)$ using $G(x_k)$ to simplify the calculation process in Newton method.
   The rule of replacement is as follows:

   The matrix $G(x_k)$ is positive ;
   $G(x_k)$ satisfy the quasi-Newton condition: $$G(x_k)(▽f(x_{k+1})-f(x_k))=x_{k+1}-x_k$$.

Obviously, the choice of $G(x_k)$ is not unique, the common algorithms for that are DEP, BFGS and Broyden algorithm.

2.4 Evaluation index of linear regression

• R-Squared(coefficient of determination)
  $$R^2=1-\frac{\sum (Y\_actual-Y\_predict{)}^2}{\sum (Y\_actual-Y\_mean{)}^2}$$
• Adjusted R-Squared(degree-of-freedom adjusted coefficient of determination)
  $$R^2\_adjusted=1-\frac{(1-R^2)(n-1)}{n-p-1}$$
• RMSE
  $$\sqrt{\frac{1}{N}\sum _{i=1}^N(Y\_predict-Y\_mean{)}^2}$$
• MSE
  $$MSE=\frac{1}{N}\sum _{i=1}^N(Y\_predict-Y\_mean{)}^2$$
• MAE
  $$MAE=\frac{1}{N}\sum _{i=1}^N|Y\_predict-Y\_mean|$$
• SSE
  $$SSE=\sum (Y\_actual-Y\_predict{)}^2$$
• F Statistic

2.5 Parameters of sklearn

call the function in sklearn

sklearn.linear_model.LinearRegression(fit_intercept=True, normalize=False, copy_X=True, n_jobs=1)

• fit_intercept: if the intercept equals zero
• normalize: whether to normalize the data
• copy_X: if X will be overwritten
• n_jobs: number of cores used in calculation

REFERENCE

[1] https://yoyoyohamapi.gitbooks.io/mit-ml/content/大规模机器学习/articles/梯度下降.html

[2] https://developers.google.com/machine-learning/crash-course/prereqs-and-prework?hl=zh-cn

[3] https://blog.csdn.net/shy19890510/article/details/79375062

[4] https://blog.csdn.net/li980828298/article/details/51273385

[5] https://zhuanlan.zhihu.com/p/38185542

[6] https://blog.csdn.net/zrh_CSDN/article/details/81190221

[7] https://www.zhihu.com/question/52398145