Tikhonov regularization 吉洪诺夫 正则化

        这个知识点很重要，但是，我不懂。

        第一个问题：为什么要做正则化？

In mathematics, statistics, and computer science, particularly in the fields of machine learning and inverse problems, regularization is a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting. 

And, what is ill-posed problem?... ...

And, what is overfitting? In statistics, overfitting is "the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably", as the next figure shows. 

Figure 1.  The green curve represents an overfitted model and the black line represents a regularized model. While the green line best follows the training data, it is too dependent on that data and it is likely to have a higher error rate on new unseen data, compared to the black line.

        第二个问题：常用的正则化方法有哪些？

        第三个问题：The advantages fo Tikhonov regularizatioin

        The fourth question: Tikhonov regularization

        Tikhonov regularization, named for Andrey Tikhonov, is the most commonly used method of regularization of ill-posed problems. In statistics, the method is known as ridge regression, in machine learning it is known as weight decay, and with multiple independent discoveries, it is also variously known as the Tikhonov–Miller method, the Phillips–Twomey method, the constrained linear inversion method, and the method of linear regularization. It is related to the Levenberg–Marquardt algorithm for non-linear least-squares problems.

        Suppose that for a known matrix A and vector b, we wish to find a vector x such that: