Regularization

Introduction

In mathematics, statistics and computer science, particularly in machine learning and inverse problems, regularization is the process of adding information in order to solve an ill-posed problem or to prevent overfitting.

Regularization applies to objective functions in ill-posed optimization problems.

Classification

One use of regularization is in classification. Empirical learning of classifiers (from a finite data set) is always an underdetermined problem, because it attempts to infer a function of any $x$ given only examples $x_1,x_2,...,x_3$.

A regularization term (or regularizer) $R(f)$ is add to a loss function:
$$mi{n}_f\sum _{i=1}^{n}V(f(x_i),y_i))+\lambda R(f)$$ where
$V$ —— an underlying loss function that describes the cost of predicting $f(x)$ when the label is $y$, such as the square loss or hinge loss.
$\lambda$ —— a parameter which controls the importance of the regularization term.
$R(f)$ —— is typically chosen to impose a penalty on the complexity of $f$.

Reference

Wiki
http://wap.sciencenet.cn/blog-261330-648033.html?mobile=1