Huber Loss function

Huber loss是为了增强平方误差损失函数（squared loss function）对噪声（或叫离群点，outliers）的鲁棒性提出的。

Definition

$$\begin{split} L_\delta(a)=\left \{ \begin{array}{ll} \frac12a^2,&\textrm{for } |a|\leq\delta,\\ \delta\cdot(|a|-\frac12\delta),&\textrm{otherwise.} \end{array} \right. \end{split}$$

参数$a$通常表示residuals，也即$(y-\hat{y})$或者写作$(y-f(x))$，当$a=y-f(x)$时，上述形式可以拓展为：

$$\begin{split} L_\delta(y, f(x))=\left\{ \begin{array}{ll} \frac12(y-f(x))^2,&\textrm{for }|y-f(x)|\leq\delta\\ \delta\cdot(|y-f(x)|-\frac12\delta),&\textrm{otherwise.} \end{array} \right. \end{split}$$

visualization

import numpy as np
import matplotlib.pyplot as plt

def huber_loss(e, d):
    return (abs(e)<=d)*e**2/2 + (e>d)*d*(abs(e)-d/2)

plt.figure(figsize=(6, 4.5), facecolor='w', edgecolor='k')
x = np.arange(-20, 20)
plt.plot(x, x**2/2, label='squared loss', lw=2, 'g')
for d in (10, 5, 3, 1):
    plt.plot(x, huber_loss(x, d), label=r'huber loss: $\delta$={}'.format(d), lw=2)

plt.legend(loc='best', frameon=False)
plt.xlabel('standard deviation')
plt.ylabel('loss')
plt.show()

References

[1] Huber Loss Wikipedia