使用梯度下降与牛顿法求解最小平方和问题

问题

已知：
$h_{W}(X) = \sum _{j = 1} ^{n} w_j x_j + w_{n + 1} = \sum _{j = 1} ^{n + 1} w_j x_j = X ^{\intercal} W,$
其中 $W = \begin{pmatrix} w_1 \\ \vdots \\ w_n \\ w_{n + 1} \end{pmatrix}, X = \begin{pmatrix} x_1 \\ \vdots \\ x_n \\ 1 \end{pmatrix}$
令 $\mathbf {X} = \begin{pmatrix} X_1^{\intercal} \\ \vdots \\ X_m^{\intercal} \end{pmatrix}, Y = \begin{pmatrix} y_1 \\ \vdots \\ y_m \end{pmatrix},$
${L} (W) = \dfrac {1} {2} \sum _{i = 1} ^{m} \left [ h_{W}(X_i) - y_i \right ] ^2 = \dfrac {1} {2} \sum _{i = 1} ^{m} \left ( X_i^{\intercal} W - y_i \right ) ^2$
$= \dfrac {1} {2} \Vert \mathbf {X}W - Y \Vert_{2} ^{2} = \dfrac {1} {2} \left (\mathbf {X}W - Y \right) ^{\intercal} \left (\mathbf {X}W - Y \right )$
求 $\min {L} (W)$

解

梯度下降法

$\operatorname {d} {L} (W) = \operatorname {d} \left [ \dfrac {1} {2} \left (\mathbf {X}W - Y \right) ^{\intercal} \left (\mathbf {X}W - Y \right ) \right] =\left (\mathbf {X}W - Y \right) ^{\intercal} \operatorname {d} \left (\mathbf {X}W - Y \right) = \left (\mathbf {X}W - Y \right) ^{\intercal} \mathbf {X} \operatorname {d} W$
因此 $\nabla {L} (W) = \left [ \left (\mathbf {X}W - Y \right) ^{\intercal} \mathbf {X} \right ] ^{\intercal} = \mathbf {X} ^{\intercal} (\mathbf {X}W - Y) = \mathbf {X} ^{\intercal} \mathbf {X}W - \mathbf {X} ^{\intercal} Y$
令 $\Delta W = - \epsilon \nabla f(W_0) ,$
于是可取 $W_1 = W_0 + \Delta W = W_0 - \epsilon \nabla f(W_0) = W_0 - \epsilon \left ( \mathbf {X} ^{\intercal} \mathbf {X}W_0 - \mathbf {X} ^{\intercal} Y \right )$

牛顿法

$\operatorname {d} \nabla {L} (W) = \operatorname {d} \left [ \mathbf {X} ^{\intercal} \mathbf {X}W - \mathbf {X} ^{\intercal} Y \right ] = \mathbf {X} ^{\intercal} \mathbf {X} \operatorname {d} W$
因此 $\operatorname {H} _{{L} (W)} = \mathbf {X} ^{\intercal} \mathbf {X}$
令 $g( \Delta W) = f(W_0 + \Delta W) = f(x_0) + \left [ \nabla f(W_0) \right ] ^{\intercal} \Delta W + \dfrac {1} {2} \left ( \Delta W \right ) ^{\intercal}\operatorname {H} _{f(W_0)} \Delta W$
则 $\operatorname {d} g( \Delta W) = \left [ \nabla f(W_0) \right ] ^{\intercal} \operatorname {d} ( \Delta W ) + \left ( \Delta W \right ) ^{\intercal}\operatorname {H} _{f(W_0)} \operatorname {d} ( \Delta W ) = \left \{ \left [ \nabla f(W_0) \right ] ^{\intercal} + \left ( \Delta W \right ) ^{\intercal}\operatorname {H} _{f(W_0)} \right \} \operatorname {d} ( \Delta W )$
因此 $\nabla g( \Delta W) = \nabla f(W_0) + \operatorname {H} _{f(W_0)} \Delta W$
令 $\nabla g( \Delta W) = \vec 0,$
则 $\Delta W = - \left ( \operatorname {H} _{f(W_0)} \right ) ^{-1} \nabla f(W_0)$
于是可取 $W_1 = W_0 + \Delta W = W_0 - \left ( \operatorname {H} _{f(W_0)} \right ) ^{-1} \nabla f(W_0)$
$= W_0 - \left ( \mathbf {X} ^{\intercal} \mathbf {X} \right ) ^{-1} \left ( \mathbf {X} ^{\intercal} \mathbf {X}W_0 - \mathbf {X} ^{\intercal} Y \right) =W_0 - W_0 + \left ( \mathbf {X} ^{\intercal} \mathbf {X} \right ) ^{-1} \mathbf {X} ^{\intercal} Y$
$= \left ( \mathbf {X} ^{\intercal} \mathbf {X} \right ) ^{-1} \mathbf {X} ^{\intercal} Y$
因此 $\min {L} (W) = {L} (W_1) =\mathbf {X} \left ( \mathbf {X} ^{\intercal} \mathbf {X} \right ) ^{-1} \mathbf {X} ^{\intercal} Y - Y = (\mathbf {X} \left ( \mathbf {X} ^{\intercal} \mathbf {X} \right ) ^{-1} \mathbf {X} ^{\intercal} - I) Y$

梯度的另一种求法

由 ${L} (W) = \dfrac {1} {2} \sum _{i = 1} ^{m} \left [ h_{W}(X_i) - y_i \right ] ^2$
得 $\dfrac {\partial L} {\partial w_j} = \sum _{i = 1} ^{m} \left [ h_{W}(X_i) - y_i \right ] \dfrac {\partial h_{W}(X_i) } {\partial w_j} = \sum _{i = 1} ^{m} \left [ h_{W}(X_i) - y_i \right ] x_{ij}$
$= \sum _{i = 1} ^{m} \left ( X_i ^{\intercal} W- y_i \right ) x_{ij} = (\mathbf {X}W - Y) ^{\intercal} X_{:j}$
因此 $\nabla {L} (W) = \mathbf {X} ^{\intercal} (\mathbf {X}W - Y) = \mathbf {X} ^{\intercal} \mathbf {X}W - \mathbf {X} ^{\intercal} Y$