回顾线性回归,logistic回归和softmax。
LMS
- 先构造线性函数进行拟合: h ( x ) = θ 0 + θ 1 x 1 + θ 2 x 2 h(x) = \theta_0 + \theta_1 x_1 + \theta_2 x_2 h(x)=θ0+θ1x1+θ2x2
- 定义cost function: J ( θ ) = 1 2 ∑ ( h ( x i ) − y i ) 2 J(\theta) = \frac{1}{2} \sum (h(x^{i}) - y^i)^2 J(θ)=21∑(h(xi)−yi)2
- 因此,可使用梯度下降进行求解
- gradient descent algorithm: θ i : = θ j − α ∂ ∂ θ j J ( θ ) \theta_i := \theta_j - \alpha\frac{\partial} {\partial \theta_j} J(\theta) θi:=θj−α∂θj∂J(θ)
- LMS update rule(Widrow-Hoff learning rule): θ j : = θ j + α ( y i − h ( x i ) ) x j i \theta_j := \theta_j +\alpha (y^i - h(x^i))x_j^i θ