这一节主要讲了一些适用于大规模机器学习的算法及策略,并比较了梯度下降、随机梯度下降和小批量梯度下降的优劣。目前来说,大规模机器学习中用的最多的还是小批量梯度下降,毕竟它在执行效率和性能之间达到了一个平衡。当然,对于小批量梯度下降来说,如何选择合适的批量大小又是一个值得深思的问题。
梯度下降
Repeat { Θ t + 1 : = Θ t − α 1 n ∑ i = 1 n ∇ cost ( f ( x ( i ) ; Θ t ) , y ( i ) ) } \begin{aligned} &\text{Repeat}\{\\ &\quad\quad\Theta_{t+1} := \Theta_t - \alpha\frac{1}{n}\sum_{i=1}^n\nabla\text{cost}\left(f(x^{(i)};\Theta_t),y^{(i)}\right)\\ &\} \end{aligned} Repeat{Θt+1:=Θt−αn1i=1∑n∇cost(f(x(i);Θt),y(i))}
随机梯度下降
Repeat { Shuffer training dataset randomly Select one example from training dataset Θ t + 1 : = Θ t − α ∇ cost ( f ( x ( i ) ; Θ t ) , y ( i ) ) } \begin{aligned} &\text{Repeat}\{\\ &\quad\quad\text{Shuffer training dataset randomly}\\ &\quad\quad \text{Select }\textbf{one}\text{ example from training dataset}\\ &\quad\quad\Theta_{t+1} := \Theta_t - \alpha\nabla\text{cost}\left(f(x^{(i)};\Theta_t),y^{(i)}\right)\\ &\} \end{aligned} Repeat{Shuffer training dataset randomlySelect one example from training datasetΘt+1:=Θt−α∇cost(f(x(i);Θt),y(i))}
小批量梯度下降
Repeat
{
Shuffer training data randomly
Select
b
examples from training dataset
Θ
t
+
1
:
=
Θ
t
−
α
1
b
∑
i
=
1
b
∇
cost
(
f
(
x
(
i
)
;
Θ
t
)
,
y
(
i
)
)
}
\begin{aligned} &\text{Repeat}\{\\ &\quad\quad\text{Shuffer training data randomly}\\ &\quad\quad\text{Select }b\text{ examples from training dataset}\\ &\quad\quad\Theta_{t+1} := \Theta_t - \alpha\frac{1}{b}\sum_{i=1}^b\nabla\text{cost}\left(f(x^{(i)};\Theta_t),y^{(i)}\right)\\ &\} \end{aligned}
Repeat{Shuffer training data randomlySelect b examples from training datasetΘt+1:=Θt−αb1i=1∑b∇cost(f(x(i);Θt),y(i))}
后面介绍了分布式机器学习中的Map-Reduce方法,其实现阶段(2020年)常用的方法还是数据并行和模型并行这两种。主要使用的架构包括参数服务器以及一些环状架构,这里就不作过多介绍了。