GD:梯度下降
while True:
loss = f(params)
d_loss_wrt_params = ...
params -= eta * d_loss_wrt_params
if <stopping condition met>:
return params
SGD:随机梯度下降
逐样本训练:
for x_i, y_i in training_data:
loss = f(params, x_i, y_i)
d_loss_wrt_params = ...
params -= eta * d_loss_wrt_params
if <stopping condition met>:
return params
更进一步,如果外层还有一个 epochs:
for j in range(epochs):
random.shuffle(training_data)
for x_i, y_i in training_data:
...
MSGD(Minibatch SGD):块随机梯度下降
n = len(training_data)
mini_batch_size = ...
mini_batches = [training_data[k:k+mini_batch_size] for k in range(0, n, mini_batch_size)]
for mini_batch in mini_batches:
loss = f(params, mini_batch)
d_loss_wrt_params = ...
params -= eta * d_loss_wrt_params
if <stopping condition met>:
return params