Tensorflow中支持11中不同的优化器，包括：

• tf.train.Optimizer
• tf.train.MomentumOptimizer
• tf.train.FtrlOptimizer
• tf.train.RMSPropOptimizer

$W^{[l]} = W^{[l]} - \alpha \text{ } dW^{[l]}$$b^{[l]} = b^{[l]} - \alpha \text{ } db^{[l]}$

（2） Momentum

optimizer = tf.train.MomentumOptimizer(learning_rate,momentum).minimize(loss)

$\begin{cases} v_{dW^{[l]}} = \beta v_{dW^{[l]}} + (1 - \beta) dW^{[l]} \\ v_{db^{[l]}} = \beta v_{db^{[l]}} + (1 - \beta) db^{[l]} \end{cases}$

$\begin{cases} W^{[l]} = W^{[l]} - \alpha v_{dW^{[l]}} \\ b^{[l]} = b^{[l]} - \alpha v_{db^{[l]}} \end{cases}$

$\beta$ : the momentum
$\alpha$ : the learning rate

beta1=0.9, beta2=0.999,
epsilon=1e-08).minimize(loss)

$\begin{cases} v_{dW^{[l]}} = \beta_1 v_{dW^{[l]}} + (1 - \beta_1) \frac{\partial \mathcal{J} }{ \partial W^{[l]} } \\ v_{db^{[l]}} = \beta_1 v_{db^{[l]}} + (1 - \beta_1) \frac{\partial \mathcal{J} }{ \partial b^{[l]} } \end{cases} (moment：\beta_1)$

$\begin{cases} s_{dW^{[l]}} = \beta_2 s_{dW^{[l]}} + (1 - \beta_2) (\frac{\partial \mathcal{J} }{\partial W^{[l]} })^2 \\ s_{db^{[l]}} = \beta_2 s_{db^{[l]}} + (1 - \beta_2) (\frac{\partial \mathcal{J} }{\partial b^{[l]} })^2 \end{cases} (RMSprop:\beta_2)$

$\begin{cases} v^{corrected}_{dW^{[l]}} = \frac{v_{dW^{[l]}}}{1 - (\beta_1)^t} \\ v^{corrected}_{dW^{[b]}} = \frac{v_{dW^{[b]}}}{1 - (\beta_1)^t} \\ s^{corrected}_{dW^{[l]}} = \frac{s_{dW^{[2]}}}{1 - (\beta_1)^t} \\ s^{corrected}_{dW^{[b]}} = \frac{s_{dW^{[2]}}}{1 - (\beta_1)^t} \end{cases} (Bias correction)$

$\begin{cases} W^{[l]} = W^{[l]} - \alpha \frac{v^{corrected}_{dW^{[l]}}}{\sqrt{s^{corrected}_{dW^{[l]}}} + \varepsilon}\\ b^{[l]} = b^{[l]} - \alpha \frac{v^{corrected}_{db^{[l]}}}{\sqrt{s^{corrected}_{db^{[l]}}} + \varepsilon} \end{cases}$

• $\beta_1$ and $\beta_2$ are hyperparameters that control the two exponentially weighted averages.
• $\alpha$ is the learning rate
• $\varepsilon$ is a very small number to avoid dividing by zero

12-11 1万+

06-01 1万+

04-25 224

08-03 1万+

11-01 1589

12-31 1948

12-19 1634

08-16 9万+

07-10 1430

06-12 838

02-11 294

04-14 2万+

11-10 76

07-11 2982

12-31 148

11-25 4825

06-28 338

04-06 999

04-14 61万+

03-13 15万+

02-19 19万+

03-01 14万+

03-04 14万+

03-06 4634

03-08 7万+

04-25 7万+

03-10 13万+

03-10 19万+

03-12 12万+

03-13 12万+

我入职阿里后，才知道原来简历这么写

©️2019 CSDN 皮肤主题: 大白 设计师: CSDN官方博客