Neural Network 5 优化器

神经网络参数优化器
引导参数优化的方法，

1.SGD

没有momentum
$$W_{t+1}=W_t-lr\ast \frac{\partial loss}{\partial W_t}$$

2.SGDM

含有momentum，在SGD基础上增加一阶动量
β 一般取值0.9
$$\begin{gathered} m_t=\beta \cdot m_{t-1}+(1-\beta )\cdot g_t,v_t=1 \\ {\eta }_t=lr\cdot \frac{m_t}{\sqrt{v_t}}=lr\cdot (\beta \cdot m_{t-1}+(1-\beta )\cdot g_t) \\ {W}_{t+1}=W_t-lr\ast lr\cdot (\beta \cdot m_{t-1}+(1-\beta )\cdot g_t) \end{gathered}$$

#更新梯度
        m_w = beta * m_w + (1 - beta) * m_w
        m_b = beta * m_b + (1 - beta) * m_b
        w1.assign_sub(lr * grads[0])
        b1.assign_sub(lr * grads[1])

(3).Adagrad

加入二阶动量,g_t表示梯度
$$\begin{gathered} m_t=g_t,v_t=\stackrel{t}{\underset{\tau =1}{\sum g_{\tau }^2}} \\ {\eta }_t=lr\cdot \frac{m_t}{\sqrt{v_t}}=lr\cdot \frac{g_t}{\sqrt{\stackrel{t}{\underset{\tau =1}{\sum g_{\tau }^2}}}} \\ {W}_{t+1}=W_t-lr\ast lr\cdot \frac{g_t}{\sqrt{\stackrel{t}{\underset{\tau =1}{\sum g_{\tau }^2}}}} \end{gathered}$$

        v_w +=tf.square(grads[0])
        v_b +=tf.square(grads[1])
        w1.assign_sub(lr * grads[0] / tf.sqrt(v_w))
        b1.assign_sub(lr * grads[1] / tf.sqrt(v_b))

(4).RMSProp

增加二阶动力
$$\begin{gathered} m_t=g_t,v_t=\beta \cdot v_{t-1}+(1-\beta )\cdot g_t^2 \\ {\eta }_t=lr\cdot \frac{m_t}{\sqrt{v_t}}=lr\cdot g_t/(\sqrt{\beta \cdot v_{t-1}+(1-\beta )\cdot g_t^2}) \\ {W}_{t+1}=W_t-lr\cdot g_t/(\sqrt{\beta \cdot v_{t-1}+(1-\beta )\cdot g_t^2}) \end{gathered}$$

        v_w =beta * v_w + (1 - beta) * tf.square(grads[0])
        v_b =beta * v_b + (1 - beta) * tf.square(grads[1])
        w1.assign_sub(lr * grads[0] / tf.sqrt(v_w))
        b1.assign_sub(lr * grads[1] / tf.sqrt(v_b))

(5)Adam 优化器

同时结合SGDM一阶动量和RMSProp二阶动量
$$\begin{gathered} m_t={\beta }_1\cdot m_{t-1}+(1-{\beta }_1)\cdot g_t \\ 修正一阶动量的偏差：\overbrace{m_t}=\frac{m_t}{1-{\beta }_1^t} \\ {v}_t={\beta }_2\cdot v_{step-1}+(1-{\beta }_2)\cdot g_t^2 \\ 修正二阶动量的偏差：\overbrace{v_t}=\frac{v_t}{1-{\beta }_2^t} \\ {\eta }_t=lr\cdot \frac{m_t}{\sqrt{v_t}}=lr\cdot {\frac{m_t}{1-{\beta }_1^t}}_t/(\sqrt{\frac{v_t}{1-{\beta }_2^t}}) \\ {W}_{t+1}=W_t-lr\cdot {\frac{m_t}{1-{\beta }_1^t}}_t/(\sqrt{\frac{v_t}{1-{\beta }_2^t}}) \end{gathered}$$

#学习率和画图用的参数的存储
lr = 0.1
train_loss_results = []
test_acc = []
epoch = 500
loss_all = 0
#加入优化器参数
m_w,m_b=0,0
v_w,v_b=0,0
beta1=0.9
beta2=0.999
delta_w,delta_b = 0,0
global_step=0

#训练 epoch 是整个数据集 而第二个for是一个batch
for epoch in range(epoch):
    for step , (x_train,y_train) in enumerate(train_db):
        #更新
        global_step += 1
        with tf.GradientTape() as tape:
            y = tf.matmul(x_train , w1) + b1
            y = tf.nn.softmax(y)
            y_ = tf.one_hot(y_train,depth=3)
            loss = tf.reduce_mean(tf.square(y_ - y))
            loss_all += loss.numpy()

        grads = tape.gradient(loss,[w1 , b1])

        #更新梯度adma
        m_w = beta1 * m_w + (1 - beta1) * grads[0]
        m_b = beta2 * m_b + (1 - beta1) * grads[1]
        v_w = beta2 * v_w + (1 - beta2) * tf.square(grads[0])
        v_b = beta2 * v_b + (1 - beta2) * tf.square(grads[1])

        m_w_correction = m_w / (1 - tf.pow(beta1,int(global_step)))
        m_b_correction = m_b / (1 - tf.pow(beta1,int(global_step)))
        v_w_correction = v_w / (1 - tf.pow(beta2,int(global_step)))
        v_b_correction = v_b / (1 - tf.pow(beta2,int(global_step)))

        w1.assign_sub(lr * m_w_correction / tf.sqrt(v_w_correction))
        b1.assign_sub(lr * m_b_correction / tf.sqrt(v_b_correction))

        #m每个epoch 打印loss的值
    print("Epoch {},loss {}:".format(epoch,loss_all/4))
    train_loss_results.append(loss_all/4)
    loss_all = 0