自适应梯度和RMSprop的理解与实现

最新推荐文章于 2023-07-14 15:04:29 发布

因吉

最新推荐文章于 2023-07-14 15:04:29 发布

阅读量789

点赞数 2

分类专栏： # 机器学习文章标签：自适应梯度 RMSprop FanSmale 因吉

本文链接：https://blog.csdn.net/weixin_44575152/article/details/112672859

版权

机器学习专栏收录该内容

23 篇文章 8 订阅

订阅专栏

文章目录

1 自适应梯度Adagradient
2 RMSprop

1 自适应梯度Adagradient

回顾动量版的SGD更新公式：
$\begin{aligned} v_t & = \gamma v_{t - 1} + lr * grad\\ \theta & = \theta - v_t = \theta - \gamma v_{t - 1} - lr * grad \end{aligned}$ 这里的 $l r$ 是一个固定值：越靠近目标值的位置，梯度的变化俞小，参数的变慢也俞慢。有没有方法解决呢？
此时一个带着参数 $s_t$ 的靓仔缓缓走来：
$\begin{aligned} s_t & = s_{t - 1} + grad_t^2\\ w_t & =w_{t - 1} - grad_t \frac{lr}{\sqrt{s_t + \epsilon}} \end{aligned}$ 把它加进动量版的SGD试试😋

还是以以下目标为例：
$\begin{aligned} f(x) &= a x+b \\ (y-f(x))^{2} & =(y-(a x+b))^{2} \\ \frac{d y}{d a} &=-2 x(y-(a x+b)) \\ \frac{d y}{d b} &=-2(y-(a x+b)) \end{aligned}$

测试1： 加入自适应梯度的SGD算法

import numpy as np
import matplotlib.pyplot as plt
from sympy import symbols, diff


def get_data():

    ret_x = np.linspace(-1, 1, 100)
    return ret_x, [(lambda x: 2 * x + 3)(x) for x in ret_x]


def grad():

    x, y, w_a, w_b = symbols(["x", "y", "w_a", "w_b"])
    loss = (y - (w_a * x + w_b))**2
    return diff(loss, w_a), diff(loss, w_b)


def test1(n_iter=10, lr=0.5, batch_size=20, epsilon=1e-6, shuffle=True):
    x, y = get_data()
    ga, gb = grad()
    n = len(x)
    idx = np.random.permutation(n)
    s_a, s_b, w_a, w_b = 0, 0, 0, 0
    move_a, move_b = [w_a], [w_b]
    move_lr_a, move_lr_b = [lr], [lr]
    for i in range(n_iter):
        if shuffle:
            np.random.shuffle(idx)
        batch_idxes = [idx[k: k + batch_size] for k in range(0, n, batch_size)]
        for idxes in batch_idxes:
            sum_ga, sum_gb = 0, 0
            for j in idxes:
                sum_ga += ga.subs({"x": x[j], "y": y[j], "w_a": w_a, "w_b": w_b})
                sum_gb += gb.subs({"x": x[j], "y": y[j], "w_a": w_a, "w_b": w_b})
            sum_ga /= batch_size
            sum_gb /= batch_size
            s_a += sum_ga**2
            s_b += sum_gb**2
            move_lr_a.append(lr / np.sqrt(float(s_a) + epsilon))
            move_lr_b.append(lr / np.sqrt(float(s_b) + epsilon))
            w_a -= sum_ga * move_lr_a[-1]
            w_b -= sum_gb * move_lr_b[-1]
            move_a.append(w_a)
            move_b.append(w_b)
    plt.subplot(211)
    plt.plot(move_a)
    plt.plot(move_b)
    plt.legend(["a", "b"])
    plt.subplot(212)
    plt.plot(move_lr_a)
    plt.plot(move_lr_b)
    plt.legend(["a", "b"])
    plt.show()


if __name__ == '__main__':
    test1()

2 RMSprop

上面老几的问题是学习率变着变着就不变了，难受对吧？所以现在这个老几加了gamma参数，注意看代码中的区别哟😊
😂

测试2： 加入gamma的自适应学习SGD

import numpy as np
import matplotlib.pyplot as plt
from sympy import symbols, diff


def get_data():

    ret_x = np.linspace(-1, 1, 100)
    return ret_x, [(lambda x: 2 * x + 3)(x) for x in ret_x]


def grad():

    x, y, w_a, w_b = symbols(["x", "y", "w_a", "w_b"])
    loss = (y - (w_a * x + w_b))**2
    return diff(loss, w_a), diff(loss, w_b)


def test2(n_iter=10, lr=0.5, batch_size=20, epsilon=1e-6, shuffle=True, gamma=0.9):
    x, y = get_data()
    ga, gb = grad()
    n = len(x)
    idx = np.random.permutation(n)
    s_a, s_b, w_a, w_b = 0, 0, 0, 0
    move_a, move_b = [w_a], [w_b]
    move_lr_a, move_lr_b = [lr], [lr]
    for i in range(n_iter):
        if shuffle:
            np.random.shuffle(idx)
        batch_idxes = [idx[k: k + batch_size] for k in range(0, n, batch_size)]
        for idxes in batch_idxes:
            sum_ga, sum_gb = 0, 0
            for j in idxes:
                sum_ga += ga.subs({"x": x[j], "y": y[j], "w_a": w_a, "w_b": w_b})
                sum_gb += gb.subs({"x": x[j], "y": y[j], "w_a": w_a, "w_b": w_b})
            sum_ga /= batch_size
            sum_gb /= batch_size
            """Note the difference."""
            s_a = gamma * s_a + sum_ga**2
            s_b = gamma * s_b + sum_gb**2
            move_lr_a.append(lr / np.sqrt(float(s_a) + epsilon))
            move_lr_b.append(lr / np.sqrt(float(s_b) + epsilon))
            w_a -= sum_ga * move_lr_a[-1]
            w_b -= sum_gb * move_lr_b[-1]
            move_a.append(w_a)
            move_b.append(w_b)
    plt.subplot(211)
    plt.plot(move_a)
    plt.plot(move_b)
    plt.legend(["a", "b"])
    plt.subplot(212)
    plt.plot(move_lr_a)
    plt.plot(move_lr_b)
    plt.legend(["a", "b"])
    plt.show()


if __name__ == '__main__':
    test2()

参考文献
【1】https://towardsdatascience.com/introduction-and-implementation-of-adagradient-rmsprop-fad64fe4991

因吉

关注

2
点赞
踩
1

收藏

觉得还不错? 一键收藏
2
评论
自适应梯度和RMSprop的理解与实现

文章目录1 自适应梯度Adagradient2 RMSprop1 自适应梯度Adagradient 回顾动量版的SGD更新公式：vt=γvt−1+lr∗gradθ=θ−vt=θ−γvt−1−lr∗grad\begin{aligned}v_t & = \gamma v_{t - 1} + lr * grad\\\theta & = \theta - v_t = \theta - \gamma v_{t - 1} - lr * grad\end{aligned}vtθ=γv
复制链接

扫一扫