梯度上升的老虎机算法

梯度上升的老虎机

接上和概要,我们定义
$$E[R_t]=\sum _b{\pi }_t(b)q(b)$$
其中${\sum }_b{\pi }_t(b)=1$表示在时刻t下选择各个动作b的概率和为1.
我们想极大化此期望,为此引入变量$H_t(a)$表示在时间t下对时间a的偏好,我们初始定义其均为0,即$H_0(a)=0$.
为与上述期望产生联系,我们利用softmax操作定义每个时刻t下选择动作a的概率:
$${\pi }_t(a)=\frac{e^{H_t(a)}}{{\sum }_{b=1}^n{e}^{H_t(b)}}$$
这样,期望E便是偏好$\vec{H}$的函数,我们在每步只要更新偏好$\vec{H}$的值即可.利用梯度上升:
$$H_{t+1}(a)=H_t(a)+\alpha \frac{\partial E[R_t]}{\partial H_t(a)}$$
其中$\alpha$为常数步长.经过下列推导,我们可得到最终更新格式:
$$\begin{gathered} H_{t+1}(A_t)=H_t(A_t)+\alpha (R_t-\overline{R_t})(1-{\pi }_t(A_t)), \\ {H}_{t+1}(a)=H_t(a)-\alpha (R_t-\overline{R_t}){\pi }_t(a),\forall a\ne A_t \end{gathered}$$
其中$A_t$为t时刻选择的动作, $\overline{R_t}$为前t时刻所有$R_k,k\le n$的平均值,加快收敛的作用(类似如同深度学习的中的归一化),换成其它数列也可.
以下为推导过程:
$$\begin{array}{rl}\frac{\partial E[R_t]}{\partial H_t(a)}& =\frac{\partial }{\partial H_t(a)}[\sum _b{\pi }_t(b)q(b)]\\ & =\sum _{b}q(b)\frac{\partial {\pi }_t(b)}{\partial H_t(a)}\\ & =\sum _b(q(b)-X_t)\frac{\partial {\pi }_t(b)}{\partial H_t(a)}\\ & =\sum _b{\pi }_t(b)(q(b)-X_t)\frac{\partial {\pi }_t(b)}{\partial H_t(a)}/{\pi }_t(b)\\ & =E[(q(A_t)-X_t)\frac{\partial {\pi }_t(A_t)}{\partial H_t(a)}/{\pi }_t(A_t)]\\ & =E[(R_t-\overline{R_t})\frac{\partial {\pi }_t(A_t)}{\partial H_t(a)}/{\pi }_t(A_t)]\\ & =E[(R_t-\overline{R_t}){\pi }_t(A_t)(\prod _{a=A_t}-{\pi }_t(a))/{\pi }_t(A_t)]\\ & =E[(R_t-\overline{R_t})(\prod _{a=A_t}-{\pi }_t(a))]\end{array}$$

其中$X_t$不与动作有关, ${\prod }_{a=A_t}(a)=1$如果$a=A_t$,不然为0.第三个等式成立时因为(根据softmax函数求导):
$$\sum _b\frac{\partial {\pi }_t(b)}{\partial H_t(a)}=0$$
第五个等式利用了数学期望的定义.第六个等式利用期望的性质,可将期望内的变量替换为等期望的变量.倒数第二个等式成立是对softmax函数求导.

以下为代码:

'''
It's an algorithm using stochastic gradient ascent to solve the Bandits problem
'''
import numpy as np
import matplotlib.pyplot as plt

np.random.seed(42)
n = 10
q = np.random.randn(n)
T = 1000
tspan = np.arange(T)

class GradientBandit:
    def __init__(self, n, q) -> None:
        self.n = n
        self.q = q
        self.Actions = np.arange(n)
    def softmax(self,H):
        return np.exp(H) / np.sum(np.exp(H))
    def play(self, T, alpha, baseline=True):
        ActionRecord = []
        Reward_avg = [0]
        H = np.zeros(self.n)
        for t in range(1,T):
            a = np.random.choice(np.arange(n), p=self.softmax(H))  # action
            ActionRecord.append(a)
            R = q[a] + np.random.randn()  # repay
            avg = (R + Reward_avg[-1] * t) / (t + 1)
            Reward_avg.append(avg)
            if baseline:
                H = H + alpha * (R - Reward_avg[-1]) * (np.eye(self.n)[a] - self.softmax(H))
            else:
                H = H + alpha * (R - 0) * (np.eye(self.n)[a] - self.softmax(H))
        return ActionRecord, Reward_avg  
        
if __name__ == "__main__":
    slot_machine = GradientBandit(n, q)
    actions, reward_avg = slot_machine.play(T, 0.1)
    actions1, reward_avg1 = slot_machine.play(T, 0.1, baseline=False)
    actions2, reward_avg2 = slot_machine.play(T, 0.2)
    actions3, reward_avg3 = slot_machine.play(T, 0.2, baseline=False)
    plt.figure()
    plt.plot(tspan, reward_avg, label='alpha=0.1,baseline')
    plt.plot(tspan, reward_avg1, label='alpha=0.1,without baseline')
    plt.plot(tspan, reward_avg2, label='alpha=0.2,baseline')
    plt.plot(tspan, reward_avg3, label='alpha=0.2,without baseline')
    plt.xlabel('Steps')
    plt.ylabel("Average reward")
    plt.title("n Armed Bandit")
    plt.legend()
    print(actions2)
    plt.show()

结果如下:

可见此算法表现良好.
到此,老虎机算法告一段落,接下来是强化学习中的马尔科夫决策过程.

参考:
RL
强化学习读书笔记之Gradient Bandit及实现（二）
梯度上升
多臂老虎机问题
UCB算法