强化学习Sarsa

Sarsa

算法如下：

相比Q-learning而言，Sarsa会比较胆小，不希望自己掉进陷阱里，从而做出的决策都比较谨慎；而Q-learning会选择一条虽然比较危险但是可以成功的道路

# off-policy
class QLearningTable(RL):
    def __init__(self, actions, learning_rate=0.01, reward_decay=0.9, e_greedy=0.9):
        super(QLearningTable, self).__init__(actions, learning_rate, reward_decay, e_greedy)

    def learn(self, s, a, r, s_):
        self.check_state_exist(s_)
        q_predict = self.q_table.loc[s, a]
        if s_ != 'terminal':
            q_target = r + self.gamma * self.q_table.loc[s_, :].max()  # next state is not terminal
        else:
            q_target = r  # next state is terminal
        self.q_table.loc[s, a] += self.lr * (q_target - q_predict)  # update


# on-policy
class SarsaTable(RL):

    def __init__(self, actions, learning_rate=0.01, reward_decay=0.9, e_greedy=0.9):
        super(SarsaTable, self).__init__(actions, learning_rate, reward_decay, e_greedy)

    def learn(self, s, a, r, s_, a_):
        self.check_state_exist(s_)
        q_predict = self.q_table.loc[s, a]
        if s_ != 'terminal':
            q_target = r + self.gamma * self.q_table.loc[s_, a_]  # next state is not terminal
        else:
            q_target = r  # next state is terminal
        self.q_table.loc[s, a] += self.lr * (q_target - q_predict)  # update

对比而言，其中比较重要的有两行

# Q-learning
q_target = r + self.gamma * self.q_table.loc[s_, :].max()  # next state is not terminal
# Sarsa
q_target = r + self.gamma * self.q_table.loc[s_, a_]  # next state is not terminal

Q-learning只是选择了针对下一个状态的最好的action纳入考虑范围，然后做差值。而Sarsa提前选择好了下一个环境所对应的action来进行差值

---

Sarsa($\lambda$)

其中的$\lambda$是指距离最终目的地的步伐的重视程度，如果其值为0那么就认为每一次更新都是相对来说和目的地没有关联的，而其值为1的时候则认为每一次更新都是在考虑了目的地的更新
代码方面，则新添加了一个矩阵作为更新的“不可或缺性参考”

# 其内容为q_table的复制
self.eligibility_trace = self.q_table.copy()
 # Method 1:若选择了这个行为，则相对应的矩阵内容进行+1操作，这样容易导致后面数值过大
self.eligibility_trace.loc[s, a] += 1
# Method 2:若选择了这个行为，则将对应的行内容进行归零并将选择的动作列进行赋值为1操作
self.eligibility_trace.loc[s, :] *= 0
self.eligibility_trace.loc[s, a] = 1

# Q update 更新的时候也要将误差放大相应的倍数
self.q_table += self.lr * error * self.eligibility_trace

 # decay eligibility trace after update
 self.eligibility_trace *= self.gamma*self.lambda_