Sarsa算法:
Sarsa算法与Q-learing算法的不同之处是什么?
一个简单的解释,引用莫凡大神的话:
- 他在当前
state
已经想好了state
对应的action
, 而且想好了 下一个state_
和下一个action_
(Qlearning 还没有想好下一个action_
) - 更新
Q(s,a)
的时候基于的是下一个Q(s_, a_)
(Qlearning 是基于maxQ(s_)
)
对于第二句话,可以从走迷宫的代码中只管体现出来:(代码来自于莫凡大神编写地址:https://github.com/MorvanZhou/Reinforcement-learning-with-tensorflow/blob/master/contents/3_Sarsa_maze/RL_brain.py)
# off-policy
class QLearningTable(RL):
def __init__(self, actions, learning_rate=0.01, reward_decay=0.9, e_greedy=0.9):
super(QLearningTable, self).__init__(actions, learning_rate, reward_decay, e_greedy)
def learn(self, s, a, r, s_):
self.check_state_exist(s_)
q_predict = self.q_table.loc[s, a]
if s_ != 'terminal':
q_target = r + self.gamma * self.q_table.loc[s_, :].max() # next state is not terminal
else:
q_target = r # next state is terminal
self.q_table.loc[s, a] += self.lr * (q_target - q_predict) # update
# on-policy
class SarsaTable(RL):
def __init__(self, actions, learning_rate=0.01, reward_decay=0.9, e_greedy=0.9):
super(SarsaTable, self).__init__(actions, learning_rate, reward_decay, e_greedy)
def learn(self, s, a, r, s_, a_):
self.check_state_exist(s_)
q_predict = self.q_table.loc[s, a]
if s_ != 'terminal':
q_target = r + self.gamma * self.q_table.loc[s_, a_] # next state is not terminal
else:
q_target = r # next state is terminal
self.q_table.loc[s, a] += self.lr * (q_target - q_predict) # update
可以看出二者的q_target不同,Q-learing取得是最大值,但是实际不一定会选,而Sarsa则是直接取到下一个a_,也就是下一个状态的动作,这个动作是下一次一定要做的。