今天继续写RL的exercise2,发现Q learning一直不收敛。本来就是个很简单的算法,改了好久都不知道fault在哪里,一开始以为是超参数调的不好,结果调了好久的参数都不行。后来发现自己犯了个错误:
target = reward + int(done) * self.gamma * max_action_q
new_estimate = old_estimate + self.alpha * (target - old_estimate)
注意这里,不应该是int(done),int(not done),很容易忽略的一个逻辑错误(其实还是自己对算法的理解不够深)
改过来之后,就收敛了,下面是训练日志:
EVALUATION: EP 1000 - MEAN RETURN -83.042 (424/500 failed episodes)
EVALUATION: EP 2000 - MEAN RETURN -13.322 (101/500 failed episodes)
EVALUATION: EP 3000 - MEAN RETURN 5.032 (13/500 failed episodes)
EVALUATION: EP 4000 - MEAN RETURN 7.898 (0/500 failed episodes)
EVALUATION: EP 5000 - MEAN RETURN 8.166 (0/500 failed episodes)
EVALUATION: EP 6000 - MEAN RETURN 7.848 (0/500 failed episodes)
EVALUATION: EP 7000 - MEAN RETURN 7.8 (0/500 failed episodes)
EVALUATION: EP 8000 - MEAN RETURN 7.88 (0/500 failed episodes)
EVALUATION: EP 9000 - MEAN RETURN 7.826 (0/500 failed episodes)
EVALUATION: EP 10000 - MEAN RETURN 7.888 (0/500 failed episodes)