本文介绍了强化学习中时间差分（TD）学习的概念，它结合了蒙特卡洛方法和动态规划的优点。TD预测方法能直接从经验中学习，而无需环境模型。TD(0)是最简单的TD方法，其更新目标不同于蒙特卡洛方法的最终回报，而是当前奖励加上折扣因子乘以后续状态的估计值。此外，TD方法因为部分基于现有估计进行更新，所以被称为引导方法。Sarsa是基于策略的TD控制方法，而Q-Learning则是离策略的TD控制方法，能直接逼近最优动作值函数。
第六章 时间差分算法
Chapter 6 Temporal-Difference Learning
- TD learning is a combination of Monte Carlo ideas and dynamic programming (DP) ideas.
- Like Monte Carlo methods, TD methods can learn directly from raw experience without a model of the environment’s dynamics. Like DP, TD methods update estimates based in part on other learned estimates, without waiting for a final outcome (they bootstrap).
6.1 TD Prediction
Constant-α MC (every visit) :
where Gt is the actual return following time t. Monte Carlo methods must wait until the end of the episode to determine the increment to V (St) (only then is Gt known), TD methods need wait only until the next time step.
The simplest TD method, known as TD(0):
最低0.47元/天 解锁文章
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
