Eligibility Traces
Eligibility Traces是强化学习中很基本很重要的一个概念。几乎所有的TD算法可以结合eligibility traces获得更一般化的算法，并且通常会更有效率。
Eligibility traces可以将TD和Monte Carlo算法统一起来。之前我们见过n-step方法将二者统一起来，eligibility traces 优于n-step的主要地方在于计算非常有效率，其只需要一个trace向量，而不需要储存n个特征向量。另外，eligibility trace的学习是逐步的，而不需要等到n-steps之后。
Eligibility traces机制是通过一个短期的向量得到的，这个向量称为eligibility trace \(\pmb{z_t} \in R^d\)。正好呼应算法中的权重向量\(\pmb{w_t}\in R^d\). 大致的思想是这样的：当权重向量的某些分量参与到估计价值当中，那么与之相对应的eligibility trace分量也会立即变化，然后变化会慢慢的消失。
强化学习的很多算法都是‘向前看’的算法，即当前状态或动作的价值更新是依赖于未来状态或动作的，这样的 算法称为‘前向视角’。前向视角实施起来多少有些不方便，毕竟未来的事，当前并知道。还有另外一种思路是所谓的‘后向视角’，通过eligibility trace来考虑最近刚刚访问过的状态或动作。
The \(\lambda\)-return
之前定义过n-steps return为最初n rewards加上n步之后的状态的价值估计的总和：
\[ G_{t:t+n} \dot = R_{t+1} + \gamma R_{t+2} + ... + \gamma^{n-1}R_{t+n} + \gamma^n \hat v(S_{t+n}, w_{t+n-1}) \]
其实一个有效的更新不一定只依赖于n－steps return，可能依赖于多个n-step return的平均会更好，比如：
\[ \frac{1}{2}G_{t:t+2} + \frac{1}{2}G_{t:t+4} \]
\(\lambda\)-return 可以理解为一种特殊的平均多个n-step return，它平均所有n-step return：
\[ G_{t}^{\lambda}\dot = (1- \lambda)\sum_{n=1}^{\infty}\gamma^{n-1}G_{t:t+n} = (1-\lambda)\sum_{n=1}^{T-t-1}\lambda^{n-1}G_{t:t+n} + \lambda^{T-t-1}G_{t} \]
基于\(\lambda\) return 的学习算法比如off-line \(\lambda\)－return algorithm:
\[ w_{t+1}\dot = w_t + \alpha [ G_t^{\lambda} - \hat v(S_t, w_t)]\nabla \hat v(S_t,w_t), t = 0,..., T-1. \]
以上其实就是一种前向视角：
TD(\(\lambda\))
TD(\(\lambda\))是上面off-line \(\lambda\)-return算法的改进，主要有三点：一是它每步更新，二是因为每步更新，它的计算过程分布在每一时间步上，三是它可以应用于连续过程而不仅限于episodic problems。
如上所述，eligibility trace 通过一个向量\(\pmb{z_t}\in R^d\) 实现，其与\(\pmb{w_t}\)维度相同，\(\pmb{w_t}\)可看作是长时记忆，而eligibility trace则是短时记忆。
\[ z_{-1}\dot = 0, \qquad z_t\dot = \gamma \lambda z_{t-1} +\nabla\hat v(S_t,w_t),\qquad 0 \le t\le T \]
其中\(\gamma\)是discount rate，\(\lambda\) 是trace decay参数。Eligibility trace记录权重向量各成分在近期状态价值评估中的贡献。
TD error:
\[ \delta_t \dot = R_{t+1} +\gamma\hat v(S_{t+1},w_t)- \hat v(S_t,w_t) \]
那么权重向量的更新：
\[ w_{t+1} \dot = w_t+\alpha \delta_t z_t \]
# semi-gradient TD(lambda) for estimating v_hat = v_pi
Input: the policy pi to be evaluated
Input: a differentiable function v_hat: S_plus x R^d --> R such that v_hat(terminal,.) = 0
Algorithm parameters: step size alpha > 0, trace decay rate lambda in [0,1]
Initialize value-function weights w arbitrarily (e.g. w = 0)
Loop for each episode:
Initialize S
z = 0
Loop for each step of episode:
Choose action A based on pi(.|S)
Take A, observe R,S'
z = gamma * lambda *z + grad V(S,w)
delta = R + gamma v_hat(S',w) - v_hat(S,w)
w = w + alpha delta z
S = S'
until S' is terminalTD(\(\lambda\))是时间上反向的，从当前状态，向后考虑已经经历过的状态。
易知，当\(\lambda = 0\) \(TD(\lambda)\)退化成为one step semi-gradient也就是TD(0)；当$ ＝ １\lambda ＝ １$时，TD(1)则表现出MC的特征。
n-step Truncated \(\lambda\)-return Methods
线下的\(\lambda-return\)算法具有很大的局限性，因为\(\lambda\)-return要在最后才知道，在连续状态情况下，甚至永远得不到。我们知道，一般\(\gamma\lambda<1\)，因此太长时间的状态价值的影响就会消失掉，那么可以想到一种近似\(\lambda\)-return的方法称为Truncated \(\lambda\)-return方法：
\[ G_{t:h}^{\lambda}\dot = (1-\lambda)\sum_{n =1}^{h-t-1} \lambda^{n-1}G_{t:t+n} + \lambda^{h-t-1}G_{t:h},\qquad 0 \le t<h\le T \]
同理可以推广出truncated TD(\(\lambda\))(TTD(\(\lambda\)))：
\[ w_{t+n}\dot = W_{t+n-1} + \alpha [G_{t:t+n}^\lambda - \hat v(S_t,w_{t+n-1})]\nabla \hat v(S_t,w_{t+n-1}) \]
此算法可以以一种比较有效率的方式来改写，使其不依赖于n（当然，背后的理论关系还是依赖n的）：
\[ G_{t:t+k}^{\lambda} = \hat v(S_t,w_{t-1}) + \sum_{i = t}^{t+k-1} (\gamma \lambda)^{i-t}\delta_i' \]
其中\(\delta_t' \dot = R_{t+1} + \gamma\hat v(S_{t+1},w_t) -\hat v(S_t,w_{t-1})\).
Redoing Updates: The On-line \(\lambda\)-return Algorithm
TTD(\(\lambda\))中n的选取是要有取舍的，n可以很大，这样更接近off-line \(\lambda\)-return algorithm；或者很小这样更新参数不用等很久。这两种好处增加一些计算复杂度可以都得到。
其思想是这样的：在每一步时，都会得到新的数据，然后从头重新更新。
\[ w_{t+1}^h \dot = w_t^h + \alpha [G_{t:h}^h - \hat v(S_t,w_t^h)]\nabla \hat v(S_t,w_t^h),\qquad 0\le t\le h\le T. \]
\(w_t \dot = w_t^t\)此即为on-line \(\lambda\)-return algorithm.
True On-line TD(\(\lambda\))
借助eligibility trace，将on-line \(\lambda\) -return 的前向视角转变成后向视角即成为true on-line TD(\(\lambda\)).
\[ w_{t+1}\dot = w_t + \alpha \delta_t z_t + \alpha(w_t^Tx_t - w_{t-1}^Tx_t)(z_t - x_t) \]
其中$\hat v(s,w) = w^T x(s), x_t\dot = x(S_t), \delta_t \dot = R_{t+1} +\gamma\hat v(S_{t+1},w_t)- \hat v(S_t,w_t), z_t = \dot = \gamma\lambda z_{t-1}+(1-\alpha \gamma\lambda z_{t-1}^Tx_t)x_t $.
# True On-line TD(lambda) for estimating w^T x = v_pi
Input: the policy pi to be evaluated
Input: a feature function x: S_plus --> R such that x(terminal,.) = 0
Algorithm parameters: step size alpha > 0, trace decay rate lambda in [0,1]
Initialize value-function weights w in R
Loop for each episode:
Initialize state and obtain initial feature vector x
z = 0
V_old = 0
Loop for each step of episode:
choose A from pi.
take action A, observe R, x'(feature vector of the next state)
V = w^T x
V' = w^T x'
delta = R + gamma V' - V
z = gamma lambda z + (1- alpha gamma z^T x)x
w = w + alpha(delta + V- V_old)z - alpha (V-V_old) x
V_old = V'
x = x'
until x' = 0(signaling arrival at a terminal state)这里的eligibility trace 称为 dutch trace, 还有一种称为replacing trace:
\[ z_{i,t} \dot =\left\{\begin{array}\\ 1\qquad if\quad x_{i,t} = 1,\\ \gamma\lambda z_{i,t-1} \qquad otherwise.\end{array}\right. \]
Dutch Traces in Monte Carlo Learning
Eligibility traces也可应用于Monte Carlo 相产算法之中。
在梯度Monte Carlo算法的线性版本中：
\[ w_{t+1} \dot = w_t +\alpha [G - w_t^Tx_t]x_t,\qquad 0 \le t \le T. \]
其中Ｇ是在episode结束后得到的奖励，因此没有下标，也没有折扣因子。因为MC方法只能在episode结束后更新参数，这个特点我们不试图改进，而是在尽力将计算分布在各个时间步上。
\[ \begin{array}\\ w_T& =& w_{T-1} + \alpha(G - w_{T-1}^Tx_{T-1})x_{T-1}\\ &=& w_{T-1} +\alpha x_{T-1}(-x_{T-1}^Tw_{T-1})+\alpha Gx_{T-1}\\ &=& (I - \alpha x_{T-1}x_{T-1}^T)w_{T-1}+\alpha Gx_{T-1}\\ &=& F_{T-1}w_{T-1}+\alpha Gx_{T-1} \end{array} \]
其中\(F_t \dot = I - \alpha x_t x_t^T\)是一个遗忘矩阵或称衰退矩阵（forgetting or fading matrix), 可以递归得到。
Ｓarsa(\(\lambda\))
Sarsa(\(\lambda\))是一个TD方法，与TD(\(\lambda\))类似，只不过，Sarsa(\(\lambda\))是针对state-action pair的：
\[ w_{t+1}\dot = w_t +\alpha \delta_t z_t \]
其中：
\[ \delta \dot = R_{t+1}+\gamma \hat q(S_{t+1},A_{t+1},w_t) - \hat q(S_t,A_t,w_t) \]
而：
\[ \begin{array}\\ z_{-1} &=& 0,\\ z_t&=& \gamma \lambda z_{t-1}+\nabla\hat q(S_t,A_t,w_t),\qquad 0\le t\le T. \end{array} \]
# Sarsa(lambda) with binary features and linear function approximation for estimating # w^T x q_{pi} or q*
Input: a function F(s,a) returning teh set of (indices of ) active features for s,a
Input: a policy pi (if estimating q_pi)
Algorithm parameteres: step size alpha > 0, trace decay rate lambda in [0,1]
Initialize: w= (w1,...,wd)^T in R^d z = (z1,...,z_d)^T in R^d
Loop for each episode:
Initialize S
Choose A from pi(.|S) or e-greedy according to q_hat(S,.,w)
z = 0
Loop for each step of episode:
take action A, observe R,S'
delta = R
Loop for i in F(S,A):
delta = delta - w_i
z_i = z_i + 1 (accumulating traces)
or z_i = 1 (replacing traces)
If S' is terminal then:
w = w + alpha delta z
Go to next episode
Choose A' in pi(.|S) or near greedily q_hat(S',.,w)
Loop for i in F(S',A'): dleta = delta + gamma w_i
w = w + alpha delta z
z = gamma lambda z
S = S', A = A' # True Online Sarsa(lambda) for estimating w^Tx = q_pi or q*
Input: a feature function x: s_plus x A = R^d such that x(terminal,.) = 0
Input: a policy pi (if estimating q_pi)
Algorithm parameters: step size alpha >0, trace decay rate lambda in [0,1]
Initialize: w in R^d
Loop for each episode:
Initialize S
Choose A from pi(.|S) or nearly greedily from S using w
x = x(S,A)
z = 0
Q_old = 0
Loop for each step of episode:
Take action A, observe R,S'
Choose A' from pi(.|S') or near greedily from S' using w
x' = x(S',A')
Q = w^Tx
Q' = w^Tx'
delta = R + gamma Q' - Q
z = gamma lambda z + (1 - alpha gamma lambda z^T x) x
w = w + alpha(delta + Q - Q_old )z - alpha(Q - Q_old) x
Q_old = Q'
x = x'
A = A'
until S' is terminal 
8198
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
