RL(Chapter 7): n-step Bootstrapping (n步自举法)

本文为强化学习笔记，主要参考以下内容：

• Reinforcement Learning: An Introduction
• 代码全部来自 GitHub
• 习题答案参考 Github

$n$-step Bootstrapping

• In this chapter we present $n$-step TD methods that generalize the MC methods and the one-step TD methods so that one can shift from one to the other smoothly as needed to meet the demands of a particular task. $n$-step methods span a spectrum with MC methods at one end and one-step TD methods at the other. The best methods are often intermediate between the two extremes.
• Another way of looking at the benefits of $n$-step methods is that they free you from the tyranny of the time step(解决更新时刻的不灵活问题). With one-step TD methods the same time step determines how often the action can be changed and the time interval over which bootstrapping is done. In many applications one wants to be able to update the action very fast to take into account anything that has changed, but bootstrapping works best if it is over a length of time in which a significant and recognizable state change has occurred. With one-step TD methods, these time intervals are the same, and so a compromise must be made. $n$-step methods enable bootstrapping to occur over multiple steps, freeing us from the tyranny of the single time step.

$n$-step TD Prediction

Consider the update of the estimated value of state $S_t$ as a result of the state–reward sequence, $S_t,R_{t+1},S_{t+1},R_{t+2},...,R_T,S_T$ (omitting the actions).

• Whereas in Monte Carlo updates the $target$ is the $return$:
  
• In one-step updates the $target$ is the $one$-$step$ $return$:
  

The subscripts on $G_{t:t+1}$ indicate that it is a truncated return for time $t$ using rewards up until time $t+1$, with the discounted estimate $\gamma V_t(S_{t+1})$ taking the place of the other terms $\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdot \cdot \cdot +{\gamma }^{T-t-1}{R}_T$ of the full return $G_t$.

• The target for a two-step update is the $two$-$step$ $return$:
  
• Similarly, the target for an arbitrary $n$-step update is the $n$-$step$ $return$:
  

All $n$-step returns can be considered approximations to the full return, truncated after $n$ steps and then corrected for the remaining missing terms by $V_{t+n-1}(S_{t+n})$. If $t+n>T$ (if the $n$-step return extends to or beyond termination), then the $n$-step return defined to be equal to the ordinary full return ($G_{t:t+n}=G_{t}ift+n>T$).

• $n$-$step$ TD: the natural state-value learning algorithm for using $n$-step returns is thus
  

Note that no changes at all are made during the first $n-1$ steps of each episode. To make up for that, an equal number of additional updates are made at the end of the episode, after termination and before starting the next episode.

Exercise 7.1
In Chapter 6 we noted that the Monte Carlo error can be written as the sum of TD errors (6.6) if the value estimates don’t change from step to step. Show that the $n$-step error used in (7.2) can also be written as a sum TD errors (again if the value estimates don’t change) generalizing the earlier result.
ANSWER

---

$error$ $reduction$ $property$ of $n$-step returns

• The $n$-step return uses the value function $V_{t+n-1}$ to correct for the missing rewards beyond $R_{t+n}$.
• An important property of $n$-step returns is that their expectation is guaranteed to be a better estimate of $v_{\pi }$ than $V_{t+n-1}$ is, in a worst-state sense. That is, the worst error of the expected $n$-step return is guaranteed to be less than or equal to ${\gamma }^n$ times the worst error under $V_{t+n-1}$:

• Because of the error reduction property, one can show formally that all $n$-step TD methods converge to the correct predictions under appropriate technical conditions.
• The $n$-step TD methods thus form a family of sound methods, with one-step TD methods and Monte Carlo methods as extreme members.

$n$-step Sarsa

The $n$-step version of Sarsa we call $n$-step Sarsa, and the original version presented in the previous chapter we henceforth call $one$-$step$ $Sarsa$, or $Sarsa(0)$.

• The main idea is to simply switch states for actions (state–action pairs) and then use an $\epsilon$-greedy policy. We redefine $n$-step returns (update targets) in terms of estimated action values:

with $G_{t:t+n}=G_{t}ift+n\ge T$.

• The $n$-step Sarsa is then

Exercise 7.4
Prove that the $n$-step return of Sarsa (7.4) can be written exactly in terms of a novel TD error, as

ANSWER

---

$n$-step Expected Sarsa

The backup diagram for the $n$-step version of Expected Sarsa is shown on the far right in Figure 7.3. It consists of a linear string of sample actions and states, just as in $n$-step Sarsa, except that its last element is a branch over all action possibilities weighted by their probability under $\pi$. This algorithm can be described by the same equation as $n$-step Sarsa (above) except with the $n$-step return redefined as

(with $G_{t:t+n}=G_{t}ift+n\ge T$) where

$n$-step Off-policy Learning

In $n$-step methods, returns are constructed over $n$ steps, so we are interested in the relative probability of just those $n$ actions. For example, to make a simple off-policy version of $n$-step TD, the update for time $t$ (actually made at time $t+n$) can simply be weighted by ${\rho }_{t:t+n-1}$:

where ${\rho }_{t:t+n-1}$, called the importance sampling ratio, is the relative probability under the two policies of taking the $n$ actions from $A_t$ to $A_{t+n-1}$

Similarly, our previous $n$-step Sarsa update can be completely replaced by a simple off-policy form:

Note that the importance sampling ratio here starts and ends one step later than for $n$-step TD (7.9). This is because here we are updating a state–action pair. We do not have to care how likely we were to select the action; now that we have selected it we want to learn fully from what happens, with importance sampling only for subsequent actions.

---

The off-policy version of $n$-step Expected Sarsa would use the same update as above for $n$-step Sarsa except that the importance sampling ratio would have one less factor in it. That is, the above equation would use ${\rho }_{t+1:t+n-1}$ instead of ${\rho }_{t+1:t+n}$, and of course it would use the Expected Sarsa version of the $n$-step return (7.7). This is because in Expected Sarsa all possible actions are taken into account in the last state; the one actually taken has no effect and does not have to be corrected for.

Off-policy Learning Without Importance Sampling: The $n$-step Tree Backup Algorithm

$n$ 步树回溯算法

The idea of the algorithm is suggested by the 3-step tree-backup backup diagram:

• Down the central spine and labeled in the diagram are three sample states and rewards, and two sample actions. These are the random variables representing the events occurring after the initial state–action pair $S_t,A_t$. Hanging off to the sides of each state are the actions that were not selected. (For the last state, all the actions are considered to have not (yet) been selected.)

• So far we have always updated the estimated value of the node at the top of the diagram toward a target combining the rewards along the way (appropriately discounted) and the estimated values of the nodes at the bottom.

• In the tree-backup update, the target includes all these things plus the estimated values of the dangling action nodes hanging off the sides, at all levels.

  • This is why it is called a $tree$-$backup$ update; it is an update from the entire tree of estimated action values.

Because we have no sample data for the unselected actions, we bootstrap and use the estimates of their values in forming the target for the update. This slightly extends the idea of a backup diagram.

• More precisely, the update is from the estimated action values of the $leaf$ $nodes$ of the tree. The action nodes in the interior, corresponding to the actual actions taken, do not participate. Each leaf node contributes to the target with a weight proportional to its probability of occurring under the target policy $\pi$.
  • Thus each first-level action $a$ contributes with a weight of $\pi (a|S_{t+1})$, except that the action actually taken, $A_{t+1}$, does not contribute at all. Its probability, $\pi (A_{t+1}|S_{t+1})$, is used to weight all the second-level action values.
  • Thus, each non-selected second-level action $a^{\prime }$ contributes with weight $\pi (A_{t+1}|S_{t+1})\pi (a^{\prime }|S_{t+2})$.
  • Each third-level action contributes with weight $\pi (A_{t+1}|S_{t+1})\pi (A_{t+2}|S_{t+2})\pi (a^{\prime \prime }|S_{t+3})$, and so on.
• It is as if each arrow to an action node in the diagram is weighted by the action’s probability of being selected under the target policy and, if there is a tree below the action, then that weight applies to all the leaf nodes in the tree.

---

We can think of the 3-step tree-backup update as consisting of 6 half-steps, alternating between sample half-steps from an action to a subsequent state, and expected half-steps considering from that state all possible actions with their probabilities of occurring under the policy.

Now let us develop the detailed equations for the $n$-step tree-backup algorithm. The one-step return (target) is the same as that of Expected Sarsa,

for $t<T-1$, and the two-step tree-backup return is

for $t<T-2$. The latter form suggests the general recursive definition of the tree-backup $n$-step return:

for $t<T-1,n\ge 2$, with the $n=1$ case handled by (7.15) except for $G_{T-1:t+n}=R_T$. This target is then used with the usual action-value update rule from $n$-step Sarsa:

for $0\le t<T$.

Exercise 7.11
Show that if the approximate action values are unchanging, then the tree-backup return (7.16) can be written as a sum of expectation-based TD errors:

where ${\delta }_t=R_{t+1}+\gamma \overline{V_t}(S_{t+1})-Q(S_t,A_t)$ and ${\overline{V}}_t$ is given by (7.8).

A Unifying Algorithm: $n$-step $Q(\sigma )$

So far in this chapter we have considered three different kinds of action-value algorithms, corresponding to the first three backup diagrams shown in Figure 7.5.

• $n$-step Sarsa has all sample transitions
• the tree-backup algorithm has all state-to-action transitions fully branched without sampling
• $n$-step Expected Sarsa has all sample transitions except for the last state-to-action one, which is fully branched with an expected value

---

To what extent can these algorithms be unified?

• One idea for unification is suggested by the fourth backup diagram in Figure 7.5. This is the idea that one might decide on a step-by-step basis whether one wanted to take the action as a sample, as in Sarsa, or consider the expectation over all actions instead, as in the tree-backup update.
• Then, if one chose always to sample, one would obtain Sarsa, whereas if one chose never to sample, one would get the tree-backup algorithm. Expected Sarsa would be the case where one chose to sample for all steps except for the last one.

---

To increase the possibilities even further we can consider a continuous variation between sampling and expectation.

• Let ${\sigma }_t\in [0,1]$ denote the degree of sampling on step $t$, with $\sigma =1$ denoting full sampling and $\sigma =0$ denoting a pure expectation with no sampling.
• The random variable ${\sigma }_t$ might be set as a function of the state, action, or state–action pair at time $t$. We call this proposed new algorithm $n$-step $Q(\sigma )$.

---

Now let us develop the equations of $n$-step $Q(\sigma )$.

• First we write the tree-backup $n$-step return (7.16) in terms of the horizon $h=t+n$ and then in terms of the expected approximate value $\overline{V}$ (7.8):

after which it is exactly like the $n$-step return for Sarsa with control variates (7.14) except with the action probability $\pi (A_{t+1}|S_{t+1})$ substituted for the importance-sampling ratio ${\rho }_{t+1}$. For $Q(\sigma )$, we slide linearly between these two cases:

for $t<h\le T$. The recursion ends with $G_{h:h}=Q_{h-1}(S_h,A_h)$ if $h<T$, or with
$G_{T-1}:T=R_T$ if $h=T$.

Then we use the earlier update for $n$-step Sarsa without importance-sampling ratios (7.5) instead of (7.11), because now the ratios are incorporated in the $n$-step return.

In Chapter 12, we will see how multi-step TD methods can be implemented with minimal memory and computational complexity using eligibility traces.