Lect5_Model_free_Control

Model Free Control

Optimise the value function of an unknown MDP

On-Policy Monte-Carlo Control

Generalised Policy Iteration

Monte-Carlo Policy Iteration

ONE

• Policy evaluation: Monte-Carlo poliy evaluation, $V=v_{\pi }$? or $Q=q_{\pi }$?

• Policy improvement: Greedy policy improvement?

  1. Greedy policy improvement over $V(s)$ requires model of MDP (model-based).
     $${\pi }^{\prime }(s)=\underset{a\in \mathcal{A}}{\mathrm{argmax}}\left({\mathcal{R}}_s^a+{\mathcal{P}}_{s{s}^{\prime }}^{a}V(s^{\prime })\right)$$

  2. Greedy policy improvement over $Q(s,a)$ is model-free.
     $${\pi }^{\prime }(s)=\underset{a\in \mathcal{A}}{\mathrm{argmax}}Q(s,a)$$

so: $\Downarrow$

TWO

• Policy evaluation: Monte-Carlo policy evaluation, $Q=q_{\pi }$

• Policy improvement: Greedy policy improvement?
  初始化时价值函数一般都是同一个值，若有一个动作比较好，greedy 就会倾向于一直选这个动作，其他动作的好坏并不知道。这就是 exploration 的问题。
  $ϵ$-Greedy Exploration: all m actions are tried with non-zero probability. With probability $1-ϵ$ choose the greedy action. With probability $ϵ$ choose an action at random.
  $$\pi (a\mid s)=\{\begin{array}{ll}ϵ/m+1-ϵ& \text{if}{a}^{\ast }=\underset{a\in \mathcal{A}}{\mathrm{argmax}}Q(s,a)\\ ϵ/m& \text{otherwise}\end{array}$$

  Theorem
  For any $ϵ$-greedy policy $\pi$, the $ϵ$-greedy policy ${\pi }^{\prime }$ with respect to $q_{\pi }$ is an improvement, $v_{{\pi }^{\prime }}(s)\ge v_{\pi }(s)$

  Proof:
  $$\begin{array}{rl}{q}_{\pi }(s,{\pi }^{\prime }(s))& =\sum _{a\in \mathcal{A}}{\pi }^{\prime }(a\mid s)q_{\pi }(s,a)\\ & =ϵ/m\sum _{a\in \mathcal{A}}{q}_{\pi }(s,a)+(1-ϵ)\underset{a\in \mathcal{A}}{\max }{q}_{\pi }(s,a)\\ & =ϵ/m\sum _{a\in \mathcal{A}}{q}_{\pi }(s,a)+(1-ϵ)\underset{a\in \mathcal{A}}{\max }{q}_{\pi }(s,a)\sum _{a\in \mathcal{A}}\frac{\pi (a\mid s)-\frac{ϵ}{m}}{1-ϵ}\\ & \ge ϵ/m\sum _{a\in \mathcal{A}}{q}_{\pi }(s,a)+(1-ϵ)\sum _{a\in \mathcal{A}}\frac{\pi (a\mid s)-\frac{ϵ}{m}}{1-ϵ}{q}_{\pi }(s,a)\\ & =\sum _{a\in \mathcal{A}}\pi (a\mid s)q_{\pi }(s,a)=v_{\pi }(s)\end{array}$$
  解释第三行和第四行。第三行根据 $ϵ$-greedy 的定义，可以算出第二个求和算式的值为1，因此是等于号。第四行将求和的部分当作 $q_{\pi }(s,a)$ 的权值，很容易想象，$q_{\pi }(s,a)$ 每一项加权求和一定小于等于其最大值。

  另外可能的误会：$\frac{\pi (a\mid s)-\frac{ϵ}{m}}{1-ϵ}$ 仅 $q_{\pi }(s,a)$ 为最大值时的action计算结果为：$\frac{\pi (a\mid s)-\frac{ϵ}{m}}{1-ϵ}=1$，其余action计算出来为：$\frac{\pi (a\mid s)-\frac{ϵ}{m}}{1-ϵ}=0$，因此第四行应该是等于号。如果这么认为的话，就相当于 $\pi$ and ${\pi }^{\prime }$ 没有区别了，要注意 $\underset{a\in \mathcal{A}}{\max }{q}_{\pi }(s,a)$ 中选中的 action 是 ${\pi }^{\prime }$ 根据 $q_{\pi }(s,a)$ 最大值选取的，这个action与 $\pi (a\mid s)=ϵ/m+1-ϵ$ 时的action 并不一定是同一个。 如果是同一个就是等于号了。

so: $\Downarrow$

THREE

• Policy evaluation: Monte-Carlo policy evaluation, $Q=q_{\pi }$
• Policy improvement: $ϵ$-Greedy policy improvement

Pseudocode

Monte-Carlo Control

Every episode

• Policy evaluation: Monte-Carlo policy evaluation, $Q\approx q_{\pi }$
• Policy improvement: $ϵ$-Greedy policy improvement

不用等很多个episode来估算Q，每一个episode完直接更新

GLIE Monte-Carlo Control

Definition of GLIE:

Greedy in the Limit with Infinite Exploration (GLIE)

• All state-action pairs are explored infinitely many times
  $$\underset{k\to \infty }{\lim }{N}_k(s,a)=\infty$$

• The policy converges on a greedy policy
  $$\underset{k\to \infty }{\lim }{\pi }_k(a\mid s)=1\left(a=\underset{a^{\prime }\in \mathcal{A}}{\mathrm{argmax}}{Q}_k(s,a^{\prime })\right)$$

Algorithm:

• Sample kth episode using π : { S 1 , A 1 , R 2 , … , S T } ∼ π \pi: \{S_1, A_1, R_2, \dots, S_T\} \sim \pi π:{S1​,A1​,R2​,…,ST​}∼π

• For each state $S_t$ and action $A_t$ in the episode
  $$\begin{array}{rl}N(S_t,A_t)& \leftarrow N(S_t,A_t)+1\\ Q(S_t,A_t)& \leftarrow Q(S_t,A_t)+\frac{1}{N(S_t,A_t)}(G_t-Q(S_t,A_t))\end{array}$$

• Improve policy based on new action-value function
  $$\begin{array}{rl}ϵ& \leftarrow 1/k\text{逐渐增大选择使Q最大的action的概率}\\ \pi & \leftarrow ϵ\text{-greedy}(Q)\end{array}$$

On-Policy Temporal-Difference Learning

Natural diet: use TD instead of MC in control loop

• Apply TD to $Q(S,A)$
• Use $ϵ$-greedy policy improvement
• Update every time-step

Update Action-Value Functions with Sarsa

On-Policy Control With Sarsa

Every time-step:

Policy evaluation Sarsa, $Q\approx q_{\pi }$

Policy improvement $ϵ$-greedy policy improvement

Sarsa Algorithm:

Sarsa($\lambda$)

n-step Sarsa

• Consider the following n-step returns for $n=1,2,\dots ,\infty$

$$\begin{array}{rl}n=1\text{(Sarsa)}{q}_t^{(1)}& =R_{t+1}+\gamma Q(S_{t+1})\\ n=2q_t^{(2)}& =R_{t+1}++\gamma R_{t+2}+{\gamma }^{2}Q(S_{t+1})\\ ⋮& \\ n=\infty \text{(MC)}{q}_t^{(\infty )}& =R_{t+1}++\gamma R_{t+2}+\cdots +{\gamma }^{T-1}{R}_T\end{array}$$

• Define the n-step Q-return
  $$q_t^{(n)}=R_{t+1}++\gamma R_{t+2}+\cdots +{\gamma }^{n-1}{R}_{t+n}+{\gamma }^{n}Q(S_{t+n})$$

• n-step Sarsa updates Q(s,a) towards the n-step Q-return

$$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha \left(q_t^{(n)}-Q(S_t,A_t)\right)$$

Forward View Sarsa($\lambda$)

• combines all n-step Q-returns $q_t^{(n)}$

• Using weight $(1-\lambda ){\lambda }^{n-1}$
  $$q_t^{\lambda }=(1-\lambda )\sum _{n=1}^{\infty }{\lambda }^{n-1}{q}_t^{(n)}$$

• Forward-view Sarsa($\lambda$)
  $$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha \left(q_t^{\lambda }-Q(S_t,A_t)\right)$$

Backward View Sarsa($\lambda$)

参考Lect4中关于TD($\lambda$)的backward-view的部分，这里不详细展开。

• Just like TD($\lambda$), we use eligibility trace in an online algorithm

• But Sarsa($\lambda$) has one eligibility trace for each state-action pair
  $$\begin{array}{rl}{E}_0(s,a)& =0\\ E_t(s,a)& =\gamma \lambda E_{t-1}(s,a)+1(S_t=s,A_t=a)\end{array}$$

• $Q(s,a)$ is updated for every state s and action a

• In proportion to TD-error ${\delta }_t$ and eligibility trace $E_t(s,a)$
  $$\begin{array}{rl}{\delta }_t& =R_{t+1}+\gamma Q(S_{t+1},A_{t+1})-Q(S_t,A_t)\\ Q(s,a)& \leftarrow Q(S,a)+\alpha {\delta }_t{E}_t(s,a)\end{array}$$

Sarsa($\lambda$) Algorithm

Off-Policy Learning

Importance Sampling

Estimate the expectation of a different distribution:
$$\begin{array}{rl}{\mathbb{E}}_{X\sim P}[f(X)]& =\sum P(X)f(X)\\ & =\sum Q(X)\frac{P(X)}{Q(X)}f(X)\\ & ={\mathbb{E}}_{X\sim Q}\left[\frac{P(X)}{Q(X)}f(X)\right]\end{array}$$

Important Sampling for Off-Policy Monte-Carlo

• Use returns generated from $\mu$ to evaluate $\pi$

• Weight return $G_t$ according to similarity between policies, multiply improtance sampling corrections along whole episode
  $$G_t^{\pi /\mu }=\frac{\pi (A_t\mid S_t)}{\mu (A_t\mid S_t)}\frac{\pi (A_{t+1}\mid S_{t+1})}{\mu (A_{t+1}\mid S_{t+1})}\dots \frac{\pi (A_T\mid S_T)}{\mu (A_T\mid S_T)}{G}_T$$

• Update value towards corrected return
  $$V(S_t)\leftarrow V(S_t)+\alpha \left(G_t^{\pi /\mu }-V(S_t)\right)$$

• Importance sampling can dramatically increase variance

Important Sampling for Off-Policy TD

• Use TD targets generated from $\mu$ to evaluate $\pi$

• Weight TD target $R+\gamma V(S^{\prime })$ by importance sampling, only need a single improtance sampling correction
  $$V(S_t)\leftarrow V(S_t)+\alpha \left(\frac{\pi (A_t\mid S_t)}{\mu (A_t\mid S_t)}\left(R_{t+1}+\gamma V(S_{t+1})\right)-V(S_t)\right)$$

• Much lower variance than Monte-Carlo improtance sampling

• Policies only need to be similar over a single step

Off-Policy Q-Learning

在状态 $S_t$ 时，根据 behavior policy 选取 action：$A_t\sim \mu (\cdot \mid S_t)$，得到相应的奖励 $R_{t+1}$，而后达到了状态 $S_{t+1}$，此时根据 estimate policy 选取action：$A_{t+1}\sim \pi (\cdot \mid S_{t+1})$。将这个action记为 $A^{\prime }$

更新：$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha \left(R_{t+1}+\gamma Q(S_{t+1},A^{\prime })-Q(S_t,A_t)\right)$

Special Case

• The target policy $\pi$ is greedy w.r.t. $Q(s,a)$
  $$\pi (S_{t+1})=\underset{a^{\prime }}{\mathrm{argmax}}Q(S_{t+1},a^{\prime })$$

• The behavior policy $\mu$ is $ϵ$-greedy w.r.t. $Q(s,a)$

The Q-learning target then simplifies:
$$\begin{array}{rl}{R}_{t+1}+\gamma Q(S_{t+1},A^{\prime })& =R_{t+1}+\gamma Q(S_{t+1},\underset{a^{\prime }}{\mathrm{argmax}}Q(S_{t+1},a^{\prime }))\\ & =R_{t+1}+\underset{a^{\prime }}{\max }\gamma Q(S_{t+1},a^{\prime })\end{array}$$
Algorithm:

Summary