Lect6_Value_Function_Approximation

Value Funtion Approximation

Introduction

Why need?

• we have represented value function by a lookup table
  • Every state s has an entry $V(s)$
  • Or every state-action pair s, a has an entry $Q(s,a)$
• Problem with large MDPs:
  • There ate too many states and/or actions to store in memory
  • It’s too slow to learn the value of each state individually

Solution for large MDPs:

• Estimate value function with function approximation
  $$\begin{array}{rl}\hat{v}(s,\mathbf{w})& \approx v_{\pi }(s)\\ or\hat{q}(s,a,\mathbf{w})& \approx q_{\pi }(s,a)\end{array}$$

Types of Value Function Approximation

Which Funtion Approximator?

There are many funtion approximators, but we consider differentiable fucntion approximators, e.g.

• Linear combinations of features
• Neural network
• Decision tree
• Nearest neighbor
• Fourier / wavelet bases
• $\dots$

Incremental Methods

Value Funtion Approx. by SGD

Goal: find parameter vector $\mathbf{w}$ Minimising mean-squared error between approximate value function $\hat{v}(s,\mathbf{w})$ and true value function $v_{\pi }(s)$ :
$$J\text{J}J(\mathbf{w})={\mathbb{E}}_{\pi }\left[(v_{\pi }(S)-\hat{v}(S,\mathbf{w}){)}^2\right]\text{\hspace{1em}(1)}$$
Gradient descent finds a local minimum:
$$\Delta \mathbf{w}=-\frac{1}{2}\alpha {\nabla }_{\mathbf{w}}J\text{J}J(\mathbf{w})=\alpha {\mathbb{E}}_{\pi }\left[(v_{\pi }(S)-\hat{v}(S,\mathbf{w})){\nabla }_{\mathbf{w}}\hat{v}(S,\mathbf{w})\right]$$
Expected update is equal to full gradient update👆

SGD samples the gradient:
$$\Delta \mathbf{w}=\alpha (v_{\pi }(S)-\hat{v}(S,\mathbf{w})){\nabla }_{\mathbf{w}}\hat{v}(S,\mathbf{w})\text{\hspace{1em}(2)}$$

Linear Funtion Approximation

Feature vectors: represent state by a feature vector
$$\mathbf{x}(S)=\left(\begin{array}{c}{\mathbf{x}}_1(S)\\ ⋮\\ {\mathbf{x}}_n(S)\end{array}\right)$$
Linear ---- Represent value function by a linear combination of features:
$$\hat{v}(S,\mathbf{w})=\mathbf{x}(S{)}^{\text{T}}\mathbf{w}=\sum _{j=1}^n{\mathbf{x}}_j(S){\mathbf{w}}_{\mathbf{j}}=\mathbf{x}(S)⨀\mathbf{w}\text{点积}\text{\hspace{1em}(3)}$$
easy to get: KaTeX parse error: \tag works only in display equations

代入式（2）：
$$\begin{gathered} \Delta \mathbf{w}=\alpha (v_{\pi }(S)-\hat{v}(S,\mathbf{w}))\mathbf{x}(S) \\ \text{Update = learning rate * prediction error * feature value} \end{gathered}$$

---

可以发现 表格型方法 是 线性价值函数估计 的一种特例，其 feature vector 为：
$${\mathbf{x}}^{table}(S)=\left(\begin{array}{c}1(\mathbf{S}=s_1)\\ ⋮\\ 1(\mathbf{S}=s_n)\end{array}\right)$$

$\mathbf{w}$ 中每一个元素代表一个状态的值：
$$\hat{v}(S,\mathbf{w})=\left(\begin{array}{c}1(\mathbf{S}=s_1)\\ ⋮\\ 1(\mathbf{S}=s_n)\end{array}\right)\cdot \left(\begin{array}{c}{\mathbf{w}}_1\\ ⋮\\ {\mathbf{w}}_n\end{array}\right)$$

Incremental Prediction Algorithms

In RL there is no supervisor, i.e. no true value funtion $v_{\pi }(s)$. In practice, we substitute a target for $v_{\pi }(s)$

• For MC, the target is the return $G_t$
  $$\Delta (\mathbf{w})=\alpha (G_t-\hat{v}(S_t,\mathbf{w})){\nabla }_{\mathbf{w}}\hat{v}(S_t,\mathbf{w})$$

• For TD(0), the target is the TD target $R_{t+1}+\gamma \hat{v}(S_{t+1},\mathbf{w})$
  $$\Delta (\mathbf{w})=\alpha (R_{t+1}=\gamma \hat{v}(S_{t+1},\mathbf{w})-\hat{v}(S_t,\mathbf{w})){\nabla }_{\mathbf{w}}\hat{v}(S_t,\mathbf{w})$$

• For TD($\lambda$), the target is the $\lambda$-return $G_t^{\lambda }$
  $$\Delta (\mathbf{w})=\alpha (G_t^{\lambda }-\hat{v}(S_t,\mathbf{w})){\nabla }_{\mathbf{w}}\hat{v}(S_t,\mathbf{w})\text{\hspace{1em}(4)}$$

  • Forward view linear TD($\lambda$)
    $$\begin{array}{rl}\Delta (\mathbf{w})& =\alpha (G_t^{\lambda }-\hat{v}(S_t,\mathbf{w})){\nabla }_{\mathbf{w}}\hat{v}(S_t,\mathbf{w})\\ & =\alpha (G_t^{\lambda }-\hat{v}(S_t,\mathbf{w}))\mathbf{x}(S_t)\end{array}\text{\hspace{1em}(4.1)}$$

  • Backward view linear TD($\lambda$)
    $$\begin{array}{rl}{\delta }_t& =R_{t+1}+\gamma \hat{v}(S_{t+1},\mathbf{w})-\hat{v}(S_t,\mathbf{w})\\ E_t& =\gamma \lambda E_{t-1}+\mathbf{x}(S_t)\\ \Delta \mathbf{w}& =\alpha {\delta }_t{E}_t\end{array}\text{\hspace{1em}(4.2)}$$

Control with Value Function Approximation

Action-Value Function Approximation

like value function Approximation
$$\hat{q}(S,A,\mathbf{w})\approx q_{\pi }(S,A)$$

$$J\text{J}J(\mathbf{w})={\mathbb{E}}_{\pi }\left[(q_{\pi }(S,A)-\hat{q}(S,A,\mathbf{w}){)}^2\right]$$

$$\Delta \mathbf{w}=-\frac{1}{2}\alpha {\nabla }_{\mathbf{w}}J\text{J}J(\mathbf{w})=\alpha {\mathbb{E}}_{\pi }\left[(q_{\pi }(S,A)-\hat{q}(S,A,\mathbf{w})){\nabla }_{\mathbf{w}}\hat{q}(S,A,\mathbf{w})\right]$$

Linear Action-Value Function Approximation

like [linear function approximation](#linear function approximation)
$$\mathbf{x}(S,A)=\left(\begin{array}{c}{\mathbf{x}}_1(S,A)\\ ⋮\\ {\mathbf{x}}_n(S,A)\end{array}\right)$$

$$\hat{q}(S,A,\mathbf{w})=\mathbf{x}(S,A{)}^{\text{T}}\mathbf{w}=\sum _{j=1}^n{\mathbf{x}}_j(S,A){\mathbf{w}}_{\mathbf{j}}=\mathbf{x}(S,A)⨀\mathbf{w}\text{点积}\text{\hspace{1em}(3)}$$

$$\Delta \mathbf{w}=\alpha (q_{\pi }(S,A)-\hat{q}(S,A,\mathbf{w}))\mathbf{x}(S,A)$$

Incremental Prediction Algorithms

Substitute a target for $q_{\pi }(S,A)$

• For MC, the target is the return $G_t$
  $$\Delta (\mathbf{w})=\alpha (G_t-\hat{q}(S_t,A_t,\mathbf{w})){\nabla }_{\mathbf{w}}\hat{q}(S_t,A_t,\mathbf{w})$$

• For TD(0), the target is the TD target $R_{t+1}+\gamma \hat{q}(S_{t+1},A_{t+1},\mathbf{w})$
  $$\Delta (\mathbf{w})=\alpha (R_{t+1}+\gamma \hat{q}(S_{t+1},A_{t+1},\mathbf{w})-\hat{q}(S_t,A_t,\mathbf{w})){\nabla }_{\mathbf{w}}\hat{q}(S_t,A_t,\mathbf{w})$$

• For TD($\lambda$), the target is the $\lambda$-return $q_t^{\lambda }$
  $$\Delta (\mathbf{w})=\alpha (q_t^{\lambda }-\hat{q}(S_t,A_t,\mathbf{w})){\nabla }_{\mathbf{w}}\hat{q}(S_t,A_t,\mathbf{w})\text{\hspace{1em}(4)}$$

  • Backward view TD($\lambda$)
    $$\begin{array}{rl}{\delta }_t& =R_{t+1}+\gamma \hat{q}(S_{t+1},A_{t+1},\mathbf{w})-\hat{q}(S_t,A_t,\mathbf{w})\\ E_t& =\gamma \lambda E_{t-1}+{\nabla }_{\mathbf{w}}\hat{q}(S_t,A_t,\mathbf{w})\\ \Delta \mathbf{w}& =\alpha {\delta }_t{E}_t\end{array}$$

Covergence of Prediction Algorithms

对于TD为什么不收敛，而 gradient-TD可以收敛的原因并不是很清楚。但原ppt是这么解释的：
TD does not follow the gradient of any objective function. This is why TD can diverge when off-policy or using non-linear function approximation. Gradient TD follows true gradient of projected Bellman error.

Covergence of Control Algorithms

Batch Methods

说白了就是之前 control 算法都是拿一个数据就更新一次，样本利用效率低，更新过程数据相关性强。

于是现在变成把一定量的数据存放起来，然后每次从其中随机 sample a btach of datas，用SGD进行梯度更新。目标函数还是真实值和估计值之间误差的平方

具体的例子可以看之前的两篇文章DQN和DDQN