Bellman equation in RL

Bellman equation

Bellman expectation equation 是强化学习中非常基础而且重要的概念，但是有些细节却不好理解，尤其是关于${\mathbb{E}}_{\pi }$ (关于 policy $\pi$ 的期望）的部分。在参考了Understanding RL: The Bellman Equations 和 Derivation of Bellman’s Equation
这两篇文章中的推导内容之后，特地将 Bellman 公式的推理过程整理在这里。

Definition

Given a finite set of states ($S$) and actions ($A$), the state transition probability is $$p(s^{\prime }|s,a)=Pr(S_{t+1}=s^{\prime }|S_t=s,A_t=a)$$ and the reward is $$R(r|s,a,s^{\prime })=Pr(R_{t+1}=r|S_t=s,A_t=a,S_{t+1}=s^{\prime })$$
Notice the reward is actually a distribtion other than a determined value, this is root of some misunderstanding, because some articles only write the expectation.

Here we could put state transition and reward distribution as a single probability $$p(s^{\prime },r|s,a)=Pr(S_{t+1}=s^{\prime },R_{t+1}=r|S_t=s,A_t=a)$$

Given a policy is a mapping from $S$ to $A$ like $\pi (a|s)$.

The value function
V π ( s ) = E π { ∑ k = 1 inf ⁡ γ k R t + k + 1 ∣ S t = s } V_\pi(s) = \mathbb{E}_\pi\{\sum_{k=1}^{\inf}{\gamma^{k}R_{t+k+1}}\big| S_t =s\} Vπ​(s)=Eπ​{k=1∑inf​γkRt+k+1​∣∣​St​=s} and action value function as
q π ( s , a ) = E π { ∑ k = 1 inf ⁡ γ k R t + k + 1 ∣ S t = s , A t = a } q_\pi(s,a) = \mathbb{E}_\pi\{\sum_{k=1}^{\inf}{\gamma^{k}R_{t+k+1}}\big| S_t =s, A_t=a\} qπ​(s,a)=Eπ​{k=1∑inf​γkRt+k+1​∣∣​St​=s,At​=a}

According to the well known law of total expectation and the state transition diagram

$$q_{\pi }(s,a)=\sum _{s^{\prime },r}[r+\gamma V_{\pi }(s^{\prime })]p(s^{\prime },r|s,a)$$

$$\begin{gathered} V_{\pi }(s)=\sum _a\pi (a|s)\sum _{s^{\prime },r}[r+\gamma V_{\pi }(s^{\prime })]p(s^{\prime },r|s,a) \\ =\sum _a\pi (a|s)q_{\pi }(s,a) \end{gathered}$$