【强化学习】Bellman Equation Derivation （贝尔曼方程的推导）

Bellman Equation Derivation:

• Return($G_t$) 说的是把奖励进行折扣后所获得的收益。
• State value function($V_t(s)$)是MRP(Markov Reward Process,$<s,r,s^{\prime }>$)的return的期望。可以定义为如下形式：
  $$\begin{gathered} V(s)=E[G_t|s_t=s] \\ =E[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+...+{\gamma }^{T-t-1}{R}_T|s_t=s] \\ =E[R_{t+1}|s_t=s]+E[\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+...+{\gamma }^{T-t-1}{R}_T|s_t=s] \\ =R(s)+\gamma E[R_{t+2}+\gamma R_{t+3}+...+{\gamma }^{T-t-1}{R}_T|s_t=s=R(s)+\gamma E[G_{t+1}|s_t=s] \end{gathered}$$根据如下等式，记性贝尔曼方程的推导：
  $$E[V(s_{t+1})|s_t]=E[E[G_{t+1}|s_{t+1}]|s_t]=E[G_{t+1}|s_t]$$(这个等式用Law of Total Expectation:$E(x)={\sum }_{i}E(X|A_i)P(A_i)$进行推导，推导过程：推导过程)$$\begin{gathered} V(s_{t+1})=E[G_{t+1}|s_{t+1}]=G_{t+1} \\ \Rightarrow V(s_{t+1})=G_{t+1} \end{gathered}$$$$\begin{gathered} V(s)=R(s)+\gamma E[G_{t+1}|s_t=s] \\ =R(s)+\gamma E[V(s_{t+1})|s_t=s] \\ =R(s)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s)V(s_{t+1}) \end{gathered}$$