贝尔曼方程推导[2]

参考内容

• 贝尔曼方程推导[1]
• 深度强化学习（张志华，王树森）

贝尔曼方程

所有的数学工具都在上面了。接下来就是分析四个表达式之间的关系。首先把上述四个表达式拓展一下，也就是拓展到下一步进行分析。
$t+1$步的动作价值函数：
$$\begin{array}{rl}V(a_{t+1}|f_a,f_r,f_t,s_{t+1})& =\sum _{s_{t+2}}\cdots \sum _{s_{t+n}}\sum _{a_{t+2}}\cdots \sum _{a_{t+n}}[r_{t+1}(s_{t+1},a_{t+1})+\gamma r_{t+2}(s_{t+2},a_{t+2})+\cdots +{\gamma }^{n-1}{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P(s_{t+2}|s_{t+1},a_{t+1})\ast P(a_{t+2}|s_{t+2})\ast \cdots \ast P(a_{t+n}|s_{t+n})]\\ & =\sum _{s_{t+2}}\cdots \sum _{s_{t+n}}\sum _{a_{t+2}}\cdots \sum _{a_{t+n}}U(a_{t+1}|s_{t+1},f_a,f_r,f_t)\ast [P(s_{t+2}|s_{t+1},a_{t+1})\ast P(a_{t+2}|s_{t+2})\ast \cdots \ast P(a_{t+n}|s_{t+n})]\end{array}$$

将上面$t$步动作价值函数抄下来：
$$\begin{array}{rl}V(a_t|f_a,f_r,f_t,s_t)& =\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}[r_t(s_t,a_t)+\gamma r_{t+1}(s_{t+1},a_{t+1})+\cdots +{\gamma }^n{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\ast \cdots \ast P(a_{t+n}|s_{t+n})]\\ & =\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}U(a_t|s_t,f_a,f_r,f_t)\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\ast \cdots \ast P(a_{t+n}|s_{t+n})\end{array}$$

由于：
$$U(a_t|s_t,f_a,f_r,f_t)=r_t(s_t,a_t)+\gamma U(a_{t+1}|s_{t+1},f_a,f_r,f_t)$$

对于$t$步动作价值函数与$t+1$步动作价值函数有贝尔曼方程$A.1$：
$$\begin{array}{rl}V(a_t|f_a,f_r,f_t,s_t)& =\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}{r}_t(s_t,a_t)[P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\ast \cdots \ast P(a_{t+n}|s_{t+n})]+\sum _{s_{t+1}}\sum _{a_{t+1}}\gamma V(a_{t+1}|f_a,f_r,f_t,s_{t+1})\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\\ & =r_t(s_t,a_t)+\sum _{s_{t+1}}\sum _{a_{t+1}}\gamma V(a_{t+1}|f_a,f_r,f_t,s_{t+1})\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\\ & =r_t(s_t,a_t)+\gamma \sum _{s_{t+1}}\cdots \sum _{s_{t+n}}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}U(a_{t+1}|s_{t+1},f_a,f_r,f_t)\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\ast \cdots \ast P(a_{t+n}|s_{t+n})\\ & =r_t(s_t,a_t)+\gamma \sum _{s_{t+1}}\sum _{a_{t+1}}\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\ast \sum _{s_{t+2}}\cdots \sum _{s_{t+n}}\sum _{a_{t+2}}\cdots \sum _{a_{t+n}}U(a_{t+1}|s_{t+1},f_a,f_r,f_t)\ast \cdots \ast P(a_{t+n}|s_{t+n})\\ & =r_t(s_t,a_t)+\gamma \sum _{s_{t+1}}\sum _{a_{t+1}}V(a_{t+1}|f_a,f_r,f_t,s_{t+1})\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\end{array}$$

$t+1$步的最优动作价值函数：
$$\begin{array}{rl}V(a_{t+1}|f_a^{\ast },f_r,f_t,s_{t+1})& =\sum _{s_{t+2}}\cdots \sum _{s_{t+n}}\sum _{a_{t+2}}\cdots \sum _{a_{t+n}}[r_{t+1}(s_{t+1},a_{t+1})+\gamma r_{t+2}(s_{t+2},a_{t+2})+\cdots +{\gamma }^{n-1}{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P(s_{t+2}|s_{t+1},a_{t+1})\ast P^{\ast }(a_{t+2}|s_{t+2})\ast \cdots \ast P^{\ast }(a_{t+n}|s_{t+n})]\\ & =\sum _{s_{t+2}}\cdots \sum _{s_{t+n}}\sum _{a_{t+2}}\cdots \sum _{a_{t+n}}U(a_{t+1}|s_{t+1},f_a,f_r,f_t)\ast [P(s_{t+2}|s_{t+1},a_{t+1})\ast P^{\ast }(a_{t+2}|s_{t+2})\ast \cdots \ast P^{\ast }(a_{t+n}|s_{t+n})]\end{array}$$

将$t$步最优动作价值函数抄下来：
$$V(a_t|f_a^{\ast },f_r,f_t,s_t)=\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}[r_t(s_t,a_t)+\gamma r_{t+1}(s_{t+1},a_{t+1})+\cdots +{\gamma }^n{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P(s_{t+1}|s_t,a_t)\ast P^{\ast }(a_{t+1}|s_{t+1})\ast \cdots \ast P^{\ast }(a_{t+n}|s_{t+n})]$$

同样有上式成立贝尔曼方程$A.1$：
$$V(a_t|f_a^{\ast },f_r,f_t,s_t)=r_t(s_t,a_t)+\sum _{s_{t+1}}\sum _{a_{t+1}}\gamma V(a_{t+1}|f_a^{\ast },f_r,f_t,s_{t+1})\ast P(s_{t+1}|s_t,a_t)\ast P^{\ast }(a_{t+1}|s_{t+1})$$

由于最优策略就是最大化动作价值函数，所以：
$$\begin{array}{rl}\underset{f_a}{\max }V(a_t|f_a,f_r,f_t,s_t)& =\underset{f_a}{\max }[r_t(s_t,a_t)+\gamma \sum _{s_{t+1}}\sum _{a_{t+1}}V(a_{t+1}|f_a,f_r,f_t,s_{t+1})\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})]\\ & =r_t(s_t,a_t)+\gamma \underset{f_a}{\max }[\sum _{s_{t+1}}\sum _{a_{t+1}}V(a_{t+1}|f_a,f_r,f_t,s_{t+1})\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})]\\ & =r_t(s_t,a_t)+\gamma \sum _{s_{t+1}}P(s_{t+1}|s_t,a_t)\ast \underset{f_a}{\max }[\sum _{a_{t+1}}V(a_{t+1}|f_a,f_r,f_t,s_{t+1})\ast P(a_{t+1}|s_{t+1})]\end{array}$$

注意：我们需要调整的就是策略$P(a_{t+k}|s_{t+k})$，当我们顺序化考虑，首先考虑优化$P(a_{t+1}|s_{t+1})$时，此时不会影响$V(a_{t+1}|f_a,f_r,f_t,s_{t+1})$，也就是不会影响后面的动作价值函数。所以最优化的结果就是：
$$P^{\ast }(a_{t+1}|s_{t+1})=[0,\cdots ,1,0,\cdots ,0]$$

其中$1$对应位置上的动作具有如下特征：
$$a_{t+1}=\arg \underset{a_{t+1}}{\max }V(a_{t+1}|f_a,f_r,f_t,s_{t+1})$$

以上就是贝尔曼方程$A.4$。

也可以换个方向理解，其中的变换是通过分析因果关系得出的：
$$\begin{array}{rl}\underset{f_a}{\max }[\sum _{a_{t+1}}V(a_{t+1}|f_a,f_r,f_t,s_{t+1})\ast P(a_{t+1}|s_{t+1})]& =\underset{P(a_{t+1}|s_{t+1})}{\max }\underset{P(a_{t+2}|s_{t+2})}{\max }\cdots \underset{P(a_{t+n}|s_{t+n})}{\max }[\sum _{a_{t+1}}V(a_{t+1}|f_a,f_r,f_t,s_{t+1})\ast P(a_{t+1}|s_{t+1})]\\ & =\underset{P(a_{t+1}|s_{t+1})}{\max }[\sum _{a_{t+1}}[\underset{P(a_{t+2}|s_{t+2})}{\max }\cdots \underset{P(a_{t+n}|s_{t+n})}{\max }V(a_{t+1}|f_a,f_r,f_t,s_{t+1})]\ast P(a_{t+1}|s_{t+1})]\end{array}$$

通过分析我们发现：调整$P(a_{t+1}|s_{t+1})$不会影响后面的$[\underset{P(a_{t+2}|s_{t+2})}{\max }\cdots \underset{P(a_{t+n}|s_{t+n})}{\max }V(a_{t+1}|f_a,f_r,f_t,s_{t+1})]$，所以可以把$[\underset{P(a_{t+2}|s_{t+2})}{\max }\cdots \underset{P(a_{t+n}|s_{t+n})}{\max }V(a_{t+1}|f_a,f_r,f_t,s_{t+1})]$看成独立于$P(a_{t+1}|s_{t+1})$的常量，最优策略函数就是：
$$P^{\ast }(a_{t+1}|s_{t+1})=[0,\cdots ,1,0,\cdots ,0]$$

其中$1$对应位置上的动作变量$a_{t+1}^{\ast }$为：
$$a_{t+1}^{\ast }=\arg \underset{a_{t+1}}{\max }[\underset{P(a_{t+2}|s_{t+2})}{\max }\cdots \underset{P(a_{t+n}|s_{t+n})}{\max }V(a_{t+1}|f_a,f_r,f_t,s_{t+1})]$$

说人话就是，在当前动作价值函数是策略意义上最优的时候，当前策略就是以$1$的概率选择使当前策略最优意义下动作价值函数达到最大值所对应的动作。这就是最优策略。

$t+1$步的状态价值函数：
$$V(s_{t+1}|f_a,f_r,f_t)=\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}\sum _{s_{t+2}}\cdots \sum _{s_{t+n}}[r_{t+1}(s_{t+1},a_{t+1})+\gamma r_{t+2}(s_{t+2},a_{t+2})+\cdots +{\gamma }^{n-1}{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P(a_{t+1}|s_{t+1})\ast P(s_{t+2}|s_{t+1},a_{t+1})\ast P(a_{t+2}|s_{t+2})\ast \cdots \ast P(a_{t+n}|s_{t+n})]$$

将$t$步状态价值函数抄下来：
$$V(s_t|f_a,f_r,f_t)=\sum _{a_t}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}[r_t(s_t,a_t)+\gamma r_{t+1}(s_{t+1},a_{t+1})+\cdots +{\gamma }^n{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P(a_t|s_t)\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\ast \cdots \ast P(a_{t+n}|s_{t+n})]$$

可以看到，通过逐个减少固定变量的个数可以消除掉其余的变量。首先固定所有，然后将最后一个变为不固定，消除掉，然后逐渐消除到$s_{t+1}$，此时固定$a_t$，发现可以消除$P(s_{t+1}|a_t,s_t)$，得到贝尔曼方程$A.3$：
$$\begin{array}{rl}V(s_t|f_a,f_r,f_t)& =\sum _{a_t}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}{r}_t(s_t,a_t)\ast [P(a_t|s_t)\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\ast \cdots \ast P(a_{t+n}|s_{t+n})]+\gamma V(s_{t+1}|f_a,f_r,f_t)\ast P(a_t|s_t)\ast P(s_{t+1}|a_t,s_t)\\ & =\sum _{a_t}{r}_t(s_t,a_t)\ast P(a_t|s_t)+\sum _{a_t}\sum _{s_{t+1}}\gamma V(s_{t+1}|f_a,f_r,f_t)\ast P(a_t|s_t)\ast P(s_{t+1}|a_t,s_t)\end{array}$$

$t+1$步的最优状态价值函数：
$$V(s_{t+1}|f_a^{\ast },f_r,f_t)=\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}\sum _{s_{t+2}}\cdots \sum _{s_{t+n}}[r_{t+1}(s_{t+1},a_{t+1})+\gamma r_{t+2}(s_{t+2},a_{t+2})+\cdots +{\gamma }^{n-1}{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P^{\ast }(a_{t+1}|s_{t+1})\ast P(s_{t+2}|s_{t+1},a_{t+1})\ast P^{\ast }(a_{t+2}|s_{t+2})\ast \cdots \ast P^{\ast }(a_{t+n}|s_{t+n})]$$

将$t$步最优状态价值函数抄下来：
$$V(s_t|f_a^{\ast },f_r,f_t)=\sum _{a_t}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}[r_t(s_t,a_t)+\gamma r_{t+1}(s_{t+1},a_{t+1})+\cdots +{\gamma }^n{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P^{\ast }(a_t|s_t)\ast P(s_{t+1}|s_t,a_t)\ast P^{\ast }(a_{t+1}|s_{t+1})\ast \cdots \ast P^{\ast }(a_{t+n}|s_{t+n})]$$

可以看出也有类似的公式贝尔曼方程$A.3$：
$$V(s_t|f_a^{\ast },f_r,f_t)=\sum _{a_t}{r}_t(s_t,a_t)\ast P^{\ast }(a_t|s_t)+\sum _{a_t}\sum _{s_{t+1}}\gamma V(s_{t+1}|f_a^{\ast },f_r,f_t)\ast P^{\ast }(a_t|s_t)\ast P(s_{t+1}|a_t,s_t)$$

然后分析一下交叉项之间的表达式，也就是动作价值函数与状态价值函数之间的关系：
$t$步动作价值函数：
$$V(a_t|f_a,f_r,f_t,s_t)=\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}[r_t(s_t,a_t)+\gamma r_{t+1}(s_{t+1},a_{t+1})+\cdots +{\gamma }^n{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\ast \cdots \ast P(a_{t+n}|s_{t+n})]$$

$t$步状态价值函数：
$$V(s_t|f_a,f_r,f_t)=\sum _{a_t}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}[r_t(s_t,a_t)+\gamma r_{t+1}(s_{t+1},a_{t+1})+\cdots +{\gamma }^n{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P(a_t|s_t)\ast P(s_{t+1}|s_t,a_t)\ast P(a_{t+1}|s_{t+1})\ast \cdots \ast P(a_{t+n}|s_{t+n})]$$

可以发现，将$a_t$固定的话，下方的表达式与上方表达式存在关系：
$$V(s_t|f_a,f_r,f_t)=\sum _{a_t}V(a_t|f_a,f_r,f_t,s_t)\ast P(a_t|s_t)$$

最优的状态价值函数与动作价值函数关系为：
最优动作价值函数：
$$V(a_t|f_a^{\ast },f_r,f_t,s_t)=\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}[r_t(s_t,a_t)+\gamma r_{t+1}(s_{t+1},a_{t+1})+\cdots +{\gamma }^n{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P(s_{t+1}|s_t,a_t)\ast P^{\ast }(a_{t+1}|s_{t+1})\ast \cdots \ast P^{\ast }(a_{t+n}|s_{t+n})]$$

最优状态价值函数：
$$V(s_t|f_a^{\ast },f_r,f_t)=\sum _{a_t}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}[r_t(s_t,a_t)+\gamma r_{t+1}(s_{t+1},a_{t+1})+\cdots +{\gamma }^n{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P^{\ast }(a_t|s_t)\ast P(s_{t+1}|s_t,a_t)\ast P^{\ast }(a_{t+1}|s_{t+1})\ast \cdots \ast P^{\ast }(a_{t+n}|s_{t+n})]$$

关系为：
$$V(s_t|f_a^{\ast },f_r,f_t)=\sum _{a_t}V(a_t|f_a^{\ast },f_r,f_t,s_t)\ast P^{\ast }(a_t|s_t)$$

最优动作价值函数与最优状态价值函数之间的关系为：
最优动作价值函数：
$$V(a_t|f_a^{\ast },f_r,f_t,s_t)=\sum _{s_{t+1}}\cdots \sum _{s_{t+n}}\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}[r_t(s_t,a_t)+\gamma r_{t+1}(s_{t+1},a_{t+1})+\cdots +{\gamma }^n{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P(s_{t+1}|s_t,a_t)\ast P^{\ast }(a_{t+1}|s_{t+1})\ast \cdots \ast P^{\ast }(a_{t+n}|s_{t+n})]$$

最优状态价值函数：
$$V(s_{t+1}|f_a^{\ast },f_r,f_t)=\sum _{a_{t+1}}\cdots \sum _{a_{t+n}}\sum _{s_{t+2}}\cdots \sum _{s_{t+n}}[r_{t+1}(s_{t+1},a_{t+1})+\gamma r_{t+2}(s_{t+2},a_{t+2})+\cdots +{\gamma }^{n-1}{r}_{t+n}(s_{t+n},a_{t+n})]\ast [P^{\ast }(a_{t+1}|s_{t+1})\ast P(s_{t+2}|s_{t+1},a_{t+1})\ast P^{\ast }(a_{t+2}|s_{t+2})\ast \cdots \ast P^{\ast }(a_{t+n}|s_{t+n})]$$

相互之间关系为贝尔曼方程$A.2$：
$$V(a_t|f_a^{\ast },f_r,f_t,s_t)=r_t(s_t,a_t)+\sum _{s_{t+1}}\gamma V(s_{t+1}|f_a^{\ast },f_r,f_t)\ast P(s_{t+1}|s_t,a_t)$$

总结

自我感觉有了这套符号系统，分析理论方便多了。