Reinforcement Learning- Exercise 3.17

Exercise 3.17 What is the Bellman equation for action values, that is, for $q_{\pi }$? It must give the action value $q_{\pi }(s,a)$ in terms of the action values, $q_{\pi }(s^{\prime },a^{\prime })$, of possible successors to the state–action pair (s, a). Hint: the backup diagram to the right corresponds to this equation. Show the sequence of equations analogous to (3.14), but for action values.

According to definition：
$$\begin{array}{rl}{Q}_{\pi }(s,a)& ={\mathbb{E}}_{\pi }(G_t|S_t=s,A_t=a)\\ & ={\mathbb{E}}_{\pi }(\sum _{k=0}^{\infty }{\gamma }^k{R}_t+k+1|S_t=s,A_t=a)\\ & =\sum _{s^{\prime }}[{\mathbb{E}}_{\pi }(\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}|S_t=s,A_t=a,S_{t+1}=s^{\prime })P(S_{t+1}=s^{\prime }|A_t=a,S_t=s)]\\ & =\sum _{s^{\prime }}\left\{\right[{\mathbb{E}}_{\pi }(R_{t+1}|S_t=s,A_t=a,S_{t+1}=s)+{\mathbb{E}}_{\pi }(\sum _{k=1}^{\infty }{\gamma }^k{R}_{t+1+k})]P(S_{t+1}=s^{\prime }|A_t=a,S_t=s)\}\end{array}$$
Denote
$$P(S_{t+1}=s^{\prime }|A_t=a,S_t=s)=P_{s,s^{\prime }}^a$$
$${\mathbb{E}}_{\pi }(R_{t+1}|S_t=s,A_t=a,S_{t+1}=s^{\prime })=R_{s,s^{\prime }}^a$$
then：
$$\begin{array}{rl}{Q}_{\pi }(s,a)& =\sum _{s^{\prime }}{R}_{s,s^{\prime }}^a{P}_{s{s}^{\prime }}^a+\sum _{s^{\prime }}[\mathbb{E}(\sum _{k=1}^{\infty }{\gamma }^k{R}_{t+1+k}|S_t=s,A_t=a,S_{t+1}=s^{\prime })P_{s,s^{\prime }}^a]\\ & =\sum _{s^{\prime }}{R}_{s,s^{\prime }}^a{P}_{s{s}^{\prime }}^a+\gamma \sum _{s^{\prime }}[\mathbb{E}(\sum _{k=1}^{\infty }{\gamma }^{k-1}{R}_{t+1+k}|S_t=s,A_t=a,S_{t+1}=s^{\prime })P_{s,s^{\prime }}^a]\\ & =\sum _{s^{\prime }}{R}_{s,s^{\prime }}^a{P}_{s{s}^{\prime }}^a+\gamma \sum _{s^{\prime }}[\mathbb{E}(\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+2+k}|S_t=s,A_t=a,S_{t+1}=s^{\prime })P_{s,s^{\prime }}^a]\\ & =\sum _{s^{\prime }}{R}_{s,s^{\prime }}^a{P}_{s{s}^{\prime }}^a+\gamma \sum _{s^{\prime }}[\mathbb{E}(\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+2+k}|S_{t+1}=s^{\prime })P_{s,s^{\prime }}^a]\\ & =\sum _{s^{\prime }}{R}_{s,s^{\prime }}^a{P}_{s{s}^{\prime }}^a+\gamma \sum _{s^{\prime }}\left\{\sum _{a^{\prime }}\right[\mathbb{E}(\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+2+k}|S_{t+1}=s^{\prime },A_{t+1}=a^{\prime })P(A_{t+1}=a^{\prime }|S_{t+1}=s^{\prime })\left]P_{s,s^{\prime }}^a\right\}\end{array}$$
According to definition
$$\begin{gathered} \mathbb{E}(\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+2+k}|S_{t+1}=s^{\prime },A_{t+1}=a^{\prime })=Q_{\pi }(s^{\prime },a^{\prime }) \\ P(A_{t+1}=a^{\prime }|S_{t+1}=s^{\prime })=\pi (s^{\prime },a^{\prime }) \end{gathered}$$
so
$$Q_{\pi }(s,a)=\sum _{s^{\prime }}{R}_{s,s^{\prime }}^a{P}_{s{s}^{\prime }}^a+\gamma \sum _{s^{\prime }}[\sum _{a^{\prime }}{Q}_{\pi }(s^{\prime },a^{\prime })\pi (s^{\prime },a^{\prime })]P_{s,s^{\prime }}^a$$
This is the Bellman equation of action-value.