Reinforcement Learning Exercise 4.5

Exercise 4.5 How would policy iteration be defined for action values? Give a complete algorithm for computing $q_{\ast }$, analogous to that on page 80 for computing $v_{\ast }$. Please pay special attention to this exercise, because the ideas involved will be used throughout the rest of the book.

Here, we can use the result of exercise 3.17:
$$Q_{\pi }(s,a)=\sum _{s^{\prime }}{R}_{s,s^{\prime }}^a{P}_{s{s}^{\prime }}^a+\gamma \sum _{s^{\prime }}[\sum _{a^{\prime }}{Q}_{\pi }(s^{\prime },a^{\prime })\pi (s^{\prime },a^{\prime })]P_{s,s^{\prime }}^a$$
Then the algorithm which analogous to that on page 80 can be like this:
$$\begin{array}{rl}& \text{1 Initialization}\\ & Q_{\pi }(s,a)\in \mathbb{R}and\pi (s,a)\in \mathcal{A}(s)arbitrarilys\in \mathcal{S}anda\in \mathcal{A}(s)\\ & \text{2 Policy Evaluation}\\ & \text{Loop:}\\ & \Delta \leftarrow 0\\ & \text{Loop for each }(s,a)\text{ pair:}\\ & q\leftarrow Q_{\pi }(s,a)\\ & Q_{\pi }(s,a)\leftarrow \sum _{s^{\prime }}{R}_{s,s^{\prime }}^a{P}_{s{s}^{\prime }}^a+\gamma \sum _{s^{\prime }}[\sum _{a^{\prime }}{Q}_{\pi }(s^{\prime },a^{\prime })\pi (s^{\prime },a^{\prime })]P_{s,s^{\prime }}^a\\ & \Delta \leftarrow \max (\Delta ,|q-Q_{\pi }(s,a)|)\\ & \text{until }\Delta <\theta \text{ (a small positive number determining the accuracy of estimation) }\\ & \text{3 Policy Improvement}\\ & policy\text{-}stable\leftarrow true\\ & \text{For each }(a,s)\text{ pair, }s\in \mathcal{S}\text{ and }a\in \mathcal{A}(s):\\ & old\text{-}action\leftarrow \pi (s,a)\\ & \pi (s)\leftarrow {\text{argmax}}_{s,a}{Q}_{\pi }(s,a)\\ & \text{If }old\text{-}action≠\pi (s)\text{, then }policy\text{-}stable\leftarrow false\\ & \text{If }policy\text{-}stable\text{, then stop and return }{Q}_{\pi }\approx q_{\ast }\text{ and }\pi \approx {\pi }_{\ast }\text{; else go to 2.}\end{array}$$