Reinforcement Learning Exercise 4.6

Exercise 4.6 Suppose you are restricted to considering only policies that are $ϵ$-soft, meaning that the probability of selecting each action in each state, $s$, is at least $ϵ/|\mathcal{A}(s)|$. Describe qualitatively the changes that would be required in each of the steps 3, 2, and 1, in that order, of the policy iteration algorithm for $v_{\ast }$ on page 80.

The algorithm on page 80 in section 4.2 is based on the assumption that the policy is deterministic. For a stochastic case, we can modify the algorithm like this:

$$\begin{array}{rl}& \text{1 Initialization}\\ & V(s)\in \mathbb{R}\text{ and }\pi (s)\in \mathcal{A}(s)\text{ arbitrarily }s\in \mathcal{S}\\ & \text{2 Policy Evaluation}\\ & \text{Loop:}\\ & \Delta \leftarrow 0\\ & \text{Loop for each }s\in \mathcal{S}:\\ & v\leftarrow V(s)\\ & V(s)\leftarrow \sum _{s^{\prime },r}\pi (a\mid s)p(s^{\prime },r\mid s,a)[r+\gamma V(s^{\prime })]\\ & \Delta \leftarrow \max (\Delta ,|v-V(s)|)\\ & \text{until }\Delta <\theta \text{ (a small positive number determining the accuracy of estimation) }\\ & \text{3 Policy Improvement}\\ & policy\text{-}stable\leftarrow true\\ & \text{For each }s\in \mathcal{S}:\\ & old\text{-}action\leftarrow \pi (s)\\ & \pi (s)\leftarrow {\text{argmax}}_a\sum _{s^{\prime },r}\pi (a\mid s)p(s^{\prime },r\mid s,a)[r+\gamma V(s^{\prime })]\\ & \text{If }old\text{-}action≠\pi (s)\text{, then }policy\text{-}stable\leftarrow false\\ & \text{If }policy\text{-}stable\text{, then stop and return }V\approx v_{\ast }\text{ and }\pi \approx {\pi }_{\ast }\text{; else go to 2.}\end{array}$$

Because the only policies are $ϵ$-soft, the probability that the policy doesn’t select action $a$ is $\frac{ϵ}{|\mathcal{A}(s)|}\cdot (|\mathcal{A}(s)|-1)$. So,
$$\begin{array}{rl}\pi (a\mid s)& =1-\frac{ϵ}{|\mathcal{A}(s)|}\cdot (|\mathcal{A}(s)|-1)\\ & =1-ϵ+\frac{ϵ}{|\mathcal{A}(s)|}\end{array}$$
Substitute this $\pi (a\mid s)$ into the algorithm, we can get the final result.