Reinforcement Learning Exercise 4.2

Exercise 4.2 In Example 4.1, suppose a new state 15 is added to the gridworld just below state 13, and its actions, left, up, right, and down, take the agent to states 12, 13, 14, and 15, respectively. Assume that the transitions from the original states are unchanged. What, then, is $v_{\pi }(15)$ for the equiprobable random policy? Now suppose the dynamics of state 13 are also changed, such that action down from state 13 takes the agent to the new state 15. What is $v_{\pi }(15)$ for the equiprobable random policy in this case?

For the assumption that the transitions from the original states are unchanged, according to equation (4.4), we have:
$$\begin{array}{rl}{v}_{\pi }(s)& =\sum _a\pi (a\mid s)\sum _{s^{\prime },r}p(s^{\prime },r\mid s,a)[r+\gamma v_{\pi }(s^{\prime })]\\ & =\sum _a\pi (a\mid s)\sum _{s^{\prime }}\left\{\sum _r\right[r\cdot p(s^{\prime },r\mid s,a)]+\sum _r[p(s^{\prime },r\mid s,a)\cdot \gamma v_{\pi }(s^{\prime })\left]\right\}\\ & =\sum _a\pi (a\mid s)\sum _{s^{\prime }}\left\{\sum _r\right[r\cdot p(r\mid s^{\prime },s,a)\cdot p(s^{\prime }\mid s,a)]+p(s^{\prime }\mid s,a)\cdot \gamma v_{\pi }(s^{\prime })\}\\ & =\sum _a\pi (a\mid s)\sum _{s^{\prime }}\{p(s^{\prime }\mid s,a)[\sum _{r}r\cdot p(r\mid s^{\prime },s,a)+\gamma v_{\pi }(s^{\prime })\left]\right\}\\ & =\sum _a\pi (a\mid s)\sum _{s^{\prime }}\left\{P_{s,s^{\prime }}^a\right[R_{s,s^{\prime }}^a+\gamma v_{\pi }(s^{\prime })\left]\right\}\end{array}$$
So,
$$\begin{array}{rl}{v}_{\pi }(15)& =\sum _a\pi (a\mid 15)\cdot \left\{P_{15,12}^{left}\right[R_{15,12}^{left}+\gamma v_{\pi }(12)]+P_{15,13}^{up}[R_{15,13}^{up}+\gamma v_{\pi }(13)]\\ & +P_{15,14}^{right}[R_{15,14}^{right}+\gamma v_{\pi }(14)]+P_{15,15}^{down}[R_{15,15}^{down}+\gamma v_{\pi }(15)]\}\end{array}$$
Because the agent follows the equiprobable random policy, for all actions $\pi (a\mid s)=1/4$. And the action is deterministic, so:
$$P_{s,s^{\prime }}^a=\{\begin{array}{ll}1& \text{ if a leads to s′}\\ 0& \text{if a doesn’t lead to s′}\end{array}$$
According to Figure 4.2, we have:
$$\begin{array}{rl}{v}_{\pi }(15)& =\frac{1}{4}\{1\cdot [-1+\gamma (-22)]+1\cdot [-1+\gamma (-20)]\\ & +1\cdot [-1+\gamma (-14)]+1\cdot [-1+\gamma v_{\pi }(15)]\}\\ & =-1-14\gamma +\gamma v_{\pi }(15)\end{array}$$
$$\therefore v_{\pi }(15)=\frac{4+56\gamma }{\gamma -4}$$
For the assumption that the dynamics of state 13 are also changed, similarly we have:
$$\begin{array}{rl}{v}_{\pi }(13)& =\sum _a\pi (a\mid 13)\cdot \left\{P_{13,12}^{left}\right[R_{13,12}^{left}+\gamma v_{\pi }(12)]+P_{13,9}^{up}[R_{13,9}^{up}+\gamma v_{\pi }(9)]\\ & +P_{13,14}^{right}[R_{13,14}^{right}+\gamma v_{\pi }(14)]+P_{13,15}^{down}[R_{13,15}^{down}+\gamma v_{\pi }(15)]\}\\ v_{\pi }(15)& =\sum _a\pi (a\mid 15)\cdot \left\{P_{15,12}^{left}\right[R_{15,12}^{left}+\gamma v_{\pi }(12)]+P_{15,13}^{up}[R_{15,13}^{up}+\gamma v_{\pi }(13)]\\ & +P_{15,14}^{right}[R_{15,14}^{right}+\gamma v_{\pi }(14)]+P_{15,15}^{down}[R_{15,15}^{down}+\gamma v_{\pi }(15)]\}\end{array}$$
$$\begin{array}{rl}{v}_{\pi }(13)& =\frac{1}{4}\cdot \left\{1\right[-1+\gamma (-22)]+1[(-1+\gamma (-20)]\\ & +1[(-1+\gamma (-14)]+1[(-1+\gamma v_{\pi }(15)]\}\\ & =-1-14\gamma +\frac{1}{4}\gamma v_{\pi }(15)(1)\\ v_{\pi }(15)& =\frac{1}{4}\cdot \left\{1\right[-1+\gamma (-22)]+1[-1+\gamma v_{\pi }(13)]\\ & +1[-1+\gamma (-14)]+1[-1+\gamma v_{\pi }(15)]\}\\ & =-1-9\gamma +\frac{1}{4}\gamma v_{\pi }(13)+\frac{1}{4}\gamma v_{\pi }(15)(2)\end{array}$$
Then we have equation set:
$$\begin{array}{rl}{v}_{\pi }(13)-\frac{1}{4}\gamma v_{\pi }(15)& =-1-14\gamma (3)\\ -\frac{1}{4}\gamma v_{\pi }(13)+(1-\frac{1}{4}\gamma )v_{\pi }(15)& =-1-9\gamma (4)\end{array}$$
By solving equation set (3) and (4), we can obtain:
$$\begin{array}{rl}{v}_{\pi }(15)& =\frac{14{\gamma }^2+37\gamma +4}{\frac{1}{4}{\gamma }^2+\gamma -4}\\ v_{\pi }(13)& =\frac{19{\gamma }^2+224\gamma -16}{{\gamma }^2+4\gamma -16}\end{array}$$