Reinforcement Learning Exercise 3.23

Exercise 3.23 Give the Bellman equation for $q_{\ast }$ for the recycling robot.

This picture shows the mechanism of the recycling robot.

To give the Bellman equation for $q_{\ast }$ for the recycling robot, we have to enumerate equations for $q_{\ast }(s_h,a_s)$, $q_{\ast }(s_h,a_w)$, $q_{\ast }(s_h,a_r)$, $q_{\ast }(s_l,a_s)$,$q_{\ast }(s_l,a_w)$ and $q_{\ast }(s_l,a_r)$. Here, the subscripts h, l, s, w, r respectively denotes ‘high’, ‘low’, ‘search’, ‘wait’, ‘recharge’. For ‘high’ status, the available actions are ‘search’ and ‘wait’, so $q_{\ast }(s_h,a_r)$ is excluded.
First, we have to introduce the equation (1) from exercise 3.22:
$$q_{\ast }(s,a)=\sum _{s^{\prime }}\left\{\right[R_{s,s^{\prime }}^a+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a^{\prime })\left]P_{s,s^{\prime }}^a\right\}(1)$$
For status ‘high’, we have:
$$\begin{array}{r}{q}_{\ast }(s_h,a_s)=[R_{s_h,s_h}^{a_s}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })]P_{s_h,s_h}^{a_s}+[R_{s_h,s_l}^{a_s}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })]P_{s_h,s_l}^{a_s}(2)\\ q_{\ast }(s_h,a_w)=[R_{s_h,s_h}^{a_w}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })]P_{s_h,s_h}^{a_w}+[R_{s_h,s_l}^{a_w}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })]P_{s_h,s_l}^{a_w}(3)\end{array}$$
For status ‘low’, there are:
$$\begin{array}{rl}{q}_{\ast }(s_l,a_s)& =[R_{s_l,s_h}^{a_s}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })]P_{s_l,s_h}^{a_s}+[R_{s_l,s_l}^{a_s}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })]P_{s_l,s_l}^{a_s}(4)\\ q_{\ast }(s_l,a_w)& =[R_{s_l,s_h}^{a_w}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })]P_{s_l,s_h}^{a_w}+[R_{s_l,s_l}^{a_w}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })]P_{s_l,s_l}^{a_w}(5)\\ q_{\ast }(s_l,a_r)& =[R_{s_l,s_h}^{a_r}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })]P_{s_l,s_h}^{a_r}+[R_{s_l,s_l}^{a_r}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })]P_{s_l,s_l}^{a_r}(6)\end{array}$$
Then according to the table in the above picture, $R_{s_h,s_h}^{a_s}=r_{search}$, $P_{s_h,s_h}^{a_s}=\alpha$, $R_{s_h,s_l}^{a_s}=r_{search}$, $P_{s_h,s_l}^{a_s}=1-\alpha$, … and so on. Plug these values into equations (2), (3), (4), (5), (6), we get:
$$\begin{array}{rl}{q}_{\ast }(s_h,a_s)& =[r_{search}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })]\alpha +[r_{search}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })](1-\alpha )\\ & =r_{search}+\gamma [\alpha \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })+(1-\alpha )\underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })](7)\\ q_{\ast }(s_h,a_w)& =[r_{wait}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })]\cdot 1+[R_{s_h,s_l}^{a_w}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })]\cdot 0\\ & =r_{wait}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })(8)\\ q_{\ast }(s_l,a_s)& =[-3+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })](1-\beta )+[r_{search}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })]\beta \\ & =(r_{search}-3)+\gamma [(1-\beta )\underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })+\beta \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })](9)\\ q_{\ast }(s_l,a_w)& =[R_{s_l,s_h}^{a_w}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })]\cdot 0+[r_{wait}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })]\cdot 1\\ & =r_{wait}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })(10)\\ q_{\ast }(s_l,a_r)& =[0+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })]\cdot 1+[R_{s_l,s_l}^{a_r}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_l,a^{\prime })]\cdot 0\\ & =\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s_h,a^{\prime })(11)\end{array}$$
For ‘high’ status, $a^{\prime }$ can only be ‘search’ and ‘wait’ while for ‘low’ status, $a^{\prime }$ can be ‘search’, ‘wait’ and ‘recharge’. So, equations (7) to (11) can be rearranged as below:
$$\begin{array}{rl}{q}_{\ast }(s_h,a_s)& =r_{search}+\gamma \left\{\alpha \underset{a^{\prime }}{\max }\right[q_{\ast }(s_h,a_s),q_{\ast }(s_h,a_w)]+(1-\alpha )\underset{a^{\prime }}{\max }[q_{\ast }(s_l,a_s),q_{\ast }(s_l,a_w),q_{\ast }(s_l,a_r)\left]\right\}(12)\\ q_{\ast }(s_h,a_w)& =r_{wait}+\gamma \underset{a^{\prime }}{\max }[q_{\ast }(s_h,a_s),q_{\ast }(s_h,a_w)](13)\\ q_{\ast }(s_l,a_s)& =(r_{search}-3)+\gamma \{(1-\beta )\underset{a^{\prime }}{\max }[q_{\ast }(s_h,a_s),q_{\ast }(s_h,a_w)]+\beta \underset{a^{\prime }}{\max }[q_{\ast }(s_l,a_s),q_{\ast }(s_l,a_w),q_{\ast }(s_l,a_r)\left]\right\}(14)\\ q_{\ast }(s_l,a_w)& =r_{wait}+\gamma \underset{a^{\prime }}{\max }[q_{\ast }(s_l,a_s),q_{\ast }(s_l,a_w),q_{\ast }(s_l,a_r)](15)\\ q_{\ast }(s_l,a_r)& =\gamma \underset{a^{\prime }}{\max }[q_{\ast }(s_h,a_s),q_{\ast }(s_h,a_w)](16)\end{array}$$
These equations from (12) to (16) are the Bellman equations for the recycling robot and can be solved in a similar way like exercise 3.22.