Dynamic Programming and Optimal Control 第一章习题

1.1 Consider the system$$x_{k+1}=x_k+u_k+w_k,k=0,1,2,3$$with initial state $x_0=5$, and the cost function $$\sum _{k=0}^3(x_k^2+u_k^2)$$Apply the DP algorithm for the following three cases:
（a）The control constraint set $U_k(x_k)$ is { u ∣ 0 ≤ x k + u ≤ 5 , u : integer } \{u|0\leq x_k+u\leq 5, u \text{: integer}\} {u∣0≤xk​+u≤5,u: integer} for all $x_k$ and $k$, and the disturbance $w_k$ is equal to zero for all $k$.
（b） The control constraint and the disturbance $w_k$ are as in (a), but there is in addition a constraint $x_4=5$ on the final state.
（c）The control constraint is as in part (a) and the disturbance $w_k$ takes the values $-1$ and $1$ with equal probability $1/2$ for all $x_k$ and $u_k$, except if $x_k+u_k$ is equal to $0$ or $5$, in which case $w_k=0$ with probability $1$.
Solution.
（a）$N=4$, we assume that $J_4(x_4)=g_4(x_4)=0$. As the condition, the system equation takes the form $$f_{k+1}=x_k+u_k+w_k,k=0,1,2,3$$and the cost function is $$g_k=x_k^2+u_k^2$$In this case, $w_k=0$ for all $k$, so J 3 ( x 3 ) = min ⁡ u 3 ∈ U 3 ( x 3 ) E w 3 { g 3 ( x 3 , u 3 , w 3 ) + J 4 ( f 3 ( x 3 , u 3 , w 3 ) ) } J_3(x_3)=\mathop{\min}\limits_{u_3\in U_3(x_3)} \mathop\mathbf{E}\limits_{w_3}\{g_3(x_3,u_3,w_3)+J_4(f_3(x_3,u_3,w_3))\} J3​(x3​)=u3​∈U3​(x3​)min​w3​E​{g3​(x3​,u3​,w3​)+J4​(f3​(x3​,u3​,w3​))}$$=\underset{u_3=-x_3,1-x_3,...,5-x_3}{\min }(x_3^2+u_3^2)=x_3^2$$i.e. $J_3(x_3)=x_3^2,{\mu }_3^{\ast }(x_3)=0.(1)$
Then $$J_2(x_2)=\underset{u_2=-x_2,1-x_2,...,5-x_2}{\min }(x_2^2+u_2^2+(x_2+u_2{)}^2)$$Let $h_{x_2}(u_2)=x_2^2+u_2^2+(x_2+u_2{)}^2$, we have $h_{x_2}^{\prime }(u_2)=4u_2+2x_2.$

If $x_2>10$, $h_{x_2}(u_2)=4(u_2+x_2)-2x_2<4(u_2+x_2)-20<0$, then $${\mu }_x^{\ast }(x_2)=5-x_2,J_x(x_2)=x_2^2+(5-x_2{)}^2+5^2=2x_2^2-10x_2+50$$

If $x_2<0$, $h_{x_2}(u_2)=4(u_2+x_2)-2x_2>-2x_2>0$, then $${\mu }_x^2(x_2)=-x_2,J_2(x_2)=2x_2^2$$

If $0\le x_2\le 10$ and $x_2$ is even, then ${\mu }_2^{\ast }(x_2)=-\frac{x_2}{2}$ and $J_2(x_2)=\frac{3}{2}{x}_2^2.$

If $0\le x_2\le 10$ and $x_2$ is odd, then ${\mu }_2^{\ast }(x_2)=-\frac{x_2-1}{2}\text{or}-\frac{x_2+1}{2}$ and $J_2(x_2)=\frac{3}{2}{x}_2^2+\frac{1}{2}.$

So
$\{\begin{array}{ll}{\mu }_x^2(x_2)=-x_2,J_2(x_2)=2x_2^2& \text{if }{x}_2<0\\ {\mu }_2^{\ast }(x_2)=-\frac{x_2}{2},J_2(x_2)=\frac{3}{2}{x}_2^2& \text{if }0\le x_2\le 10\text{and}{x}_2\text{is even}\\ {\mu }_2^{\ast }(x_2)=-\frac{x_2-1}{2}\text{or}-\frac{x_2+1}{2},J_2(x_2)=\frac{3}{2}{x}_2^2+\frac{1}{2}& \text{if }0\le x_2\le 10\text{and}{x}_2\text{is odd}\\ {\mu }_x^{\ast }(x_2)=5-x_2,J_x(x_2)=x_2^2+(5-x_2{)}^2+5^2=2x_2^2-10x_2+50& \text{if }{x}_2>10\end{array}$

In case $k=1$
$$J_1(x_1)=\underset{u_1=-x_1,1-x_1,...,5-x_1}{\min }(x_1^2+u_1^2+J_2(x_1+u_1))$$$$=\underset{u_1=-x_1,1-x_1,...,5-x_1}{\min }(x_1^2+u_1^2+\frac{3}{2}(x_1+u_1{)}^2+\frac{1}{2}\cdot 1_{x_1+u_1\text{is odd}})$$ In this case, we give an caculation in a direct way:

Let $h_{x_1}(u_1)=x_1^2+u_1^2+\frac{3}{2}(x_1+u_1{)}^2+\frac{1}{2}\cdot 1_{x_1+u_1\text{is odd}}$
$$h_{x_1}(-x_1)=2x_1^2$$$$h_{x_1}(1-x_1)=2x_1^2-2x_1+3$$$$h_{x_1}(2-x_1)=2x_1^2-4x_1+10$$$$h_{x_1}(3-x_1)=2x_1^2-6x_1+23$$$$h_{x_1}(4-x_1)=2x_1^2-8x_1+40$$$$h_{x_1}(5-x_1)=2x_1^2-10x_1+63$$Then we can get the following result easily
$\{\begin{array}{ll}{\mu }_1^{\ast }(x_1)=-x_1,J_1(x_1)=2x_1^2& \text{if }{x}_1\le \frac{3}{2}\\ {\mu }_1^{\ast }(x_1)=1-x_1,J_1(x_1)=2x_1^2-2x_1+3& \text{if }\frac{3}{2}<x_1\le \frac{7}{2}\\ {\mu }_1^{\ast }(x_1)=2-x_1,J_1(x_1)=2x_1^2-4x_1+10& \text{if }\frac{7}{2}<x_1\le \frac{13}{2}\\ {\mu }_1^{\ast }(x_1)=3-x_1,J_1(x_1)=2x_1^2-6x_1+23& \text{if }\frac{13}{2}<x_1\le \frac{17}{2}\\ {\mu }_1^{\ast }(x_1)=4-x_1,J_1(x_1)=2x_1^2-8x_1+40& \text{if }\frac{17}{2}<x_1\le \frac{23}{2}\\ {\mu }_1^{\ast }(x_1)=5-x_1,J_1(x_1)=2x_1^2-10x_1+63& \text{if }{x}_1>\frac{23}{2}\end{array}$

For case $k=0,x_0=5$,$$J_0(x_0)=\underset{u_0=-x_0,1-x_0,...,5-x_0}{\min }(x_0^2+u_0^2+J_1(x_0+u_0))=\underset{u_0=-5,-4,...,0}{\min }(25+u_0^2+J_1(5+u_0))$$Let$$h_0(u_0)=25+u_0^2+J_1(5+u_0)$$we have
$$h_0(-5)=50,h_0(-4)=43,h_0(-3)=41,h_0(-2)=44,h_0(-1)=50,h_0(0)=65$$So ${\mu }_0^{\ast }(x_0)=-3$ and $J_0(x_0)=41$.

（b） In this case ,we still asume that $J_4(x_4)=g_4(x_4)=0$. By the additional condition, U 3 ( x 3 ) = { 5 − x 3 } U_3(x_3)=\{5-x_3\} U3​(x3​)={5−x3​}, and
$$J_3(x_3)=\underset{u_3\in U_3(x_3)}{\min }(x_3^2+u_3^2)=x_3^2+(5-x_3{)}^2$$Then$$J_2(x_2)=\underset{u_2=-x_2,1-x_2,...,5-x_2}{\min }(x_2^2+u_2^2+(5-x_2-u_2{)}^2)$$Let $h_{x_2}(u_2)=x_2^2+u_2^2+(5-x_2-u_2{)}^2$, take some easy caculation
$$h_{x_2}(-x_2)=2x_2^2+25$$$$h_{x_2}(1-x_2)=2x_2^2-2x_2+18$$$$h_{x_2}(2-x_2)=2x_2^2-4x_1+17$$$$h_{x_2}(3-x_2)=2x_2^2-6x_2+22$$$$h_{x_2}(4-x_2)=2x_2^2-8x_2+33$$$$h_{x_2}(5-x_2)=2x_2^2-10x_2+50$$and we get
$\{\begin{array}{ll}{\mu }_2^{\ast }(x_2)=-x_2,J_2(x_2)=2x_2^2+25& \text{if }{x}_1\le -\frac{7}{2}\\ {\mu }_2^{\ast }(x_2)=1-x_2,J_2(x_2)=2x_2^2-2x_2+18& \text{if }-\frac{7}{2}<x_1\le -\frac{1}{2}\\ {\mu }_2^{\ast }(x_2)=2-x_2,J_2(x_2)=2x_2^2-4x_2+17& \text{if }-\frac{1}{2}<x_1\le \frac{5}{2}\\ {\mu }_2^{\ast }(x_2)=3-x_2,J_2(x_2)=2x_2^2-6x_2+22& \text{if }\frac{5}{2}<x_1\le \frac{11}{2}\\ {\mu }_2^{\ast }(x_2)=4-x_2,J_2(x_2)=2x_2^2-8x_2+33& \text{if }\frac{11}{2}<x_1\le \frac{17}{2}\\ {\mu }_2^{\ast }(x_2)=5-x_2,J_2(x_2)=2x_2^2-10x_2+50& \text{if }{x}_1>\frac{17}{2}\end{array}$

Similarly, for $k=1$ we have
$\{\begin{array}{ll}{\mu }_1^{\ast }(x_1)=-x_1,J_1(x_1)=2x_1^2+17& \text{if }{x}_1\le -\frac{1}{2}\\ {\mu }_1^{\ast }(x_1)=1-x_1,J_1(x_1)=2x_1^2-2x_1+16& \text{if }-\frac{1}{2}<x_1\le \frac{5}{2}\\ {\mu }_1^{\ast }(x_1)=2-x_1,J_1(x_1)=2x_1^2-4x_1+21& \text{if }\frac{5}{2}<x_1\le 5\\ {\mu }_1^{\ast }(x_1)=3-x_1,J_1(x_1)=2x_1^2-6x_1+31& \text{if }5<x_1\le \frac{15}{2}\\ {\mu }_1^{\ast }(x_1)=4-x_1,J_1(x_1)=2x_1^2-8x_1+46& \text{if }\frac{15}{2}<x_1\le \frac{21}{2}\\ {\mu }_1^{\ast }(x_1)=5-x_1,J_1(x_1)=2x_1^2-10x_1+67& \text{if }{x}_1>\frac{21}{2}\end{array}$

For case $k=0,x_0=5$
$$h_0(-5)=66,h_0(-4)=57,h_0(-3)=54,h_0(-2)=56,h_0(-1)=63,h_0(0)=76$$So ${\mu }_0^{\ast }(x_0)=-3$ and $J_0(x_0)=54$.

（c）In this case, $w_k$ is not always be $0$ and satisfies certern distributuion. J 3 ( x 3 ) = min ⁡ u 3 ∈ U 3 ( x 3 ) E w 3 { g 3 ( x 3 , u 3 , w 3 ) + J 4 ( f 3 ( x 3 , u 3 , w 3 ) ) } = min ⁡ u 3 ∈ U 3 ( x 3 ) E w 3 ( x 3 2 + u 3 2 ) J_3(x_3)=\mathop{\min}\limits_{u_3\in U_3(x_3)} \mathop\mathbf{E}\limits_{w_3}\{g_3(x_3,u_3,w_3)+J_4(f_3(x_3,u_3,w_3))\}=\mathop{\min}\limits_{u_3\in U_3(x_3)} \mathop\mathbf{E}\limits_{w_3}(x_3^2+u_3^2) J3​(x3​)=u3​∈U3​(x3​)min​w3​E​{g3​(x3​,u3​,w3​)+J4​(f3​(x3​,u3​,w3​))}=u3​∈U3​(x3​)min​w3​E​(x32​+u32​)which is not affected by the distritution of $w_k$, so the result is the same as case (a):

$\{\begin{array}{ll}{\mu }_3^{\ast }(x_3)=-x_3,J_3(x_3)=2x_3^2& \text{if }{x}_3\le \frac{1}{2}\\ {\mu }_3^{\ast }(x_3)=1-x_3,J_3(x_3)=2x_3^2-2x_3+1& \text{if }\frac{1}{2}<x_3\le \frac{3}{2}\\ {\mu }_3^{\ast }(x_3)=2-x_3,J_3(x_3)=2x_3^2-4x_3+4& \text{if }\frac{3}{2}<x_3\le \frac{5}{2}\\ {\mu }_3^{\ast }(x_3)=3-x_3,J_3(x_3)=2x_3^2-6x_3+9& \text{if }\frac{5}{2}<x_3\le \frac{7}{2}\\ {\mu }_3^{\ast }(x_3)=4-x_3,J_3(x_3)=2x_3^2-8x_3+16& \text{if }\frac{7}{2}<x_3\le \frac{9}{2}\\ {\mu }_3^{\ast }(x_3)=5-x_3,J_3(x_3)=2x_3^2-10x_3+25& \text{if }{x}_3>\frac{9}{2}\end{array}$

Especially, we have $J_3(0)=0,J_3(1)=1,J_3(2)=4,J_3(3)=9,J_3(4)=16,J_3(5)=25,$ which will be used in the case $k=2$. Now $$J_2(x_2)=\underset{u_2\in U_2(x_2)}{\min }\underset{w_2}{E}(x_2^2+u_2^2+J_3(x_2+u_2+w_2))$$We calculate the expectation of the right side for each of the possible values of $u_2$:
u 2 = − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( − x 2 ) 2 + J 3 ( 0 ) = 2 x 2 2 u_2=-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(-x_2)^2+J_3(0)=2x_2^2 u2​=−x2​:E{⋅}=x22​+(−x2​)2+J3​(0)=2x22​ u 2 = 1 − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( 1 − x 2 ) 2 + 1 2 [ J 3 ( 0 ) + J 3 ( 2 ) ] = 2 x 2 2 − 2 x 2 + 3 u_2=1-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(1-x_2)^2+\frac{1}{2}[J_3(0)+J_3(2)]=2x_2^2-2x_2+3 u2​=1−x2​:E{⋅}=x22​+(1−x2​)2+21​[J3​(0)+J3​(2)]=2x22​−2x2​+3 u 2 = 2 − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( 2 − x 2 ) 2 + 1 2 [ J 3 ( 1 ) + J 3 ( 3 ) ] = 2 x 2 2 − 4 x 2 + 9 u_2=2-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(2-x_2)^2+\frac{1}{2}[J_3(1)+J_3(3)]=2x_2^2-4x_2+9 u2​=2−x2​:E{⋅}=x22​+(2−x2​)2+21​[J3​(1)+J3​(3)]=2x22​−4x2​+9 u 2 = 3 − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( 3 − x 2 ) 2 + 1 2 [ J 3 ( 2 ) + J 3 ( 4 ) ] = 2 x 2 2 − 6 x 2 + 19 u_2=3-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(3-x_2)^2+\frac{1}{2}[J_3(2)+J_3(4)]=2x_2^2-6x_2+19 u2​=3−x2​:E{⋅}=x22​+(3−x2​)2+21​[J3​(2)+J3​(4)]=2x22​−6x2​+19 u 2 = 4 − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( 4 − x 2 ) 2 + 1 2 [ J 3 ( 3 ) + J 3 ( 5 ) ] = 2 x 2 2 − 8 x 2 + 33 u_2=4-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(4-x_2)^2+\frac{1}{2}[J_3(3)+J_3(5)]=2x_2^2-8x_2+33 u2​=4−x2​:E{⋅}=x22​+(4−x2​)2+21​[J3​(3)+J3​(5)]=2x22​−8x2​+33 u 2 = 5 − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( 5 − x 2 ) 2 + J 3 ( 5 ) = 2 x 2 2 − 10 x 2 + 50 u_2=5-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(5-x_2)^2+J_3(5)=2x_2^2-10x_2+50 u2​=5−x2​:E{⋅}=x22​+(5−x2​)2+J3​(5)=2x22​−10x2​+50Hence we have

$\{\begin{array}{ll}{\mu }_2^{\ast }(x_2)=-x_2,J_2(x_2)=2x_2^2& \text{if }{x}_2\le \frac{3}{2}\\ {\mu }_2^{\ast }(x_2)=1-x_2,J_2(x_2)=2x_2^2-2x_2+3& \text{if }\frac{3}{2}<x_2\le 3\\ {\mu }_2^{\ast }(x_2)=2-x_2,J_2(x_2)=2x_2^2-4x_2+9& \text{if }3<x_2\le 5\\ {\mu }_2^{\ast }(x_2)=3-x_2,J_2(x_2)=2x_2^2-6x_2+19& \text{if }5<x_2\le 7\\ {\mu }_2^{\ast }(x_2)=4-x_2,J_2(x_2)=2x_2^2-8x_2+33& \text{if }7<x_2\le \frac{17}{2}\\ {\mu }_2^{\ast }(x_2)=5-x_2,J_2(x_2)=2x_2^2-10x_2+50& \text{if }{x}_2>\frac{17}{2}\end{array}$

Especially, $J_2(0)=0,J_2(1)=2,J_2(2)=7,J_2(3)=15,J_2(4)=25,J_2(5)=39.$

For case $k=1$$$J_1(x_1)=\underset{u_1\in U_1(x_1)}{\min }\underset{w_1}{E}(x_1^2+u_1^2+J_2(x_1+u_1+w_1))$$We calculate the expectation of the right side for each of the possible values of $u_1$:
u 1 = − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( − x 1 ) 2 + J 2 ( 0 ) = 2 x 1 2 u_1=-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(-x_1)^2+J_2(0)=2x_1^2 u1​=−x1​:E{⋅}=x12​+(−x1​)2+J2​(0)=2x12​ u 2 = 1 − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( 1 − x 1 ) 2 + 1 2 [ J 2 ( 0 ) + J 2 ( 2 ) ] = 2 x 1 2 − 2 x 1 + 9 2 u_2=1-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(1-x_1)^2+\frac{1}{2}[J_2(0)+J_2(2)]=2x_1^2-2x_1+\frac{9}{2} u2​=1−x1​:E{⋅}=x12​+(1−x1​)2+21​[J2​(0)+J2​(2)]=2x12​−2x1​+29​ u 1 = 2 − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( 2 − x 1 ) 2 + 1 2 [ J 2 ( 1 ) + J 2 ( 3 ) ] = 2 x 1 2 − 4 x 1 + 25 2 u_1=2-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(2-x_1)^2+\frac{1}{2}[J_2(1)+J_2(3)]=2x_1^2-4x_1+\frac{25}{2} u1​=2−x1​:E{⋅}=x12​+(2−x1​)2+21​[J2​(1)+J2​(3)]=2x12​−4x1​+225​ u 1 = 3 − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( 3 − x 1 ) 2 + 1 2 [ J 2 ( 2 ) + J 2 ( 4 ) ] = 2 x 1 2 − 6 x 1 + 25 u_1=3-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(3-x_1)^2+\frac{1}{2}[J_2(2)+J_2(4)]=2x_1^2-6x_1+25 u1​=3−x1​:E{⋅}=x12​+(3−x1​)2+21​[J2​(2)+J2​(4)]=2x12​−6x1​+25 u 1 = 4 − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( 4 − x 1 ) 2 + 1 2 [ J 2 ( 3 ) + J 2 ( 5 ) ] = 2 x 1 2 − 8 x 1 + 43 u_1=4-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(4-x_1)^2+\frac{1}{2}[J_2(3)+J_2(5)]=2x_1^2-8x_1+43 u1​=4−x1​:E{⋅}=x12​+(4−x1​)2+21​[J2​(3)+J2​(5)]=2x12​−8x1​+43 u 1 = 5 − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( 5 − x 1 ) 2 + J 2 ( 5 ) = 2 x 1 2 − 10 x 1 + 64 u_1=5-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(5-x_1)^2+J_2(5)=2x_1^2-10x_1+64 u1​=5−x1​:E{⋅}=x12​+(5−x1​)2+J2​(5)=2x12​−10x1​+64Hence we have

$\{\begin{array}{ll}{\mu }_1^{\ast }(x_1)=-x_1,J_1(x_1)=2x_1^2& \text{if }{x}_1\le \frac{9}{4}\\ {\mu }_1^{\ast }(x_1)=1-x_1,J_1(x_1)=2x_1^2-2x_1+\frac{9}{2}& \text{if }\frac{9}{4}<x_1\le 4\\ {\mu }_1^{\ast }(x_1)=2-x_1,J_1(x_1)=2x_1^2-4x_1+\frac{25}{2}& \text{if }4<x_1\le \frac{25}{4}\\ {\mu }_1^{\ast }(x_1)=3-x_1,J_1(x_1)=2x_1^2-6x_1+25& \text{if }\frac{25}{4}<x_1\le 9\\ {\mu }_1^{\ast }(x_1)=4-x_1,J_1(x_1)=2x_1^2-8x_1+43& \text{if }9<x_1\le \frac{21}{2}\\ {\mu }_1^{\ast }(x_1)=5-x_1,J_1(x_1)=2x_1^2-10x_1+64& \text{if }{x}_1>\frac{21}{2}\end{array}$

Especially, $J_1(0)=0,J_1(1)=2,J_1(2)=8,J_1(3)=\frac{15}{2},J_1(4)=\frac{57}{2},J_1(5)=\frac{85}{2}.$

The last step, the algorithm takes the form
$$J_0(x_0)=\underset{u_0\in U_0(x_0)}{\min }\underset{w_0}{E}(x_0^2+u_0^2+J_1(x_0+u_0+w_0))=\underset{u_0\in U_0(x_0)}{\min }\underset{w_0}{E}(25+u_0^2+J_1(5+u_0+w_0))$$ u 0 = − 5 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 25 + J 1 ( 0 ) = 50 u_0=-5\;:\;\mathbf{E}\{\cdot\}=25+25+J_1(0)=50 u0​=−5:E{⋅}=25+25+J1​(0)=50 u 0 = − 4 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 16 + 1 2 [ J 1 ( 0 ) + J 1 ( 2 ) ] = 45 u_0=-4\;:\;\mathbf{E}\{\cdot\}=25+16+\frac{1}{2}[J_1(0)+J_1(2)]=45 u0​=−4:E{⋅}=25+16+21​[J1​(0)+J1​(2)]=45 u 0 = − 3 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 9 + 1 2 [ J 1 ( 1 ) + J 1 ( 3 ) ] = 38 3 4 u_0=-3\;:\;\mathbf{E}\{\cdot\}=25+9+\frac{1}{2}[J_1(1)+J_1(3)]=38\frac{3}{4} u0​=−3:E{⋅}=25+9+21​[J1​(1)+J1​(3)]=3843​ u 0 = − 2 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 4 + 1 2 [ J 1 ( 2 ) + J 1 ( 4 ) ] = 47 1 4 u_0=-2\;:\;\mathbf{E}\{\cdot\}=25+4+\frac{1}{2}[J_1(2)+J_1(4)]=47\frac{1}{4} u0​=−2:E{⋅}=25+4+21​[J1​(2)+J1​(4)]=4741​ u 1 = − 1 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 1 + 1 2 [ J 1 ( 3 ) + J 1 ( 5 ) ] = 51 u_1=-1\;:\;\mathbf{E}\{\cdot\}=25+1+\frac{1}{2}[J_1(3)+J_1(5)]=51 u1​=−1:E{⋅}=25+1+21​[J1​(3)+J1​(5)]=51 u 0 = 0 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 0 + J 1 ( 5 ) = 67 1 2 u_0=0\;:\;\mathbf{E}\{\cdot\}=25+0+J_1(5)=67\frac{1}{2} u0​=0:E{⋅}=25+0+J1​(5)=6721​So ${\mu }_0^{\ast }(x_0)=-3,J_0(x_0)=38\frac{3}{4}.\square$
PS: In the book, a hint is given : Alternatively, you may use a terminal cost $g_4(x_4)$ equal to a very large number for $x_4̸=5$. Why a very large number ? My caculation is wrong?

1.2 Carry out the calculations needed to verify that $J_0(1)=2.67$ and $J_0(2)=2.608$ in Example 1.3.2.
Solution. The result in the exercise is wrong, even not the same as that in the text.
For $x_0=1$, we have J 0 ( 1 ) = min ⁡ u 0 = 0 , 1 E w 0 = 0 , 1 , 2 { u 0 + ( 1 + u 0 − w 0 ) 2 + J 1 ( max ⁡ ( 0 , 1 + u 0 − w 0 ) ) ) } J_0(1)=\min_{u_0=0,1} \mathop{\bf E}\limits_{w_0=0,1,2}\{u_0+(1+u_0-w_0)^2+J_1(\max(0,1+u_0-w_0)))\} J0​(1)=u0​=0,1min​w0​=0,1,2E​{u0​+(1+u0​−w0​)2+J1​(max(0,1+u0​−w0​)))} u 0 = 0 : &ThickSpace; E ( . ) = E w 0 = 0 , 1 , 2 { ( 1 − w 0 ) 2 + J 1 ( max ⁡ ( 0 , 1 − w 0 ) ) } u_0=0: \;\mathbf{E}(.)=\mathop{\mathbf{E}}\limits_{w_0=0,1,2}\{(1-w_0)^2+J_1(\max(0,1-w_0))\} u0​=0:E(.)=w0​=0,1,2E​{(1−w0​)2+J1​(max(0,1−w0​))}
$=0.1\times [1+J_1(1)]+0.7\times J_1(0)+0.2\times [1+J_1(0)]$
$=0.1\times 2.5+0.7\times 2.5+0.2\times 3.5$
$=2.7$
u 0 = 1 : &ThickSpace; E ( . ) = E w 0 = 0 , 1 , 2 { 1 + ( 2 − w 0 ) 2 + J 1 ( max ⁡ ( 0 , 2 − w 0 ) ) } u_0=1: \;\mathbf{E}(.)=\mathop{\mathbf{E}}\limits_{w_0=0,1,2}\{1+(2-w_0)^2+J_1(\max(0,2-w_0))\} u0​=1:E(.)=w0​=0,1,2E​{1+(2−w0​)2+J1​(max(0,2−w0​))}
$=0.1\times [5+J_1(2)]+0.7\times [2+J_1(1)]+0.2\times [1+J_1(0)]$
$=0.1\times 6.68+0.7\times 3.5+0.2\times 3.5$
$=3.818$
So $J_0(1)=2.7,{\mu }_0^{\ast }(1)=0.$

For $x_0=2$, we have J 0 ( 2 ) = min ⁡ u 0 = 0 E w 0 = 0 , 1 , 2 { u 0 + ( 2 + u 0 − w 0 ) 2 + J 1 ( max ⁡ ( 0 , 2 + u 0 − w 0 ) ) ) } J_0(2)=\min_{u_0=0} \mathop{\bf E}\limits_{w_0=0,1,2}\{u_0+(2+u_0-w_0)^2+J_1(\max(0,2+u_0-w_0)))\} J0​(2)=u0​=0min​w0​=0,1,2E​{u0​+(2+u0​−w0​)2+J1​(max(0,2+u0​−w0​)))} = E w 0 = 0 , 1 , 2 { ( 2 − w 0 ) 2 + J 1 ( max ⁡ ( 0 , 2 − w 0 ) ) } \qquad\qquad\qquad\qquad=\mathop{\mathbf{E}}\limits_{w_0=0,1,2}\{(2-w_0)^2+J_1(\max(0,2-w_0))\} =w0​=0,1,2E​{(2−w0​)2+J1​(max(0,2−w0​))}
$=0.1\times [4+J_1(2)]+0.7\times [1+J_1(1)]+0.2\times J_1(0)$
$=0.1\times 5.68+0.7\times 2.5+0.2\times 2.5$
$=2.818$
So $J_0(2)=2.818,{\mu }_0^{\ast }(2)=0.\square$

1.6 (Discouted Cost per State) In the framework of the basic problem, consider the case where the cost is of the form $$\underset{\underset{k=0,1,...,N-1}{w_k}}{E}\left\{{\alpha }^N{g}_N(x_N)+\sum _{k=0}^{N-1}{\alpha }^k{g}_k(x_k,u_k,w_k)\right\}$$where $\alpha$ is a discount factor with $0<\alpha <1$. Show that an alternate form the DP algorithm is given by $$V_N(x_N)=g_N(x_N)$$$$V_k(x_k)=\underset{u_k\in U_k(x_k)}{\min }\underset{w_k}{E}\left\{g_k(x_k,u_k,w_k)+\alpha V_{k+1}(f_k(x_k,u_k,w_k))\right\}.$$Solution. Using similar argument to Proposition 1.3.1, in the dicounted cost case, the minimizing cost takes the form
$$V^{\ast }(x_0)=\underset{\pi \in \Pi }{\min }\underset{\underset{k=0,1,...,N-1}{w_k}}{E}\left\{{\alpha }^N{g}_N(x_N)+\sum _{k=0}^{N-1}{\alpha }^k{g}_k(x_k,u_k,w_k)\right\}$$ Let $V_k^{\ast }(x_k)$ be the optimal cost for the $(N-k)$-stage problem that starts at state $x_k$ and time $k$, and ends at time $N$.
$$V_k^{\ast }(x_k)=\underset{{\pi }^k}{\min }\underset{w_k,...,w_{N-1}}{E}\left\{{\alpha }^{N-k}{g}_N(x_N)+\sum _{i=k}^{N-1}{\alpha }^{i-k}{g}_i(x_i,u_i,w_i)\right\}$$ Claim: $V_k^{\ast }(x_k)=V_k(x_k).$
We prove this cliam by induction on $k$. Firstly, $V_N^{\ast }(x_N)=g_N(x_N)=V_N(x_N)$. Suppose the claim is true for any $k+1,...,N$, then
$$V_k^{\ast }(x_k)=\underset{{\pi }^k}{\min }\underset{w_k,...,w_{N-1}}{E}\left\{g_k(x_k,u_k,w_k)+{\alpha }^{N-k}{g}_N(x_N)+\sum _{i=k+1}^{N-1}{\alpha }^{i-k}{g}_i(x_i,u_i,w_i)\right\}$$ = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k } + α ⋅ min ⁡ π k E w k , . . . , w N − 1 { α N − k − 1 g N ( x N ) + ∑ i = k + 1 N − 1 α i − k − 1 g i ( x i , u i , w i ) } =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha\cdot\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{w_k,...,w_{N-1}}\left\{\alpha^{N-k-1}g_N(x_N)+\sum_{i=k+1}^{N-1}\alpha^{i-k-1}g_i(x_i,u_i,w_i)\right\} =uk​∈Uk​(xk​)min​wk​E​{gk​(xk​,uk​,wk​}+α⋅πkmin​wk​,...,wN−1​E​{αN−k−1gN​(xN​)+i=k+1∑N−1​αi−k−1gi​(xi​,ui​,wi​)} = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k } + α ⋅ min ⁡ π k + 1 E w k + 1 , . . . , w N − 1 { α N − k − 1 g N ( x N ) + ∑ i = k + 1 N − 1 α i − k − 1 g i ( x i , u i , w i ) } =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha\cdot\mathop{\min}\limits_{\pi^{k+1}}\mathop\mathbf{E}\limits_{w_{k+1},...,w_{N-1}}\left\{\alpha^{N-k-1}g_N(x_N)+\sum_{i=k+1}^{N-1}\alpha^{i-k-1}g_i(x_i,u_i,w_i)\right\} =uk​∈Uk​(xk​)min​wk​E​{gk​(xk​,uk​,wk​}+α⋅πk+1min​wk+1​,...,wN−1​E​{αN−k−1gN​(xN​)+i=k+1∑N−1​αi−k−1gi​(xi​,ui​,wi​)} = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k } + α V k + 1 ∗ ( x k + 1 ) =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha V_{k+1}^*(x_{k+1})\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad =uk​∈Uk​(xk​)min​wk​E​{gk​(xk​,uk​,wk​}+αVk+1∗​(xk+1​) = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k } + α V k + 1 ( x k + 1 ) =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha V_{k+1}(x_{k+1})\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad =uk​∈Uk​(xk​)min​wk​E​{gk​(xk​,uk​,wk​}+αVk+1​(xk+1​) = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k } + α ⋅ min ⁡ u k ∈ U k ( x k ) E w k V k + 1 ( f k ( x k , u k , w k ) ) &ThickSpace; =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha \cdot\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}V_{k+1}(f_k(x_k,u_k,w_k))\qquad\qquad\qquad\qquad\qquad\qquad\; =uk​∈Uk​(xk​)min​wk​E​{gk​(xk​,uk​,wk​}+α⋅uk​∈Uk​(xk​)min​wk​E​Vk+1​(fk​(xk​,uk​,wk​))$$=\underset{u_k\in U_k(x_k)}{\min }\underset{w_k}{E}\left\{g_k(x_k,u_k,w_k)+\alpha V_{k+1}(f_k(x_k,u_k,w_k))\right\}=V_k(x_k)\square$$

1.7 (Exponential Cost Function) In the framework of the basic problem, consider the case where the cost is of the form $$\underset{\underset{k=0,1,...,N-1}{w_k}}{E}\left\{\exp \left(g_N(x_N)+\sum _{k=0}^{N-1}{g}_k(x_k,u_k,w_k)\right)\right\}$$（a）Show that the optimal cost and an optimal policy can be obtained from the DP-like algorithm $$J_N(x_N)=\exp \left(g_N(x_N)\right)$$ J k ( x k ) = min ⁡ u k ∈ U k ( x k ) E w k { J k + 1 ( f k ( x k , u k , w k ) ) exp ⁡ ( g k ( x k , u k , w k ) ) } J_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\left\{J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\exp\left(g_k(x_k,u_k,w_k)\right)\right\} Jk​(xk​)=uk​∈Uk​(xk​)min​wk​E​{Jk+1​(fk​(xk​,uk​,wk​))exp(gk​(xk​,uk​,wk​))} （b）Define the functions $V_k(x_k)=\ln J_k(x_k)$. Assume also that $g_k$ is a function of $x_k$ and $u_k$ only (and not of $w_k$). Show that the above algorithm can be rewritten as $$V_N(x_N)=g_N(x_N)$$ V k ( x k ) = min ⁡ u k ∈ U k ( x k ) { g k ( x k , u k ) + ln ⁡ E w k { exp ⁡ ( V k + 1 ( f k ( x k , u k , w k ) ) ) } } V_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{g_k(x_k,u_k)+\ln \mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(V_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right)\right\}\right\} Vk​(xk​)=uk​∈Uk​(xk​)min​{gk​(xk​,uk​)+lnwk​E​{exp(Vk+1​(fk​(xk​,uk​,wk​)))}}
Solution. (a) Define $$J_k^{\ast }(x_k)=\underset{{\pi }^k}{\min }\underset{\underset{i=k,k+1,...,N-1}{w_i}}{E}\left\{\exp \left(g_N(x_N)+\sum _{i=k}^{N-1}{g}_i(x_i,u_i,w_i)\right)\right\}$$ It is easy to check that $J_k^{\ast }(x_k)=J_k(x_k)$.
J k ∗ ( x k ) = min ⁡ u k E w k { exp ⁡ ( g k ( x k , u k , w k ) ) } min ⁡ π k E w i i = k , k + 1 , . . . , N − 1 { exp ⁡ ( g N ( x N ) + ∑ i = k + 1 N − 1 g i ( x i , u i , w i ) ) } J_k^*(x_k)=\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{\mathop{w_i}\limits_{i=k,k+1,...,N-1}}\left\{\exp\left(g_N(x_N)+\sum_{i=k+1}^{N-1}g_i(x_i,u_i,w_i)\right)\right\} Jk∗​(xk​)=uk​min​wk​E​{exp(gk​(xk​,uk​,wk​))}πkmin​i=k,k+1,...,N−1wi​​E​{exp(gN​(xN​)+i=k+1∑N−1​gi​(xi​,ui​,wi​))} = min ⁡ u k E w k { exp ⁡ ( g k ( x k , u k , w k ) ) } min ⁡ π k + 1 x k + 1 = f k ( x k , u k , w k ) E w i i = k + 1 , . . . . , N − 1 { exp ⁡ ( g N ( x N ) + ∑ i = k + 1 N − 1 g i ( x i , u i , w i ) ) } =\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\mathop{\min}\limits_{\mathop{\pi^{k+1}}\limits_{x_{k+1}=f_k(x_k,u_k,w_k)}}\mathop\mathbf{E}\limits_{\mathop{w_i}\limits_{i=k+1,....,N-1}}\left\{\exp\left(g_N(x_N)+\sum_{i=k+1}^{N-1}g_i(x_i,u_i,w_i)\right)\right\} =uk​min​wk​E​{exp(gk​(xk​,uk​,wk​))}xk+1​=fk​(xk​,uk​,wk​)πk+1​min​i=k+1,....,N−1wi​​E​{exp(gN​(xN​)+i=k+1∑N−1​gi​(xi​,ui​,wi​))} = min ⁡ u k E w k { exp ⁡ ( g k ( x k , u k , w k ) ) } ⋅ min ⁡ u k E w k J k + 1 ∗ ( f k ( x k , u k , w k ) ) ) =\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\cdot \mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}J_{k+1}^*(f_k(x_k,u_k,w_k)))\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad =uk​min​wk​E​{exp(gk​(xk​,uk​,wk​))}⋅uk​min​wk​E​Jk+1∗​(fk​(xk​,uk​,wk​))) = min ⁡ u k E w k { exp ⁡ ( g k ( x k , u k , w k ) ) } ⋅ min ⁡ u k E w k J k + 1 ( f k ( x k , u k , w k ) ) ) = J k ( x k ) =\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\cdot \mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}J_{k+1}(f_k(x_k,u_k,w_k)))=J_k(x_k)\qquad\qquad\qquad\qquad\qquad =uk​min​wk​E​{exp(gk​(xk​,uk​,wk​))}⋅uk​min​wk​E​Jk+1​(fk​(xk​,uk​,wk​)))=Jk​(xk​) (b) Since $g_k$ is a function not of $w_k$, we can rewrite $J_k(x_k)$ as follows: J k ( x k ) = min ⁡ u k ∈ U k ( x k ) { exp ⁡ ( g k ( x k , u k , w k ) ) E w k { J k + 1 ( f k ( x k , u k , w k ) ) } } J_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\mathop\mathbf{E}\limits_{w_k}\left\{J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\}\right\} Jk​(xk​)=uk​∈Uk​(xk​)min​{exp(gk​(xk​,uk​,wk​))wk​E​{Jk+1​(fk​(xk​,uk​,wk​))}} So $$V_k(x_k)=\ln J_k(x_k)$$ = min ⁡ u k ∈ U k ( x k ) { g k ( x k , u k , w k ) + ln ⁡ E w k { J k + 1 ( f k ( x k , u k , w k ) ) } } =\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{g_k(x_k,u_k,w_k)+\ln\mathop\mathbf{E}\limits_{w_k}\left\{J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\}\right\}\qquad\qquad\qquad =uk​∈Uk​(xk​)min​{gk​(xk​,uk​,wk​)+lnwk​E​{Jk+1​(fk​(xk​,uk​,wk​))}} = min ⁡ u k ∈ U k ( x k ) { g k ( x k , u k , w k ) + ln ⁡ E w k { exp ⁡ ( V k + 1 ( f k ( x k , u k , w k ) ) ) } } &ThickSpace; □ =\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{g_k(x_k,u_k,w_k)+\ln\mathop\mathbf{E}\limits_{w_k}\left\{\exp \left(V_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right)\right\}\right\}\qquad\;\Box =uk​∈Uk​(xk​)min​{gk​(xk​,uk​,wk​)+lnwk​E​{exp(Vk+1​(fk​(xk​,uk​,wk​)))}}□

1.8 (Terminating Process) In the framework of the basic problem, consider the case where the system evolution terminates at time $i$ when a given value $\overline{w_i}$ of the disurbance at time $i$ occurs, or when a termination decesion $u_i$ is made by the controller. If termination occurs at time $i$, the resulting cost is $$T+\sum _{k=0}^i{g}_k(x_k,u_k,w_k)$$where $T$ is a termination cost. If the process has not terminated up to the final time $N$, the resulting cost is $g_N(x_N)+{\sum }_{k=0}^{N-1}{g}_k(x_k,u_k,w_k)$. Reformulate the problem into the framework of the basic problem.

1.9 (Multiplicative Cost) In the framework of the basic problem, consider the case where the cost has the multiplicative form E w k k = 0 , 1 , . . . , N − 1 { g N ( x N ) ⋅ g N − 1 ( x N − 1 , u N − 1 , w N − 1 ) … g 0 ( x 0 , u 0 , w 0 ) } \mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,...,N-1}}\left\{g_N(x_N)\cdot g_{N-1}(x_{N-1}, u_{N-1}, w_{N-1})\ldots g_0(x_0,u_0,w_0)\right\} k=0,1,...,N−1wk​​E​{gN​(xN​)⋅gN−1​(xN−1​,uN−1​,wN−1​)…g0​(x0​,u0​,w0​)} Develop a DP-like algorithm for this problem assuming that $g_k(x_k,u_k,w_k)\ge 0$ for all $x_k,u_k,w_k$ and $k$.
Solution. The DP-like algorithm for this problem is as follows:
$$J_N(x_N)=g_N(x_N)$$ J k ( x k ) = E w k { g k ( x k , u k , w k ) J k + 1 ( f k ( x k , u k , w k ) ) } J_k(x_k)=\mathop\mathbf{E}\limits_{w_k}\left\{g_k(x_k,u_k,w_k)J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\} Jk​(xk​)=wk​E​{gk​(xk​,uk​,wk​)Jk+1​(fk​(xk​,uk​,wk​))} The proof is standard as above, we may define
J k ∗ ( x k ) = E w k i = k , . . . , N − 1 { g N ( x N ) … g i ( x i , u i , w i ) … g k ( x k , u k , w k ) } J_k^*(x_k)=\mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{i=k,...,N-1}} \left\{g_N(x_N)\ldots g_i(x_i,u_i,w_i)\ldots g_k(x_k,u_k,w_k) \right\} Jk∗​(xk​)=i=k,...,N−1wk​​E​{gN​(xN​)…gi​(xi​,ui​,wi​)…gk​(xk​,uk​,wk​)} and prove $J_k^{\ast }(x_k)=J_k(x_k)$. $\square$

1.10 Assume that we have a vessel whose maximum weight capacity is $z$ and whose cargo is to consist of different quantities of $N$ different items. Let $v_i$ denote the value of the $i$-th type of item, $w_i$ the weight of $i$-th type of item, and $x_i$ the number of items of type $i$ that are loaded in the vessel. The problem is to find the most valuable cargo, i.e., to maximize ${\sum }_{i=1}^N{x}_i{v}_i$ subject to the constraints ${\sum }_{i=1}^N{x}_i{w}_i\le z$ and $x_i=0,1,2,\dots$ Formulate this problem in terms of DP.

Solution. We formulate this problem in terms of DP as follows:
\qquad a state $x_k^{\prime }$ is defined by $x_k^{\prime }=\sum _{i=0}^{k-1}{x}_i{w}_i$
\qquad an action $u_k^{\prime }$ is taken on the space $U_k(x_k^{\prime })=\left\{0,1,\dots ,\left[\frac{z-x_k^{\prime }}{w_k}\right]\right\}$
\qquad a random parameter $w_k^{\prime }$ is always be 0
\qquad state transform function is defined as $x_{k+1}^{\prime }=f_k(x_k^{\prime },u_k^{\prime },w_k^{\prime })=x_k^{\prime }+u_k^{\prime }{w}_k$
\qquad cost function is defined as $g_k(x_k^{\prime },u_k^{\prime },w_k^{\prime })=-u_k^{\prime }{v}_k$
So the problem is reformulated to get an optimal solution to the minimization problem min ⁡ π = { u 1 ′ , … , u N ′ } g k ( x k ′ , u k ′ , w k ′ ) \mathop{\min}\limits_{\pi=\{u&#x27;_1,\ldots,u&#x27;_N\}}g_k(x&#x27;_k,u&#x27;_k,w&#x27;_k) π={u1′​,…,uN′​}min​gk​(xk′​,uk′​,wk′​)the problem can be solved by the following DP-algorithm:
$$J_{N+1}(x_{N+1}^{\prime })=0$$ J k ( x k ′ ) = min ⁡ u k ′ ∈ U k ( x k ′ ) { − u k ′ v k + J k + 1 ( f k ( x k ′ , u k ′ , w k ′ ) ) } &ThickSpace; , k = 1 , 2 , … , N J_k(x&#x27;_k)=\mathop{\min}\limits_{u&#x27;_k\in U_k(x&#x27;_k)}\{-u&#x27;_kv_k+J_{k+1}(f_k(x&#x27;_k,u&#x27;_k,w&#x27;_k))\}\;,\qquad k=1,2,\ldots,N Jk​(xk′​)=uk′​∈Uk​(xk′​)min​{−uk′​vk​+Jk+1​(fk​(xk′​,uk′​,wk′​))},k=1,2,…,N$$\square$$

1.11 Consider a device consisting of $N$ stages connected in series, where each stage consists of a particular component. The components are subject to failure, and to increase the reliablity of the device duplicate components are provided. For $j=1,2,\dots ,N$, let $(1+m_j)$ be the number of components for the $j$-th stage, let $p_j(m_j)$ be the probability of success operation of the $j$-th stage when $(1+m_j)$ components are used, and let $c_j$ denote the cost of a single component at the $j$-th stage. Formulate in terms of DP the problem of finding the number of components at each stage that maximize the realibility of the device expressed by $$p_1(m_1)\cdot p_2(m_2)\dots p_N(m_N)$$ subject to the cost constraint ${\sum }_{j=1}^N{c}_j{m}_j\le A$, where $A>a$ is given.

1.12 (Minimization over a Subset of Policies) This problem is primarily of theoretical interest. Consider a variant of the basic problem whereby we seek $$\underset{\pi \in \tilde{\Pi }}{\min }{J}_k(x_0)$$ where $\tilde{\Pi }$ is some given subset of the set of sequences { μ 0 , μ 1 , … , μ N − 1 } \{\mu_0,\mu_1,\ldots, \mu_{N-1}\} {μ0​,μ1​,…,μN−1​} of functions ${\mu }_k:S_k\to C_k$ with ${\mu }_k(x_k)\in U_k(x_k)$ for all $x_k\in S_k$. Assume that π ∗ = { μ 0 ∗ , μ 1 ∗ , … , μ N − 1 ∗ } \pi^*=\{\mu_0^*,\mu_1^*,\ldots, \mu_{N-1}^*\} π∗={μ0∗​,μ1∗​,…,μN−1∗​} belongs to $\tilde{\Pi }$ and attains the minimum in the DP algorithm; that is, for all $k=0,1,\dots ,N-1$ and $x_k\in S_k$, J k ( x k ) = E w k { g k ( x k , μ k ∗ ( x k ) , w k ) + J k + 1 ( f k ( x k , μ k ∗ ( x k ) , w k ) ) } J_k(x_k)=\mathop{\mathbf{E}}\limits_{w_k}\left\{g_k(x_k,\mu_k^*(x_k),w_k)+J_{k+1}\left(f_k(x_k,\mu_k^*(x_k),w_k)\right)\right\} Jk​(xk​)=wk​E​{gk​(xk​,μk∗​(xk​),wk​)+Jk+1​(fk​(xk​,μk∗​(xk​),wk​))} min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k ) + J k + 1 ( f k ( x k , u k , w k ) ) } \mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop{\mathbf{E}}\limits_{w_k}\left\{g_k(x_k,u_k,w_k)+J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\} uk​∈Uk​(xk​)min​wk​E​{gk​(xk​,uk​,wk​)+Jk+1​(fk​(xk​,uk​,wk​))} with $J_N(x_N)=g_N(x_N)$. Assume further that the function $J_k$ are real-valued and that the preceding expectations are well-defined and finite. Show that ${\pi }^{\ast }$ is optimal within $\tilde{\Pi }$ and that the corresponding optimal cost is equal to $J_0(x_0)$.

1.13 (Semilinear Systems) Consider a problem involving the system $$x_{k+1}=A_k{x}_k+f_k(u_k)+w_k$$ where $x_k\in {\mathfrak{R}}^n$, $f_k$ are given functions, and $A_k$ and $w_k$ are random $n\times n$ matices and $n$-vectors, respectively, with given probability distributions that do not depend on $x_k,u_k$ or prior values of $A_k$ and $w_k$. Assume that the cost is of the form $$\underset{\underset{k=0,1,\dots ,N-1}{A_k,w_k}}{E}\left\{c_N^{\prime }{x}_n+\sum _{k=0}^{N-1}\left(c_k^{\prime }{x}_k+g_k\left({\mu }_k(x_k)\right)\right)\right\}$$where $c_k$ are given vectors and $g_k$ are given functions. Show that if the optimal cost for this problem is finite and the control constraint sets $U_k(x_k)$ are independent of $x_k$, then the cost-to-go functions of the DP algorithm are affine (linear plus constraint). Assuming that there is at least one optimal policy, show that there exists an optimal policy that consist of constant functions ${\mu }_k^{\ast }$ that is ${\mu }_k^{\ast }(x_k)=$ constant for all $x_k\in {\mathfrak{R}}^n$.

1.14 A farmer annually producing $x_k$ units of a certain crop stores $(1-u_k)x_k$ units of his production, where $0\le u_k\le 1$ and invests the remaining $u_k{x}_k$ units, thus increasing the next year’s production to a level $x_{k+1}$ given by $$x_{k+1}=x_k+w_k{u}_k{x}_k,k=0,1,\cdots ,N-1$$The scalars $w_k$ are independent random variables with identical probability distributions that do not depend either on $x_k$ or $u_k$. Furthermore, E { w k } = w ‾ &gt; 0 \mathbf{E}\{w_k\}=\overline{w}&gt;0 E{wk​}=w>0. The problem is to find the optimal investment policy that maximizes the total expected product stored over $N$ years $$\underset{\underset{k=0,1,\dots ,N-1}{w_k}}{E}\left\{x_N+\sum _{k=0}^{N-1}(1-u_k)x_k\right\}$$ Show the optimality of the following policy that consists of constant functions:
(1) If $\overline{w}>1$, ${\mu }_0^{\ast }(x_0)=\cdots ={\mu }_{N-1}^{\ast }(x_{N-1})=1$.
(2) If $0<\overline{w}<1/N$, ${\mu }_0^{\ast }(x_0)=\cdots ={\mu }_{N-1}^{\ast }(x_{N-1})=0$.
(3) If $1/N\le \overline{w}\le 1$, $${\mu }_0^{\ast }(x_0)=\cdots ={\mu }_{N-\bar{k}-1}^{\ast }(x_{N-\bar{k}-1})=1$$$${\mu }_{N-\bar{k}}^{\ast }(x_{N-\bar{k}})=\cdots ={\mu }_{N-1}^{\ast }(x_{N-1})=0$$ where $\bar{k}$ is such that $1/(\bar{k}+1)<\overline{w}\le 1/\bar{k}$.
Solution. The DP-algorithm can be formed as follows: $$J_N(x_N)=x_N$$ J k ( x k ) = max ⁡ u k E w k { ( 1 − u k ) x k + J k + 1 ( x k + w k u k x k ) } &ThinSpace; , &ThickSpace;&ThickSpace; k = 0 , 1 , ⋯ &ThinSpace; , N − 1 J_k(x_k)=\mathop{\max}\limits_{u_k}\mathop{\mathbf E}\limits_{w_k}\left\{(1-u_k)x_k+J_{k+1}(x_k+w_ku_kx_k)\right\}\,,\;\; k=0,1,\cdots,N-1 Jk​(xk​)=uk​max​wk​E​{(1−uk​)xk​+Jk+1​(xk​+wk​uk​xk​)},k=0,1,⋯,N−1 (1) In the case $\overline{w}>1$ J N − 1 ( x N − 1 ) = max ⁡ u N − 1 E w N − 1 { ( 1 − u N − 1 ) x N − 1 + x N − 1 + w N − 1 u N − 1 x N − 1 } J_{N-1}(x_{N-1})=\mathop{\max}\limits_{u_{N-1}}\mathop{\mathbf E}\limits_{w_{N-1}}\left\{(1-u_{N-1})x_{N-1}+x_{N-1}+w_{N-1}u_{N-1}x_{N-1}\right\} JN−1​(xN−1​)=uN−1​max​wN−1​E​{(1−uN−1​)xN−1​+xN−1​+wN−1​uN−1​xN−1​} = max ⁡ u N − 1 E w N − 1 { 2 x N − 1 + ( w N − 1 − 1 ) u N − 1 x N − 1 } =\mathop{\max}\limits_{u_{N-1}}\mathop{\mathbf E}\limits_{w_{N-1}}\left\{2x_{N-1}+(w_{N-1}-1)u_{N-1}x_{N-1}\right\} =uN−1​max​wN−1​E​{2xN−1​+(wN−1​−1)uN−1​xN−1​} = max ⁡ u N − 1 { 2 x N − 1 + ( w ‾ − 1 ) u N − 1 x N − 1 } =\mathop{\max}\limits_{u_{N-1}}\left\{2x_{N-1}+(\overline{w}-1)u_{N-1}x_{N-1}\right\}\qquad\qquad =uN−1​max​{2xN−1​+(w−1)uN−1​xN−1​} we obtain that ${\mu }_{N-1}^{\ast }(x_{N-1})=1$ and $J_{N-1}(x_{N-1})=(\overline{w}+1)x_{N-1}$. By indunction and similar caculations, we can get $${\mu }_k^{\ast }(x_k)=1,J_k(x_k)=(\overline{w}+N-k)x_k\text{for all }k=0,1,\cdots ,N-1$$(2) In the case $\overline{w}<1/N$,
J N − 1 ( x N − 1 ) = max ⁡ u N − 1 { 2 x N − 1 + ( w ‾ − 1 ) u N − 1 x N − 1 } = 2 x N − 1 J_{N-1}(x_{N-1})=\mathop{\max}\limits_{u_{N-1}}\left\{2x_{N-1}+(\overline{w}-1)u_{N-1}x_{N-1}\right\}=2x_{N-1} JN−1​(xN−1​)=uN−1​max​{2xN−1​+(w−1)uN−1​xN−1​}=2xN−1​ with ${\mu }_{N-1}^{\ast }(x_{N-1})=0$. And then
J N − 2 ( x N − 2 ) = max ⁡ u N − 2 E w N − 2 { ( 1 − u N − 2 ) x N − 2 + 2 [ x N − 2 + w N − 2 u N − 2 x N − 2 ] } J_{N-2}(x_{N-2})=\mathop{\max}\limits_{u_{N-2}}\mathop{\mathbf E}\limits_{w_{N-2}}\left\{(1-u_{N-2})x_{N-2}+2[x_{N-2}+w_{N-2}u_{N-2}x_{N-2}]\right\} JN−2​(xN−2​)=uN−2​max​wN−2​E​{(1−uN−2​)xN−2​+2[xN−2​+wN−2​uN−2​xN−2​]} = max ⁡ u N − 2 { 3 x N − 2 + ( 2 w ‾ − 1 ) u N − 2 x N − 2 } = 3 x N − 2 =\mathop{\max}\limits_{u_{N-2}}\left\{3x_{N-2}+(2\overline{w}-1)u_{N-2}x_{N-2}\right\}=3x_{N-2} =uN−2​max​{3xN−2​+(2w−1)uN−2​xN−2​}=3xN−2​ with ${\mu }_{N-2}^{\ast }(x_{N-2})=0$. Since $\overline{w}<1/N$, we can prove by induction that for all $k=0,1,\cdots ,N-1$, $${\mu }_k^{\ast }(x_k)=0,J_k(x_k)=(N-k+1)x_k$$ (3) In the case $1/N\le \overline{w}\le 1$, there is some $k=\bar{k}$ satisfying $1/(\bar{k}+1)<\overline{w}\le 1/\bar{k}$, after which step the induction argument in (2) will stop.
For any $k\ge N-\bar{k}$, we have J k ( x k ) = max ⁡ u k { ( N − k + 1 ) x k + ( ( N − k ) w ‾ − 1 ) u k x k } J_k(x_k)=\mathop{\max}\limits_{u_k}\left\{(N-k+1)x_k+((N-k)\overline{w}-1)u_kx_k\right\} Jk​(xk​)=uk​max​{(N−k+1)xk​+((N−k)w−1)uk​xk​}Since $(N-k)\overline{w}\le \bar{k}\overline{w}\le 1$, then ${\mu }_k^{\ast }(x_k)=0$ and $J_k(x_k)=(N-k+1)x_k$.
Next for $k=N-\bar{k}-1$
$$J_{N-\bar{k}-1}(x_{N-\bar{k}-1})=\underset{u_{N-\bar{k}-1}}{\max }\left\{(\bar{k}+2)x_{N-\bar{k}-1}+((\bar{k}+1)\overline{w}-1)u_{N-\bar{k}-1}{x}_{N-\bar{k}-1}\right\}$$ In this case, $(\bar{k}+1)\overline{w}-1>0$, yielding ${\mu }_{N-\bar{k}-1}^{\ast }(x_{N-\bar{k}-1})=1$ and $J_{N-\bar{k}-1}(x_{N-\bar{k}-1})=[(\bar{k}+1)\overline{w}+\bar{k}+1]x_{N-\bar{k}-1}$.
And then by induction, we can prove that for all $k<N-\bar{k}-1$, ${\mu }_k^{\ast }(x_k)=1$.$\square$

1.15 Let $x_k$ denote the number of educators in a certain country at time $k$ and let $y_k$ denote the number of research scientists at time $k$. New scientists (potential educators or research scientists) are produced during the $k$th period by educators at a rate ${\gamma }_k$ per educator, while educators and research scientists leave the field due to death, retirement, and transfer at a rate ${\delta }_k$. The scalars ${\gamma }_k$, $k=0,1,...,N-1$, are independent identically distributed random variables taking values within a closed and bounded interval of positive numbers. Similarly ${\delta }_k$, $k=0,1,...,N-1$, are independent identically distributed and take values in an interval $[\delta ,{\delta }^{\prime }]$ with $0<\delta \le {\delta }^{\prime }<1$. By means of incentives, a science policy maker can determine the proportion $u_k$ of new scientists produced at time $k$ who become educators. Thus, the number of research scientists and educators evolves according to the equations $$x_{k+1}=(1-{\delta }_k)x_k+u_k{\gamma }_k{x}_k$$ $$y_{k+1}=(1-{\delta }_k)y_k+(1-u_k){\gamma }_k{x}_k$$ The initial numbers $x_0,y_0$ are known, and it is required to find a policy { μ 0 ∗ ( x 0 , y 0 ) , ⋯ &ThinSpace; , μ N − 1 ∗ ( x N − 1 , y N − 1 ) } \{\mu_0^*(x_0,y_0),\cdots,\mu_{N-1}^*(x_{N-1},y_{N-1})\} {μ0∗​(x0​,y0​),⋯,μN−1∗​(xN−1​,yN−1​)} with $$0<\alpha \le {\mu }_k^{\ast }(x_k,y_k)\le \beta <1,\text{for all }{x}_k,y_k\text{ and }k$$ which maximizes E γ k , δ k { y N } \mathbf{E}_{\gamma_k,\delta_k}\{y_N\} Eγk​,δk​​{yN​} (i.e. the expected final number of research scientists after $N$ periods). The scalars $\alpha$ and $\beta$ are given.
(1) Show that the cost-to-go functions $J_k(x_k,y_k)$ are linear; that is, for some scalars ${\xi }_k,{\zeta }_k,$ $$J_k(x_k,y_k)={\xi }_k{x}_k+{\zeta }_k{y}_k$$ (2) Derive an optimal policy { μ 0 ∗ , ⋯ &ThinSpace; , μ N − 1 ∗ } \{\mu_0^*,\cdots,\mu_{N-1}^*\} {μ0∗​,⋯,μN−1∗​}, under the assumption E { γ k } &gt; E { δ k } \mathbf{E}\{\gamma_k\}&gt;\mathbf{E}\{\delta_k\} E{γk​}>E{δk​} and show that this optimal policy can consist of constant functions.
(3) Assume that the proportion of new scientists who become educators at time $k$ is $u_k+{ϵ}_k$ (rather than $u_k$), where ${ϵ}_k$ are identically distributed independent random variables that are also independent of ${\gamma }_k,{\delta }_k$ and take values in the interval $[-\alpha ,1-\beta ]$. Derive the form of the cost-to-go functions and the optimal policy.