Dynamic Programming and Optimal Control 第一章习题

1.1 Consider the system x k + 1 = x k + u k + w k , k = 0 , 1 , 2 , 3 x_{k+1}=x_k+u_k+w_k,\qquad k=0,1,2,3 xk+1=xk+uk+wk,k=0,1,2,3with initial state x 0 = 5 x_0=5 x0=5, and the cost function ∑ k = 0 3 ( x k 2 + u k 2 ) \sum_{k=0}^3(x_k^2+u_k^2) k=03(xk2+uk2)Apply the DP algorithm for the following three cases:
(a)The control constraint set U k ( x k ) U_k(x_k) Uk(xk) is { u ∣ 0 ≤ x k + u ≤ 5 , u : integer } \{u|0\leq x_k+u\leq 5, u \text{: integer}\} {u0xk+u5,u: integer} for all x k x_k xk and k k k, and the disturbance w k w_k wk is equal to zero for all k k k.
(b) The control constraint and the disturbance w k w_k wk are as in (a), but there is in addition a constraint x 4 = 5 x_4=5 x4=5 on the final state.
(c)The control constraint is as in part (a) and the disturbance w k w_k wk takes the values − 1 -1 1 and 1 1 1 with equal probability 1 / 2 1/2 1/2 for all x k x_k xk and u k u_k uk, except if x k + u k x_k+u_k xk+uk is equal to 0 0 0 or 5 5 5, in which case w k = 0 w_k=0 wk=0 with probability 1 1 1.

Solution.
(a) N = 4 N=4 N=4, we assume that J 4 ( x 4 ) = g 4 ( x 4 ) = 0 J_4(x_4)=g_4(x_4)=0 J4(x4)=g4(x4)=0. As the condition, the system equation takes the form f k + 1 = x k + u k + w k , k = 0 , 1 , 2 , 3 f_{k+1}=x_k+u_k+w_k,\qquad k=0,1,2,3 fk+1=xk+uk+wk,k=0,1,2,3and the cost function is g k = x k 2 + u k 2 g_k=x_k^2+u_k^2 gk=xk2+uk2In this case, w k = 0 w_k=0 wk=0 for all k k k, so J 3 ( x 3 ) = min ⁡ u 3 ∈ U 3 ( x 3 ) E w 3 { g 3 ( x 3 , u 3 , w 3 ) + J 4 ( f 3 ( x 3 , u 3 , w 3 ) ) } J_3(x_3)=\mathop{\min}\limits_{u_3\in U_3(x_3)} \mathop\mathbf{E}\limits_{w_3}\{g_3(x_3,u_3,w_3)+J_4(f_3(x_3,u_3,w_3))\} J3(x3)=u3U3(x3)minw3E{g3(x3,u3,w3)+J4(f3(x3,u3,w3))} = min ⁡ u 3 = − x 3 , 1 − x 3 , . . . , 5 − x 3 ( x 3 2 + u 3 2 ) = x 3 2 =\mathop{\min}\limits_{u_3=-x_3,1-x_3,...,5-x_3}(x_3^2+u_3^2)=x_3^2\qquad\qquad\qquad =u3=x3,1x3,...,5x3min(x32+u32)=x32i.e. J 3 ( x 3 ) = x 3 2 , μ 3 ∗ ( x 3 ) = 0. ( 1 ) \qquad J_3(x_3)=x_3^2, \mu_3^*(x_3)=0.\qquad\qquad\qquad\qquad\qquad\qquad (1) J3(x3)=x32,μ3(x3)=0.(1)
Then J 2 ( x 2 ) = min ⁡ u 2 = − x 2 , 1 − x 2 , . . . , 5 − x 2 ( x 2 2 + u 2 2 + ( x 2 + u 2 ) 2 ) J_2(x_2)=\mathop{\min}\limits_{u_2=-x_2,1-x_2,...,5-x_2}(x_2^2+u_2^2+(x_2+u_2)^2) J2(x2)=u2=x2,1x2,...,5x2min(x22+u22+(x2+u2)2)Let h x 2 ( u 2 ) = x 2 2 + u 2 2 + ( x 2 + u 2 ) 2 h_{x_2}(u_2)=x_2^2+u_2^2+(x_2+u_2)^2 hx2(u2)=x22+u22+(x2+u2)2, we have h x 2 ′ ( u 2 ) = 4 u 2 + 2 x 2 . h'_{x_2}(u_2)=4u_2+2x_2. hx2(u2)=4u2+2x2.

If x 2 &gt; 10 x_2&gt;10 x2>10, h x 2 ( u 2 ) = 4 ( u 2 + x 2 ) − 2 x 2 &lt; 4 ( u 2 + x 2 ) − 20 &lt; 0 h_{x_2}(u_2)=4(u_2+x_2)-2x_2 &lt; 4(u_2+x_2)-20 &lt; 0 hx2(u2)=4(u2+x2)2x2<4(u2+x2)20<0, then μ x ∗ ( x 2 ) = 5 − x 2 &ThickSpace; , &ThickSpace; J x ( x 2 ) = x 2 2 + ( 5 − x 2 ) 2 + 5 2 = 2 x 2 2 − 10 x 2 + 50 \mu_x^*(x_2)=5-x_2\;,\;J_x(x_2)=x_2^2+(5-x_2)^2+5^2=2x_2^2-10x_2+50 μx(x2)=5x2,Jx(x2)=x22+(5x2)2+52=2x2210x2+50

If x 2 &lt; 0 x_2&lt;0 x2<0, h x 2 ( u 2 ) = 4 ( u 2 + x 2 ) − 2 x 2 &gt; − 2 x 2 &gt; 0 h_{x_2}(u_2)=4(u_2+x_2)-2x_2&gt;-2x_2&gt;0 hx2(u2)=4(u2+x2)2x2>2x2>0, then μ x 2 ( x 2 ) = − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 \mu_x^2(x_2)=-x_2\;,\;J_2(x_2)=2x_2^2 μx2(x2)=x2,J2(x2)=2x22

If 0 ≤ x 2 ≤ 10 0\leq x_2\leq 10 0x210 and x 2 x_2 x2 is even, then μ 2 ∗ ( x 2 ) = − x 2 2 \mu_2^*(x_2)=-\frac{x_2}{2} μ2(x2)=2x2 and J 2 ( x 2 ) = 3 2 x 2 2 . J_2(x_2) = \frac{3}{2}x_2^2. J2(x2)=23x22.

If 0 ≤ x 2 ≤ 10 0\leq x_2\leq 10 0x210 and x 2 x_2 x2 is odd, then μ 2 ∗ ( x 2 ) = − x 2 − 1 2 &ThickSpace; or &ThickSpace; − x 2 + 1 2 \mu_2^*(x_2)=-\frac{x_2-1}{2}\;\text{or}\;-\frac{x_2+1}{2} μ2(x2)=2x21or2x2+1 and J 2 ( x 2 ) = 3 2 x 2 2 + 1 2 . J_2(x_2) = \frac{3}{2}x_2^2+\frac{1}{2}. J2(x2)=23x22+21.

So
{ μ x 2 ( x 2 ) = − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 if  &ThickSpace; x 2 &lt; 0 μ 2 ∗ ( x 2 ) = − x 2 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 3 2 x 2 2 if  &ThickSpace; 0 ≤ x 2 ≤ 10 &ThickSpace; and &ThickSpace; x 2 &ThickSpace; is even μ 2 ∗ ( x 2 ) = − x 2 − 1 2 &ThickSpace; or &ThickSpace; − x 2 + 1 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 3 2 x 2 2 + 1 2 if  &ThickSpace; 0 ≤ x 2 ≤ 10 &ThickSpace; and &ThickSpace; x 2 &ThickSpace; is odd μ x ∗ ( x 2 ) = 5 − x 2 &ThickSpace; , &ThickSpace; J x ( x 2 ) = x 2 2 + ( 5 − x 2 ) 2 + 5 2 = 2 x 2 2 − 10 x 2 + 50 if  &ThickSpace; x 2 &gt; 10 \begin{cases} \mu_x^2(x_2)=-x_2\;,\;J_2(x_2)=2x_2^2 &amp; \qquad\text{if }\; x_2&lt;0 \\ \mu_2^*(x_2)=-\frac{x_2}{2}\;,\;J_2(x_2) = \frac{3}{2}x_2^2 &amp; \qquad\text{if }\;0\leq x_2\leq 10 \;\text{and} \;x_2\; \text{is even}\\ \mu_2^*(x_2)=-\frac{x_2-1}{2}\;\text{or}\;-\frac{x_2+1}{2}\;,\;J_2(x_2) = \frac{3}{2}x_2^2+\frac{1}{2} &amp; \qquad\text{if }\;0\leq x_2\leq 10 \;\text{and} \;x_2\; \text{is odd}\\ \mu_x^*(x_2)=5-x_2\;,\;J_x(x_2)=x_2^2+(5-x_2)^2+5^2=2x_2^2-10x_2+50 &amp; \qquad\text{if }\; x_2&gt;10 \end{cases} μx2(x2)=x2,J2(x2)=2x22μ2(x2)=2x2,J2(x2)=23x22μ2(x2)=2x21or2x2+1,J2(x2)=23x22+21μx(x2)=5x2,Jx(x2)=x22+(5x2)2+52=2x2210x2+50if x2<0if 0x210andx2is evenif 0x210andx2is oddif x2>10

In case k = 1 k=1 k=1
J 1 ( x 1 ) = min ⁡ u 1 = − x 1 , 1 − x 1 , . . . , 5 − x 1 ( x 1 2 + u 1 2 + J 2 ( x 1 + u 1 ) ) &ThickSpace;&ThickSpace;&ThickSpace; J_1(x_1)=\mathop{\min}\limits_{u_1=-x_1,1-x_1,...,5-x_1}(x_1^2+u_1^2+J_2(x_1+u_1))\qquad\qquad\qquad\qquad\qquad\;\;\; J1(x1)=u1=x1,1x1,...,5x1min(x12+u12+J2(x1+u1)) = min ⁡ u 1 = − x 1 , 1 − x 1 , . . . , 5 − x 1 ( x 1 2 + u 1 2 + 3 2 ( x 1 + u 1 ) 2 + 1 2 ⋅ 1 x 1 + u 1 &ThickSpace; is odd ) =\mathop{\min}\limits_{u_1=-x_1,1-x_1,...,5-x_1}\big(x_1^2+u_1^2+\frac{3}{2}(x_1+u_1)^2+\frac{1}{2}\cdot\mathbf{1}_{x_1+u_1\;\text{is odd}}\big) =u1=x1,1x1,...,5x1min(x12+u12+23(x1+u1)2+211x1+u1is odd) In this case, we give an caculation in a direct way:

Let h x 1 ( u 1 ) = x 1 2 + u 1 2 + 3 2 ( x 1 + u 1 ) 2 + 1 2 ⋅ 1 x 1 + u 1 &ThickSpace; is odd h_{x_1}(u_1)=x_1^2+u_1^2+\frac{3}{2}(x_1+u_1)^2+\frac{1}{2}\cdot\mathbf{1}_{x_1+u_1\;\text{is odd}} hx1(u1)=x12+u12+23(x1+u1)2+211x1+u1is odd
h x 1 ( − x 1 ) = 2 x 1 2 h_{x_1}(-x_1)=2x_1^2 hx1(x1)=2x12 h x 1 ( 1 − x 1 ) = 2 x 1 2 − 2 x 1 + 3 h_{x_1}(1-x_1)=2x_1^2-2x_1+3 hx1(1x1)=2x122x1+3 h x 1 ( 2 − x 1 ) = 2 x 1 2 − 4 x 1 + 10 h_{x_1}(2-x_1)=2x_1^2-4x_1+10 hx1(2x1)=2x124x1+10 h x 1 ( 3 − x 1 ) = 2 x 1 2 − 6 x 1 + 23 h_{x_1}(3-x_1)=2x_1^2-6x_1+23 hx1(3x1)=2x126x1+23 h x 1 ( 4 − x 1 ) = 2 x 1 2 − 8 x 1 + 40 h_{x_1}(4-x_1)=2x_1^2-8x_1+40 hx1(4x1)=2x128x1+40 h x 1 ( 5 − x 1 ) = 2 x 1 2 − 10 x 1 + 63 h_{x_1}(5-x_1)=2x_1^2-10x_1+63 hx1(5x1)=2x1210x1+63Then we can get the following result easily
{ μ 1 ∗ ( x 1 ) = − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 if  &ThickSpace; x 1 ≤ 3 2 μ 1 ∗ ( x 1 ) = 1 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 2 x 1 + 3 if  &ThickSpace; 3 2 &lt; x 1 ≤ 7 2 μ 1 ∗ ( x 1 ) = 2 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 4 x 1 + 10 if  &ThickSpace; 7 2 &lt; x 1 ≤ 13 2 μ 1 ∗ ( x 1 ) = 3 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 6 x 1 + 23 if  &ThickSpace; 13 2 &lt; x 1 ≤ 17 2 μ 1 ∗ ( x 1 ) = 4 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 8 x 1 + 40 if  &ThickSpace; 17 2 &lt; x 1 ≤ 23 2 μ 1 ∗ ( x 1 ) = 5 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 10 x 1 + 63 if  &ThickSpace; x 1 &gt; 23 2 \begin{cases} \mu_1^*(x_1)=-x_1\;,\;J_1(x_1)=2x_1^2 &amp; \qquad\text{if }\; x_1\leq\frac{3}{2} \\ \mu_1^*(x_1)=1-x_1\;,\;J_1(x_1)=2x_1^2-2x_1+3 &amp; \qquad\text{if }\; \frac{3}{2}&lt;x_1\leq\frac{7}{2} \\ \mu_1^*(x_1)=2-x_1\;,\;J_1(x_1)=2x_1^2-4x_1+10 &amp; \qquad\text{if }\; \frac{7}{2}&lt;x_1\leq\frac{13}{2} \\ \mu_1^*(x_1)=3-x_1\;,\;J_1(x_1)=2x_1^2-6x_1+23 &amp; \qquad\text{if }\; \frac{13}{2}&lt;x_1\leq\frac{17}{2} \\ \mu_1^*(x_1)=4-x_1\;,\;J_1(x_1)=2x_1^2-8x_1+40 &amp; \qquad\text{if }\; \frac{17}{2}&lt;x_1\leq\frac{23}{2} \\ \mu_1^*(x_1)=5-x_1\;,\;J_1(x_1)=2x_1^2-10x_1+63 &amp; \qquad\text{if }\; x_1 &gt;\frac{23}{2} \\ \end{cases} μ1(x1)=x1,J1(x1)=2x12μ1(x1)=1x1,J1(x1)=2x122x1+3μ1(x1)=2x1,J1(x1)=2x124x1+10μ1(x1)=3x1,J1(x1)=2x126x1+23μ1(x1)=4x1,J1(x1)=2x128x1+40μ1(x1)=5x1,J1(x1)=2x1210x1+63if x123if 23<x127if 27<x1213if 213<x1217if 217<x1223if x1>223

For case k = 0 &ThinSpace; , &ThickSpace; x 0 = 5 k=0\,,\; x_0=5 k=0,x0=5, J 0 ( x 0 ) = min ⁡ u 0 = − x 0 , 1 − x 0 , . . . , 5 − x 0 ( x 0 2 + u 0 2 + J 1 ( x 0 + u 0 ) ) = min ⁡ u 0 = − 5 , − 4 , . . . , 0 ( 25 + u 0 2 + J 1 ( 5 + u 0 ) ) J_0(x_0)=\mathop{\min}\limits_{u_0=-x_0,1-x_0,...,5-x_0}(x_0^2+u_0^2+J_1(x_0+u_0))=\mathop{\min}\limits_{u_0=-5,-4,...,0}(25+u_0^2+J_1(5+u_0)) J0(x0)=u0=x0,1x0,...,5x0min(x02+u02+J1(x0+u0))=u0=5,4,...,0min(25+u02+J1(5+u0))Let h 0 ( u 0 ) = 25 + u 0 2 + J 1 ( 5 + u 0 ) h_0(u_0)=25+u_0^2+J_1(5+u_0) h0(u0)=25+u02+J1(5+u0)we have
h 0 ( − 5 ) = 50 , &ThickSpace; h 0 ( − 4 ) = 43 , &ThickSpace; h 0 ( − 3 ) = 41 , &ThickSpace; h 0 ( − 2 ) = 44 , &ThickSpace; h 0 ( − 1 ) = 50 , &ThickSpace; h 0 ( 0 ) = 65 h_0(-5)=50,\; h_0(-4)=43,\;h_0(-3)=41,\;h_0(-2)=44,\;h_0(-1)=50,\;h_0(0)=65 h0(5)=50,h0(4)=43,h0(3)=41,h0(2)=44,h0(1)=50,h0(0)=65So μ 0 ∗ ( x 0 ) = − 3 \mu_0^*(x_0)=-3 μ0(x0)=3 and J 0 ( x 0 ) = 41 J_0(x_0)=41 J0(x0)=41.

(b) In this case ,we still asume that J 4 ( x 4 ) = g 4 ( x 4 ) = 0 J_4(x_4)=g_4(x_4)=0 J4(x4)=g4(x4)=0. By the additional condition, U 3 ( x 3 ) = { 5 − x 3 } U_3(x_3)=\{5-x_3\} U3(x3)={5x3}, and
J 3 ( x 3 ) = min ⁡ u 3 ∈ U 3 ( x 3 ) ( x 3 2 + u 3 2 ) = x 3 2 + ( 5 − x 3 ) 2 J_3(x_3)=\mathop{\min}\limits_{u_3\in U_3(x_3)}(x_3^2+u_3^2)=x_3^2+(5-x_3)^2 J3(x3)=u3U3(x3)min(x32+u32)=x32+(5x3)2Then J 2 ( x 2 ) = min ⁡ u 2 = − x 2 , 1 − x 2 , . . . , 5 − x 2 ( x 2 2 + u 2 2 + ( 5 − x 2 − u 2 ) 2 ) J_2(x_2)=\mathop{\min}\limits_{u_2=-x_2,1-x_2,...,5-x_2}(x_2^2+u_2^2+(5-x_2-u_2)^2) J2(x2)=u2=x2,1x2,...,5x2min(x22+u22+(5x2u2)2)Let h x 2 ( u 2 ) = x 2 2 + u 2 2 + ( 5 − x 2 − u 2 ) 2 h_{x_2}(u_2)=x_2^2+u_2^2+(5-x_2-u_2)^2 hx2(u2)=x22+u22+(5x2u2)2, take some easy caculation
h x 2 ( − x 2 ) = 2 x 2 2 + 25 h_{x_2}(-x_2)=2x_2^2+25 hx2(x2)=2x22+25 h x 2 ( 1 − x 2 ) = 2 x 2 2 − 2 x 2 + 18 h_{x_2}(1-x_2)=2x_2^2-2x_2+18 hx2(1x2)=2x222x2+18 h x 2 ( 2 − x 2 ) = 2 x 2 2 − 4 x 1 + 17 h_{x_2}(2-x_2)=2x_2^2-4x_1+17 hx2(2x2)=2x224x1+17 h x 2 ( 3 − x 2 ) = 2 x 2 2 − 6 x 2 + 22 h_{x_2}(3-x_2)=2x_2^2-6x_2+22 hx2(3x2)=2x226x2+22 h x 2 ( 4 − x 2 ) = 2 x 2 2 − 8 x 2 + 33 h_{x_2}(4-x_2)=2x_2^2-8x_2+33 hx2(4x2)=2x228x2+33 h x 2 ( 5 − x 2 ) = 2 x 2 2 − 10 x 2 + 50 h_{x_2}(5-x_2)=2x_2^2-10x_2+50 hx2(5x2)=2x2210x2+50and we get
{ μ 2 ∗ ( x 2 ) = − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 + 25 if  &ThickSpace; x 1 ≤ − 7 2 μ 2 ∗ ( x 2 ) = 1 − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 − 2 x 2 + 18 if  &ThickSpace; − 7 2 &lt; x 1 ≤ − 1 2 μ 2 ∗ ( x 2 ) = 2 − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 − 4 x 2 + 17 if  &ThickSpace; − 1 2 &lt; x 1 ≤ 5 2 μ 2 ∗ ( x 2 ) = 3 − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 − 6 x 2 + 22 if  &ThickSpace; 5 2 &lt; x 1 ≤ 11 2 μ 2 ∗ ( x 2 ) = 4 − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 − 8 x 2 + 33 if  &ThickSpace; 11 2 &lt; x 1 ≤ 17 2 μ 2 ∗ ( x 2 ) = 5 − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 − 10 x 2 + 50 if  &ThickSpace; x 1 &gt; 17 2 \begin{cases} \mu_2^*(x_2)=-x_2\;,\;J_2(x_2)=2x_2^2+25 &amp; \qquad\text{if }\; x_1\leq-\frac{7}{2} \\ \mu_2^*(x_2)=1-x_2\;,\;J_2(x_2)=2x_2^2-2x_2+18 &amp; \qquad\text{if }\;- \frac{7}{2}&lt;x_1\leq-\frac{1}{2} \\ \mu_2^*(x_2)=2-x_2\;,\;J_2(x_2)=2x_2^2-4x_2+17 &amp; \qquad\text{if }\; -\frac{1}{2}&lt;x_1\leq\frac{5}{2} \\ \mu_2^*(x_2)=3-x_2\;,\;J_2(x_2)=2x_2^2-6x_2+22 &amp; \qquad\text{if }\; \frac{5}{2}&lt;x_1\leq\frac{11}{2} \\ \mu_2^*(x_2)=4-x_2\;,\;J_2(x_2)=2x_2^2-8x_2+33 &amp; \qquad\text{if }\; \frac{11}{2}&lt;x_1\leq\frac{17}{2} \\ \mu_2^*(x_2)=5-x_2\;,\;J_2(x_2)=2x_2^2-10x_2+50 &amp; \qquad\text{if }\; x_1 &gt;\frac{17}{2} \\ \end{cases} μ2(x2)=x2,J2(x2)=2x22+25μ2(x2)=1x2,J2(x2)=2x222x2+18μ2(x2)=2x2,J2(x2)=2x224x2+17μ2(x2)=3x2,J2(x2)=2x226x2+22μ2(x2)=4x2,J2(x2)=2x228x2+33μ2(x2)=5x2,J2(x2)=2x2210x2+50if x127if 27<x121if 21<x125if 25<x1211if 211<x1217if x1>217

Similarly, for k = 1 k=1 k=1 we have
{ μ 1 ∗ ( x 1 ) = − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 + 17 if  &ThickSpace; x 1 ≤ − 1 2 μ 1 ∗ ( x 1 ) = 1 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 2 x 1 + 16 if  &ThickSpace; − 1 2 &lt; x 1 ≤ 5 2 μ 1 ∗ ( x 1 ) = 2 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 4 x 1 + 21 if  &ThickSpace; 5 2 &lt; x 1 ≤ 5 μ 1 ∗ ( x 1 ) = 3 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 6 x 1 + 31 if  &ThickSpace; 5 &lt; x 1 ≤ 15 2 μ 1 ∗ ( x 1 ) = 4 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 8 x 1 + 46 if  &ThickSpace; 15 2 &lt; x 1 ≤ 21 2 μ 1 ∗ ( x 1 ) = 5 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 10 x 1 + 67 if  &ThickSpace; x 1 &gt; 21 2 \begin{cases} \mu_1^*(x_1)=-x_1\;,\;J_1(x_1)=2x_1^2 +17&amp; \qquad\text{if }\; x_1\leq-\frac{1}{2} \\ \mu_1^*(x_1)=1-x_1\;,\;J_1(x_1)=2x_1^2-2x_1+16 &amp; \qquad\text{if }\; -\frac{1}{2}&lt;x_1\leq\frac{5}{2} \\ \mu_1^*(x_1)=2-x_1\;,\;J_1(x_1)=2x_1^2-4x_1+21 &amp; \qquad\text{if }\; \frac{5}{2}&lt;x_1\leq 5 \\ \mu_1^*(x_1)=3-x_1\;,\;J_1(x_1)=2x_1^2-6x_1+31 &amp; \qquad\text{if }\; 5&lt;x_1\leq\frac{15}{2} \\ \mu_1^*(x_1)=4-x_1\;,\;J_1(x_1)=2x_1^2-8x_1+46 &amp; \qquad\text{if }\; \frac{15}{2}&lt;x_1\leq\frac{21}{2} \\ \mu_1^*(x_1)=5-x_1\;,\;J_1(x_1)=2x_1^2-10x_1+67 &amp; \qquad\text{if }\; x_1 &gt;\frac{21}{2} \\ \end{cases} μ1(x1)=x1,J1(x1)=2x12+17μ1(x1)=1x1,J1(x1)=2x122x1+16μ1(x1)=2x1,J1(x1)=2x124x1+21μ1(x1)=3x1,J1(x1)=2x126x1+31μ1(x1)=4x1,J1(x1)=2x128x1+46μ1(x1)=5x1,J1(x1)=2x1210x1+67if x121if 21<x125if 25<x15if 5<x1215if 215<x1221if x1>221

For case k = 0 &ThinSpace; , &ThickSpace; x 0 = 5 k=0\,,\; x_0=5 k=0,x0=5
h 0 ( − 5 ) = 66 , &ThickSpace; h 0 ( − 4 ) = 57 , &ThickSpace; h 0 ( − 3 ) = 54 , &ThickSpace; h 0 ( − 2 ) = 56 , &ThickSpace; h 0 ( − 1 ) = 63 , &ThickSpace; h 0 ( 0 ) = 76 h_0(-5)=66,\; h_0(-4)=57,\;h_0(-3)=54,\;h_0(-2)=56,\;h_0(-1)=63,\;h_0(0)=76 h0(5)=66,h0(4)=57,h0(3)=54,h0(2)=56,h0(1)=63,h0(0)=76So μ 0 ∗ ( x 0 ) = − 3 \mu_0^*(x_0)=-3 μ0(x0)=3 and J 0 ( x 0 ) = 54 J_0(x_0)=54 J0(x0)=54.

(c)In this case, w k w_k wk is not always be 0 0 0 and satisfies certern distributuion. J 3 ( x 3 ) = min ⁡ u 3 ∈ U 3 ( x 3 ) E w 3 { g 3 ( x 3 , u 3 , w 3 ) + J 4 ( f 3 ( x 3 , u 3 , w 3 ) ) } = min ⁡ u 3 ∈ U 3 ( x 3 ) E w 3 ( x 3 2 + u 3 2 ) J_3(x_3)=\mathop{\min}\limits_{u_3\in U_3(x_3)} \mathop\mathbf{E}\limits_{w_3}\{g_3(x_3,u_3,w_3)+J_4(f_3(x_3,u_3,w_3))\}=\mathop{\min}\limits_{u_3\in U_3(x_3)} \mathop\mathbf{E}\limits_{w_3}(x_3^2+u_3^2) J3(x3)=u3U3(x3)minw3E{g3(x3,u3,w3)+J4(f3(x3,u3,w3))}=u3U3(x3)minw3E(x32+u32)which is not affected by the distritution of w k w_k wk, so the result is the same as case (a):

{ μ 3 ∗ ( x 3 ) = − x 3 &ThickSpace; , &ThickSpace; J 3 ( x 3 ) = 2 x 3 2 if  &ThickSpace; x 3 ≤ 1 2 μ 3 ∗ ( x 3 ) = 1 − x 3 &ThickSpace; , &ThickSpace; J 3 ( x 3 ) = 2 x 3 2 − 2 x 3 + 1 if  &ThickSpace; 1 2 &lt; x 3 ≤ 3 2 μ 3 ∗ ( x 3 ) = 2 − x 3 &ThickSpace; , &ThickSpace; J 3 ( x 3 ) = 2 x 3 2 − 4 x 3 + 4 if  &ThickSpace; 3 2 &lt; x 3 ≤ 5 2 μ 3 ∗ ( x 3 ) = 3 − x 3 &ThickSpace; , &ThickSpace; J 3 ( x 3 ) = 2 x 3 2 − 6 x 3 + 9 if  &ThickSpace; 5 2 &lt; x 3 ≤ 7 2 μ 3 ∗ ( x 3 ) = 4 − x 3 &ThickSpace; , &ThickSpace; J 3 ( x 3 ) = 2 x 3 2 − 8 x 3 + 16 if  &ThickSpace; 7 2 &lt; x 3 ≤ 9 2 μ 3 ∗ ( x 3 ) = 5 − x 3 &ThickSpace; , &ThickSpace; J 3 ( x 3 ) = 2 x 3 2 − 10 x 3 + 25 if  &ThickSpace; x 3 &gt; 9 2 \begin{cases} \mu_3^*(x_3)=-x_3\;,\;J_3(x_3)=2x_3^2 &amp; \qquad\text{if }\; x_3\leq\frac{1}{2} \\ \mu_3^*(x_3)=1-x_3\;,\;J_3(x_3)=2x_3^2-2x_3+1 &amp; \qquad\text{if }\; \frac{1}{2}&lt;x_3\leq\frac{3}{2} \\ \mu_3^*(x_3)=2-x_3\;,\;J_3(x_3)=2x_3^2-4x_3+4 &amp; \qquad\text{if }\; \frac{3}{2}&lt;x_3\leq\frac{5}{2} \\ \mu_3^*(x_3)=3-x_3\;,\;J_3(x_3)=2x_3^2-6x_3+9 &amp; \qquad\text{if }\; \frac{5}{2}&lt;x_3\leq\frac{7}{2} \\ \mu_3^*(x_3)=4-x_3\;,\;J_3(x_3)=2x_3^2-8x_3+16 &amp; \qquad\text{if }\; \frac{7}{2}&lt;x_3\leq\frac{9}{2} \\ \mu_3^*(x_3)=5-x_3\;,\;J_3(x_3)=2x_3^2-10x_3+25 &amp; \qquad\text{if }\; x_3 &gt;\frac{9}{2} \\ \end{cases} μ3(x3)=x3,J3(x3)=2x32μ3(x3)=1x3,J3(x3)=2x322x3+1μ3(x3)=2x3,J3(x3)=2x324x3+4μ3(x3)=3x3,J3(x3)=2x326x3+9μ3(x3)=4x3,J3(x3)=2x328x3+16μ3(x3)=5x3,J3(x3)=2x3210x3+25if x321if 21<x323if 23<x325if 25<x327if 27<x329if x3>29

Especially, we have J 3 ( 0 ) = 0 , &ThinSpace; J 3 ( 1 ) = 1 , &ThinSpace; J 3 ( 2 ) = 4 , &ThinSpace; J 3 ( 3 ) = 9 , &ThinSpace; J 3 ( 4 ) = 16 , &ThinSpace; J 3 ( 5 ) = 25 , J_3(0)=0, \,J_3(1)=1,\,J_3(2)=4,\,J_3(3)=9,\,J_3(4)=16,\,J_3(5)=25, J3(0)=0,J3(1)=1,J3(2)=4,J3(3)=9,J3(4)=16,J3(5)=25, which will be used in the case k = 2 k=2 k=2. Now J 2 ( x 2 ) = min ⁡ u 2 ∈ U 2 ( x 2 ) E w 2 ( x 2 2 + u 2 2 + J 3 ( x 2 + u 2 + w 2 ) ) J_2(x_2)=\mathop{\min}\limits_{u_2\in U_2(x_2)} \mathop\mathbf{E}\limits_{w_2}(x_2^2+u_2^2+J_3(x_2+u_2+w_2)) J2(x2)=u2U2(x2)minw2E(x22+u22+J3(x2+u2+w2))We calculate the expectation of the right side for each of the possible values of u 2 u_2 u2:
u 2 = − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( − x 2 ) 2 + J 3 ( 0 ) = 2 x 2 2 u_2=-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(-x_2)^2+J_3(0)=2x_2^2 u2=x2:E{}=x22+(x2)2+J3(0)=2x22 u 2 = 1 − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( 1 − x 2 ) 2 + 1 2 [ J 3 ( 0 ) + J 3 ( 2 ) ] = 2 x 2 2 − 2 x 2 + 3 u_2=1-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(1-x_2)^2+\frac{1}{2}[J_3(0)+J_3(2)]=2x_2^2-2x_2+3 u2=1x2:E{}=x22+(1x2)2+21[J3(0)+J3(2)]=2x222x2+3 u 2 = 2 − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( 2 − x 2 ) 2 + 1 2 [ J 3 ( 1 ) + J 3 ( 3 ) ] = 2 x 2 2 − 4 x 2 + 9 u_2=2-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(2-x_2)^2+\frac{1}{2}[J_3(1)+J_3(3)]=2x_2^2-4x_2+9 u2=2x2:E{}=x22+(2x2)2+21[J3(1)+J3(3)]=2x224x2+9 u 2 = 3 − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( 3 − x 2 ) 2 + 1 2 [ J 3 ( 2 ) + J 3 ( 4 ) ] = 2 x 2 2 − 6 x 2 + 19 u_2=3-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(3-x_2)^2+\frac{1}{2}[J_3(2)+J_3(4)]=2x_2^2-6x_2+19 u2=3x2:E{}=x22+(3x2)2+21[J3(2)+J3(4)]=2x226x2+19 u 2 = 4 − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( 4 − x 2 ) 2 + 1 2 [ J 3 ( 3 ) + J 3 ( 5 ) ] = 2 x 2 2 − 8 x 2 + 33 u_2=4-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(4-x_2)^2+\frac{1}{2}[J_3(3)+J_3(5)]=2x_2^2-8x_2+33 u2=4x2:E{}=x22+(4x2)2+21[J3(3)+J3(5)]=2x228x2+33 u 2 = 5 − x 2 &ThickSpace; : &ThickSpace; E { ⋅ } = x 2 2 + ( 5 − x 2 ) 2 + J 3 ( 5 ) = 2 x 2 2 − 10 x 2 + 50 u_2=5-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(5-x_2)^2+J_3(5)=2x_2^2-10x_2+50 u2=5x2:E{}=x22+(5x2)2+J3(5)=2x2210x2+50Hence we have

{ μ 2 ∗ ( x 2 ) = − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 if  &ThickSpace; x 2 ≤ 3 2 μ 2 ∗ ( x 2 ) = 1 − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 − 2 x 2 + 3 if  &ThickSpace; 3 2 &lt; x 2 ≤ 3 μ 2 ∗ ( x 2 ) = 2 − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 − 4 x 2 + 9 if  &ThickSpace; 3 &lt; x 2 ≤ 5 μ 2 ∗ ( x 2 ) = 3 − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 − 6 x 2 + 19 if  &ThickSpace; 5 &lt; x 2 ≤ 7 μ 2 ∗ ( x 2 ) = 4 − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 − 8 x 2 + 33 if  &ThickSpace; 7 &lt; x 2 ≤ 17 2 μ 2 ∗ ( x 2 ) = 5 − x 2 &ThickSpace; , &ThickSpace; J 2 ( x 2 ) = 2 x 2 2 − 10 x 2 + 50 if  &ThickSpace; x 2 &gt; 17 2 \begin{cases} \mu_2^*(x_2)=-x_2\;,\;J_2(x_2)=2x_2^2 &amp; \qquad\text{if }\; x_2\leq\frac{3}{2} \\ \mu_2^*(x_2)=1-x_2\;,\;J_2(x_2)=2x_2^2-2x_2+3 &amp; \qquad\text{if }\; \frac{3}{2}&lt;x_2\leq 3 \\ \mu_2^*(x_2)=2-x_2\;,\;J_2(x_2)=2x_2^2-4x_2+9 &amp; \qquad\text{if }\; 3&lt;x_2\leq5 \\ \mu_2^*(x_2)=3-x_2\;,\;J_2(x_2)=2x_2^2-6x_2+19 &amp; \qquad\text{if }\; 5&lt;x_2\leq7 \\ \mu_2^*(x_2)=4-x_2\;,\;J_2(x_2)=2x_2^2-8x_2+33 &amp; \qquad\text{if }\; 7&lt;x_2\leq\frac{17}{2} \\ \mu_2^*(x_2)=5-x_2\;,\;J_2(x_2)=2x_2^2-10x_2+50 &amp; \qquad\text{if }\; x_2 &gt;\frac{17}{2} \\ \end{cases} μ2(x2)=x2,J2(x2)=2x22μ2(x2)=1x2,J2(x2)=2x222x2+3μ2(x2)=2x2,J2(x2)=2x224x2+9μ2(x2)=3x2,J2(x2)=2x226x2+19μ2(x2)=4x2,J2(x2)=2x228x2+33μ2(x2)=5x2,J2(x2)=2x2210x2+50if x223if 23<x23if 3<x25if 5<x27if 7<x2217if x2>217

Especially, J 2 ( 0 ) = 0 , &ThinSpace; J 2 ( 1 ) = 2 , &ThinSpace; J 2 ( 2 ) = 7 , &ThinSpace; J 2 ( 3 ) = 15 , &ThinSpace; J 2 ( 4 ) = 25 , &ThinSpace; J 2 ( 5 ) = 39. J_2(0)=0,\,J_2(1)=2,\,J_2(2)=7,\,J_2(3)=15,\,J_2(4)=25,\,J_2(5)=39. J2(0)=0,J2(1)=2,J2(2)=7,J2(3)=15,J2(4)=25,J2(5)=39.

For case k = 1 k=1 k=1 J 1 ( x 1 ) = min ⁡ u 1 ∈ U 1 ( x 1 ) E w 1 ( x 1 2 + u 1 2 + J 2 ( x 1 + u 1 + w 1 ) ) J_1(x_1)=\mathop{\min}\limits_{u_1\in U_1(x_1)} \mathop\mathbf{E}\limits_{w_1}(x_1^2+u_1^2+J_2(x_1+u_1+w_1)) J1(x1)=u1U1(x1)minw1E(x12+u12+J2(x1+u1+w1))We calculate the expectation of the right side for each of the possible values of u 1 u_1 u1:
u 1 = − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( − x 1 ) 2 + J 2 ( 0 ) = 2 x 1 2 u_1=-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(-x_1)^2+J_2(0)=2x_1^2 u1=x1:E{}=x12+(x1)2+J2(0)=2x12 u 2 = 1 − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( 1 − x 1 ) 2 + 1 2 [ J 2 ( 0 ) + J 2 ( 2 ) ] = 2 x 1 2 − 2 x 1 + 9 2 u_2=1-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(1-x_1)^2+\frac{1}{2}[J_2(0)+J_2(2)]=2x_1^2-2x_1+\frac{9}{2} u2=1x1:E{}=x12+(1x1)2+21[J2(0)+J2(2)]=2x122x1+29 u 1 = 2 − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( 2 − x 1 ) 2 + 1 2 [ J 2 ( 1 ) + J 2 ( 3 ) ] = 2 x 1 2 − 4 x 1 + 25 2 u_1=2-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(2-x_1)^2+\frac{1}{2}[J_2(1)+J_2(3)]=2x_1^2-4x_1+\frac{25}{2} u1=2x1:E{}=x12+(2x1)2+21[J2(1)+J2(3)]=2x124x1+225 u 1 = 3 − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( 3 − x 1 ) 2 + 1 2 [ J 2 ( 2 ) + J 2 ( 4 ) ] = 2 x 1 2 − 6 x 1 + 25 u_1=3-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(3-x_1)^2+\frac{1}{2}[J_2(2)+J_2(4)]=2x_1^2-6x_1+25 u1=3x1:E{}=x12+(3x1)2+21[J2(2)+J2(4)]=2x126x1+25 u 1 = 4 − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( 4 − x 1 ) 2 + 1 2 [ J 2 ( 3 ) + J 2 ( 5 ) ] = 2 x 1 2 − 8 x 1 + 43 u_1=4-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(4-x_1)^2+\frac{1}{2}[J_2(3)+J_2(5)]=2x_1^2-8x_1+43 u1=4x1:E{}=x12+(4x1)2+21[J2(3)+J2(5)]=2x128x1+43 u 1 = 5 − x 1 &ThickSpace; : &ThickSpace; E { ⋅ } = x 1 2 + ( 5 − x 1 ) 2 + J 2 ( 5 ) = 2 x 1 2 − 10 x 1 + 64 u_1=5-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(5-x_1)^2+J_2(5)=2x_1^2-10x_1+64 u1=5x1:E{}=x12+(5x1)2+J2(5)=2x1210x1+64Hence we have

{ μ 1 ∗ ( x 1 ) = − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 if  &ThickSpace; x 1 ≤ 9 4 μ 1 ∗ ( x 1 ) = 1 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 2 x 1 + 9 2 if  &ThickSpace; 9 4 &lt; x 1 ≤ 4 μ 1 ∗ ( x 1 ) = 2 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 4 x 1 + 25 2 if  &ThickSpace; 4 &lt; x 1 ≤ 25 4 μ 1 ∗ ( x 1 ) = 3 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 6 x 1 + 25 if  &ThickSpace; 25 4 &lt; x 1 ≤ 9 μ 1 ∗ ( x 1 ) = 4 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 8 x 1 + 43 if  &ThickSpace; 9 &lt; x 1 ≤ 21 2 μ 1 ∗ ( x 1 ) = 5 − x 1 &ThickSpace; , &ThickSpace; J 1 ( x 1 ) = 2 x 1 2 − 10 x 1 + 64 if  &ThickSpace; x 1 &gt; 21 2 \begin{cases} \mu_1^*(x_1)=-x_1\;,\;J_1(x_1)=2x_1^2 &amp; \qquad\text{if }\; x_1\leq\frac{9}{4} \\ \mu_1^*(x_1)=1-x_1\;,\;J_1(x_1)=2x_1^2-2x_1+\frac{9}{2} &amp; \qquad\text{if }\; \frac{9}{4}&lt;x_1\leq 4 \\ \mu_1^*(x_1)=2-x_1\;,\;J_1(x_1)=2x_1^2-4x_1+\frac{25}{2}&amp; \qquad\text{if }\; 4&lt;x_1\leq\frac{25}{4} \\ \mu_1^*(x_1)=3-x_1\;,\;J_1(x_1)=2x_1^2-6x_1+25&amp; \qquad\text{if }\; \frac{25}{4}&lt;x_1\leq9 \\ \mu_1^*(x_1)=4-x_1\;,\;J_1(x_1)=2x_1^2-8x_1+43&amp; \qquad\text{if }\; 9&lt;x_1\leq\frac{21}{2} \\ \mu_1^*(x_1)=5-x_1\;,\;J_1(x_1)=2x_1^2-10x_1+64&amp; \qquad\text{if }\; x_1 &gt;\frac{21}{2} \\ \end{cases} μ1(x1)=x1,J1(x1)=2x12μ1(x1)=1x1,J1(x1)=2x122x1+29μ1(x1)=2x1,J1(x1)=2x124x1+225μ1(x1)=3x1,J1(x1)=2x126x1+25μ1(x1)=4x1,J1(x1)=2x128x1+43μ1(x1)=5x1,J1(x1)=2x1210x1+64if x149if 49<x14if 4<x1425if 425<x19if 9<x1221if x1>221

Especially, J 1 ( 0 ) = 0 , &ThinSpace; J 1 ( 1 ) = 2 , &ThinSpace; J 1 ( 2 ) = 8 , &ThinSpace; J 1 ( 3 ) = 15 2 , &ThinSpace; J 1 ( 4 ) = 57 2 , &ThinSpace; J 1 ( 5 ) = 85 2 . J_1(0)=0,\,J_1(1)=2,\,J_1(2)=8,\,J_1(3)=\frac{15}{2},\,J_1(4)=\frac{57}{2},\,J_1(5)=\frac{85}{2}. J1(0)=0,J1(1)=2,J1(2)=8,J1(3)=215,J1(4)=257,J1(5)=285.

The last step, the algorithm takes the form
J 0 ( x 0 ) = min ⁡ u 0 ∈ U 0 ( x 0 ) E w 0 ( x 0 2 + u 0 2 + J 1 ( x 0 + u 0 + w 0 ) ) = min ⁡ u 0 ∈ U 0 ( x 0 ) E w 0 ( 25 + u 0 2 + J 1 ( 5 + u 0 + w 0 ) ) J_0(x_0)=\mathop{\min}\limits_{u_0\in U_0(x_0)} \mathop\mathbf{E}\limits_{w_0}(x_0^2+u_0^2+J_1(x_0+u_0+w_0))=\mathop{\min}\limits_{u_0\in U_0(x_0)} \mathop\mathbf{E}\limits_{w_0}\big(25+u_0^2+J_1(5+u_0+w_0)\big) J0(x0)=u0U0(x0)minw0E(x02+u02+J1(x0+u0+w0))=u0U0(x0)minw0E(25+u02+J1(5+u0+w0)) u 0 = − 5 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 25 + J 1 ( 0 ) = 50 u_0=-5\;:\;\mathbf{E}\{\cdot\}=25+25+J_1(0)=50 u0=5:E{}=25+25+J1(0)=50 u 0 = − 4 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 16 + 1 2 [ J 1 ( 0 ) + J 1 ( 2 ) ] = 45 u_0=-4\;:\;\mathbf{E}\{\cdot\}=25+16+\frac{1}{2}[J_1(0)+J_1(2)]=45 u0=4:E{}=25+16+21[J1(0)+J1(2)]=45 u 0 = − 3 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 9 + 1 2 [ J 1 ( 1 ) + J 1 ( 3 ) ] = 38 3 4 u_0=-3\;:\;\mathbf{E}\{\cdot\}=25+9+\frac{1}{2}[J_1(1)+J_1(3)]=38\frac{3}{4} u0=3:E{}=25+9+21[J1(1)+J1(3)]=3843 u 0 = − 2 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 4 + 1 2 [ J 1 ( 2 ) + J 1 ( 4 ) ] = 47 1 4 u_0=-2\;:\;\mathbf{E}\{\cdot\}=25+4+\frac{1}{2}[J_1(2)+J_1(4)]=47\frac{1}{4} u0=2:E{}=25+4+21[J1(2)+J1(4)]=4741 u 1 = − 1 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 1 + 1 2 [ J 1 ( 3 ) + J 1 ( 5 ) ] = 51 u_1=-1\;:\;\mathbf{E}\{\cdot\}=25+1+\frac{1}{2}[J_1(3)+J_1(5)]=51 u1=1:E{}=25+1+21[J1(3)+J1(5)]=51 u 0 = 0 &ThickSpace; : &ThickSpace; E { ⋅ } = 25 + 0 + J 1 ( 5 ) = 67 1 2 u_0=0\;:\;\mathbf{E}\{\cdot\}=25+0+J_1(5)=67\frac{1}{2} u0=0:E{}=25+0+J1(5)=6721So μ 0 ∗ ( x 0 ) = − 3 , J 0 ( x 0 ) = 38 3 4 . □ \mu_0^*(x_0)=-3,J_0(x_0)=38\frac{3}{4}.\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\Box μ0(x0)=3,J0(x0)=3843.
PS: In the book, a hint is given : Alternatively, you may use a terminal cost g 4 ( x 4 ) g_4(x_4) g4(x4) equal to a very large number for x 4 ≠ 5 x_4\neq 5 x4̸=5. Why a very large number ? My caculation is wrong?

1.2 Carry out the calculations needed to verify that J 0 ( 1 ) = 2.67 J_0(1)=2.67 J0(1)=2.67 and J 0 ( 2 ) = 2.608 J_0(2)=2.608 J0(2)=2.608 in Example 1.3.2.
Solution. The result in the exercise is wrong, even not the same as that in the text.
For x 0 = 1 x_0=1 x0=1, we have J 0 ( 1 ) = min ⁡ u 0 = 0 , 1 E w 0 = 0 , 1 , 2 { u 0 + ( 1 + u 0 − w 0 ) 2 + J 1 ( max ⁡ ( 0 , 1 + u 0 − w 0 ) ) ) } J_0(1)=\min_{u_0=0,1} \mathop{\bf E}\limits_{w_0=0,1,2}\{u_0+(1+u_0-w_0)^2+J_1(\max(0,1+u_0-w_0)))\} J0(1)=u0=0,1minw0=0,1,2E{u0+(1+u0w0)2+J1(max(0,1+u0w0)))} u 0 = 0 : &ThickSpace; E ( . ) = E w 0 = 0 , 1 , 2 { ( 1 − w 0 ) 2 + J 1 ( max ⁡ ( 0 , 1 − w 0 ) ) } u_0=0: \;\mathbf{E}(.)=\mathop{\mathbf{E}}\limits_{w_0=0,1,2}\{(1-w_0)^2+J_1(\max(0,1-w_0))\} u0=0:E(.)=w0=0,1,2E{(1w0)2+J1(max(0,1w0))}
= 0.1 × [ 1 + J 1 ( 1 ) ] + 0.7 × J 1 ( 0 ) + 0.2 × [ 1 + J 1 ( 0 ) ] \qquad\qquad\qquad=0.1\times[1+J_1(1)]+0.7\times J_1(0)+0.2\times[1+J_1(0)] =0.1×[1+J1(1)]+0.7×J1(0)+0.2×[1+J1(0)]
= 0.1 × 2.5 + 0.7 × 2.5 + 0.2 × 3.5 \qquad\qquad\qquad=0.1\times2.5+0.7\times 2.5 + 0.2\times 3.5 =0.1×2.5+0.7×2.5+0.2×3.5
= 2.7 \qquad\qquad\qquad=2.7 =2.7
u 0 = 1 : &ThickSpace; E ( . ) = E w 0 = 0 , 1 , 2 { 1 + ( 2 − w 0 ) 2 + J 1 ( max ⁡ ( 0 , 2 − w 0 ) ) } u_0=1: \;\mathbf{E}(.)=\mathop{\mathbf{E}}\limits_{w_0=0,1,2}\{1+(2-w_0)^2+J_1(\max(0,2-w_0))\} u0=1:E(.)=w0=0,1,2E{1+(2w0)2+J1(max(0,2w0))}
= 0.1 × [ 5 + J 1 ( 2 ) ] + 0.7 × [ 2 + J 1 ( 1 ) ] + 0.2 × [ 1 + J 1 ( 0 ) ] \qquad\qquad\qquad=0.1\times[5+J_1(2)]+0.7\times [2+J_1(1)]+0.2\times[1+J_1(0)] =0.1×[5+J1(2)]+0.7×[2+J1(1)]+0.2×[1+J1(0)]
= 0.1 × 6.68 + 0.7 × 3.5 + 0.2 × 3.5 \qquad\qquad\qquad=0.1\times6.68+0.7\times3.5 + 0.2\times 3.5 =0.1×6.68+0.7×3.5+0.2×3.5
= 3.818 \qquad\qquad\qquad=3.818 =3.818
So J 0 ( 1 ) = 2.7 , μ 0 ∗ ( 1 ) = 0. J_0(1)=2.7, \mu_0^*(1)=0. J0(1)=2.7,μ0(1)=0.

For x 0 = 2 x_0=2 x0=2, we have J 0 ( 2 ) = min ⁡ u 0 = 0 E w 0 = 0 , 1 , 2 { u 0 + ( 2 + u 0 − w 0 ) 2 + J 1 ( max ⁡ ( 0 , 2 + u 0 − w 0 ) ) ) } J_0(2)=\min_{u_0=0} \mathop{\bf E}\limits_{w_0=0,1,2}\{u_0+(2+u_0-w_0)^2+J_1(\max(0,2+u_0-w_0)))\} J0(2)=u0=0minw0=0,1,2E{u0+(2+u0w0)2+J1(max(0,2+u0w0)))} = E w 0 = 0 , 1 , 2 { ( 2 − w 0 ) 2 + J 1 ( max ⁡ ( 0 , 2 − w 0 ) ) } \qquad\qquad\qquad\qquad=\mathop{\mathbf{E}}\limits_{w_0=0,1,2}\{(2-w_0)^2+J_1(\max(0,2-w_0))\} =w0=0,1,2E{(2w0)2+J1(max(0,2w0))}
= 0.1 × [ 4 + J 1 ( 2 ) ] + 0.7 × [ 1 + J 1 ( 1 ) ] + 0.2 × J 1 ( 0 ) \qquad\qquad\qquad\qquad=0.1\times[4+J_1(2)]+0.7\times[1+J_1(1)]+0.2\times J_1(0) =0.1×[4+J1(2)]+0.7×[1+J1(1)]+0.2×J1(0)
= 0.1 × 5.68 + 0.7 × 2.5 + 0.2 × 2.5 \qquad\qquad\qquad\qquad=0.1\times5.68+0.7\times2.5+0.2\times 2.5 =0.1×5.68+0.7×2.5+0.2×2.5
= 2.818 \qquad\qquad\qquad\qquad=2.818 =2.818
So J 0 ( 2 ) = 2.818 , μ 0 ∗ ( 2 ) = 0. □ J_0(2)=2.818,\mu_0^*(2)=0.\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\Box J0(2)=2.818,μ0(2)=0.

1.6 (Discouted Cost per State) In the framework of the basic problem, consider the case where the cost is of the form E w k k = 0 , 1 , . . . , N − 1 { α N g N ( x N ) + ∑ k = 0 N − 1 α k g k ( x k , u k , w k ) } \mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,...,N-1}}\left\{\alpha^Ng_N(x_N)+\sum_{k=0}^{N-1}\alpha^kg_k(x_k,u_k,w_k)\right\} k=0,1,...,N1wkE{αNgN(xN)+k=0N1αkgk(xk,uk,wk)}where α \alpha α is a discount factor with 0 &lt; α &lt; 1 0&lt;\alpha&lt;1 0<α<1. Show that an alternate form the DP algorithm is given by V N ( x N ) = g N ( x N ) V_N(x_N)=g_N(x_N) VN(xN)=gN(xN) V k ( x k ) = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k ) + α V k + 1 ( f k ( x k , u k , w k ) ) } . V_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\left\{g_k(x_k,u_k,w_k)+\alpha V_{k+1}\big(f_k(x_k,u_k,w_k)\big)\right\}. Vk(xk)=ukUk(xk)minwkE{gk(xk,uk,wk)+αVk+1(fk(xk,uk,wk))}.Solution. Using similar argument to Proposition 1.3.1, in the dicounted cost case, the minimizing cost takes the form
V ∗ ( x 0 ) = min ⁡ π ∈ Π E w k k = 0 , 1 , . . . , N − 1 { α N g N ( x N ) + ∑ k = 0 N − 1 α k g k ( x k , u k , w k ) } V^*(x_0)=\mathop{\min}\limits_{\pi\in\Pi}\mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,...,N-1}}\left\{\alpha^Ng_N(x_N)+\sum_{k=0}^{N-1}\alpha^kg_k(x_k,u_k,w_k)\right\} V(x0)=πΠmink=0,1,...,N1wkE{αNgN(xN)+k=0N1αkgk(xk,uk,wk)} Let V k ∗ ( x k ) V_k^*(x_k) Vk(xk) be the optimal cost for the ( N − k ) (N-k) (Nk)-stage problem that starts at state x k x_k xk and time k k k, and ends at time N N N.
V k ∗ ( x k ) = min ⁡ π k E w k , . . . , w N − 1 { α N − k g N ( x N ) + ∑ i = k N − 1 α i − k g i ( x i , u i , w i ) } V_k^*(x_k)=\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{w_k,...,w_{N-1}}\left\{\alpha^{N-k}g_N(x_N)+\sum_{i=k}^{N-1}\alpha^{i-k}g_i(x_i,u_i,w_i)\right\} Vk(xk)=πkminwk,...,wN1E{αNkgN(xN)+i=kN1αikgi(xi,ui,wi)} Claim: &ThickSpace; V k ∗ ( x k ) = V k ( x k ) . \; V_k^*(x_k)=V_k(x_k). Vk(xk)=Vk(xk).
We prove this cliam by induction on k k k. Firstly, V N ∗ ( x N ) = g N ( x N ) = V N ( x N ) V_N^*(x_N)=g_N(x_N)=V_N(x_N) VN(xN)=gN(xN)=VN(xN). Suppose the claim is true for any k + 1 , . . . , N k+1, ..., N k+1,...,N, then
V k ∗ ( x k ) = min ⁡ π k E w k , . . . , w N − 1 { g k ( x k , u k , w k ) + α N − k g N ( x N ) + ∑ i = k + 1 N − 1 α i − k g i ( x i , u i , w i ) } V_k^*(x_k)=\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{w_k,...,w_{N-1}}\left\{g_k(x_k,u_k,w_k)+\alpha^{N-k}g_N(x_N)+\sum_{i=k+1}^{N-1}\alpha^{i-k}g_i(x_i,u_i,w_i)\right\}\qquad\qquad\quad Vk(xk)=πkminwk,...,wN1E{gk(xk,uk,wk)+αNkgN(xN)+i=k+1N1αikgi(xi,ui,wi)} = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k } + α ⋅ min ⁡ π k E w k , . . . , w N − 1 { α N − k − 1 g N ( x N ) + ∑ i = k + 1 N − 1 α i − k − 1 g i ( x i , u i , w i ) } =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha\cdot\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{w_k,...,w_{N-1}}\left\{\alpha^{N-k-1}g_N(x_N)+\sum_{i=k+1}^{N-1}\alpha^{i-k-1}g_i(x_i,u_i,w_i)\right\} =ukUk(xk)minwkE{gk(xk,uk,wk}+απkminwk,...,wN1E{αNk1gN(xN)+i=k+1N1αik1gi(xi,ui,wi)} = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k } + α ⋅ min ⁡ π k + 1 E w k + 1 , . . . , w N − 1 { α N − k − 1 g N ( x N ) + ∑ i = k + 1 N − 1 α i − k − 1 g i ( x i , u i , w i ) } =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha\cdot\mathop{\min}\limits_{\pi^{k+1}}\mathop\mathbf{E}\limits_{w_{k+1},...,w_{N-1}}\left\{\alpha^{N-k-1}g_N(x_N)+\sum_{i=k+1}^{N-1}\alpha^{i-k-1}g_i(x_i,u_i,w_i)\right\} =ukUk(xk)minwkE{gk(xk,uk,wk}+απk+1minwk+1,...,wN1E{αNk1gN(xN)+i=k+1N1αik1gi(xi,ui,wi)} = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k } + α V k + 1 ∗ ( x k + 1 ) =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha V_{k+1}^*(x_{k+1})\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad =ukUk(xk)minwkE{gk(xk,uk,wk}+αVk+1(xk+1) = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k } + α V k + 1 ( x k + 1 ) =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha V_{k+1}(x_{k+1})\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad =ukUk(xk)minwkE{gk(xk,uk,wk}+αVk+1(xk+1) = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k } + α ⋅ min ⁡ u k ∈ U k ( x k ) E w k V k + 1 ( f k ( x k , u k , w k ) ) &ThickSpace; =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha \cdot\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}V_{k+1}(f_k(x_k,u_k,w_k))\qquad\qquad\qquad\qquad\qquad\qquad\; =ukUk(xk)minwkE{gk(xk,uk,wk}+αukUk(xk)minwkEVk+1(fk(xk,uk,wk)) = min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k ) + α V k + 1 ( f k ( x k , u k , w k ) ) } = V k ( x k ) □ =\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\left\{g_k(x_k,u_k,w_k)+\alpha V_{k+1}\big(f_k(x_k,u_k,w_k)\big)\right\}=V_k(x_k)\qquad\qquad\qquad\qquad\qquad\qquad\Box =ukUk(xk)minwkE{gk(xk,uk,wk)+αVk+1(fk(xk,uk,wk))}=Vk(xk)

1.7 (Exponential Cost Function) In the framework of the basic problem, consider the case where the cost is of the form E w k k = 0 , 1 , . . . , N − 1 { exp ⁡ ( g N ( x N ) + ∑ k = 0 N − 1 g k ( x k , u k , w k ) ) } \mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,...,N-1}}\left\{\exp\left(g_N(x_N)+\sum_{k=0}^{N-1}g_k(x_k,u_k,w_k)\right)\right\} k=0,1,...,N1wkE{exp(gN(xN)+k=0N1gk(xk,uk,wk))}(a)Show that the optimal cost and an optimal policy can be obtained from the DP-like algorithm J N ( x N ) = exp ⁡ ( g N ( x N ) ) J_N(x_N)=\exp\left(g_N(x_N)\right) JN(xN)=exp(gN(xN)) J k ( x k ) = min ⁡ u k ∈ U k ( x k ) E w k { J k + 1 ( f k ( x k , u k , w k ) ) exp ⁡ ( g k ( x k , u k , w k ) ) } J_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\left\{J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\exp\left(g_k(x_k,u_k,w_k)\right)\right\} Jk(xk)=ukUk(xk)minwkE{Jk+1(fk(xk,uk,wk))exp(gk(xk,uk,wk))} (b)Define the functions V k ( x k ) = ln ⁡ J k ( x k ) V_k(x_k)=\ln J_k(x_k) Vk(xk)=lnJk(xk). Assume also that g k g_k gk is a function of x k x_k xk and u k u_k uk only (and not of w k w_k wk). Show that the above algorithm can be rewritten as V N ( x N ) = g N ( x N ) V_N(x_N)=g_N(x_N) VN(xN)=gN(xN) V k ( x k ) = min ⁡ u k ∈ U k ( x k ) { g k ( x k , u k ) + ln ⁡ E w k { exp ⁡ ( V k + 1 ( f k ( x k , u k , w k ) ) ) } } V_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{g_k(x_k,u_k)+\ln \mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(V_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right)\right\}\right\} Vk(xk)=ukUk(xk)min{gk(xk,uk)+lnwkE{exp(Vk+1(fk(xk,uk,wk)))}}
Solution. (a) Define J k ∗ ( x k ) = min ⁡ π k E w i i = k , k + 1 , . . . , N − 1 { exp ⁡ ( g N ( x N ) + ∑ i = k N − 1 g i ( x i , u i , w i ) ) } J_k^*(x_k)=\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{\mathop{w_i}\limits_{i=k,k+1,...,N-1}}\left\{\exp\left(g_N(x_N)+\sum_{i=k}^{N-1}g_i(x_i,u_i,w_i)\right)\right\} Jk(xk)=πkmini=k,k+1,...,N1wiE{exp(gN(xN)+i=kN1gi(xi,ui,wi))} It is easy to check that J k ∗ ( x k ) = J k ( x k ) J_k^*(x_k)=J_k(x_k) Jk(xk)=Jk(xk).
J k ∗ ( x k ) = min ⁡ u k E w k { exp ⁡ ( g k ( x k , u k , w k ) ) } min ⁡ π k E w i i = k , k + 1 , . . . , N − 1 { exp ⁡ ( g N ( x N ) + ∑ i = k + 1 N − 1 g i ( x i , u i , w i ) ) } J_k^*(x_k)=\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{\mathop{w_i}\limits_{i=k,k+1,...,N-1}}\left\{\exp\left(g_N(x_N)+\sum_{i=k+1}^{N-1}g_i(x_i,u_i,w_i)\right)\right\} Jk(xk)=ukminwkE{exp(gk(xk,uk,wk))}πkmini=k,k+1,...,N1wiE{exp(gN(xN)+i=k+1N1gi(xi,ui,wi))} = min ⁡ u k E w k { exp ⁡ ( g k ( x k , u k , w k ) ) } min ⁡ π k + 1 x k + 1 = f k ( x k , u k , w k ) E w i i = k + 1 , . . . . , N − 1 { exp ⁡ ( g N ( x N ) + ∑ i = k + 1 N − 1 g i ( x i , u i , w i ) ) } =\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\mathop{\min}\limits_{\mathop{\pi^{k+1}}\limits_{x_{k+1}=f_k(x_k,u_k,w_k)}}\mathop\mathbf{E}\limits_{\mathop{w_i}\limits_{i=k+1,....,N-1}}\left\{\exp\left(g_N(x_N)+\sum_{i=k+1}^{N-1}g_i(x_i,u_i,w_i)\right)\right\} =ukminwkE{exp(gk(xk,uk,wk))}xk+1=fk(xk,uk,wk)πk+1mini=k+1,....,N1wiE{exp(gN(xN)+i=k+1N1gi(xi,ui,wi))} = min ⁡ u k E w k { exp ⁡ ( g k ( x k , u k , w k ) ) } ⋅ min ⁡ u k E w k J k + 1 ∗ ( f k ( x k , u k , w k ) ) ) =\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\cdot \mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}J_{k+1}^*(f_k(x_k,u_k,w_k)))\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad =ukminwkE{exp(gk(xk,uk,wk))}ukminwkEJk+1(fk(xk,uk,wk))) = min ⁡ u k E w k { exp ⁡ ( g k ( x k , u k , w k ) ) } ⋅ min ⁡ u k E w k J k + 1 ( f k ( x k , u k , w k ) ) ) = J k ( x k ) =\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\cdot \mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}J_{k+1}(f_k(x_k,u_k,w_k)))=J_k(x_k)\qquad\qquad\qquad\qquad\qquad =ukminwkE{exp(gk(xk,uk,wk))}ukminwkEJk+1(fk(xk,uk,wk)))=Jk(xk) (b) Since g k g_k gk is a function not of w k w_k wk, we can rewrite J k ( x k ) J_k(x_k) Jk(xk) as follows: J k ( x k ) = min ⁡ u k ∈ U k ( x k ) { exp ⁡ ( g k ( x k , u k , w k ) ) E w k { J k + 1 ( f k ( x k , u k , w k ) ) } } J_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\mathop\mathbf{E}\limits_{w_k}\left\{J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\}\right\} Jk(xk)=ukUk(xk)min{exp(gk(xk,uk,wk))wkE{Jk+1(fk(xk,uk,wk))}} So V k ( x k ) = ln ⁡ J k ( x k ) V_k(x_k)=\ln J_k(x_k)\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad Vk(xk)=lnJk(xk) = min ⁡ u k ∈ U k ( x k ) { g k ( x k , u k , w k ) + ln ⁡ E w k { J k + 1 ( f k ( x k , u k , w k ) ) } } =\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{g_k(x_k,u_k,w_k)+\ln\mathop\mathbf{E}\limits_{w_k}\left\{J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\}\right\}\qquad\qquad\qquad =ukUk(xk)min{gk(xk,uk,wk)+lnwkE{Jk+1(fk(xk,uk,wk))}} = min ⁡ u k ∈ U k ( x k ) { g k ( x k , u k , w k ) + ln ⁡ E w k { exp ⁡ ( V k + 1 ( f k ( x k , u k , w k ) ) ) } } &ThickSpace; □ =\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{g_k(x_k,u_k,w_k)+\ln\mathop\mathbf{E}\limits_{w_k}\left\{\exp \left(V_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right)\right\}\right\}\qquad\;\Box =ukUk(xk)min{gk(xk,uk,wk)+lnwkE{exp(Vk+1(fk(xk,uk,wk)))}}

1.8 (Terminating Process) In the framework of the basic problem, consider the case where the system evolution terminates at time i i i when a given value w i ˉ \bar{w_i} wiˉ of the disurbance at time i i i occurs, or when a termination decesion u i u_i ui is made by the controller. If termination occurs at time i i i, the resulting cost is T + ∑ k = 0 i g k ( x k , u k , w k ) T+\sum_{k=0}^ig_k(x_k,u_k,w_k) T+k=0igk(xk,uk,wk)where T T T is a termination cost. If the process has not terminated up to the final time N N N, the resulting cost is g N ( x N ) + ∑ k = 0 N − 1 g k ( x k , u k , w k ) g_N(x_N)+\sum_{k=0}^{N-1}g_k(x_k,u_k,w_k) gN(xN)+k=0N1gk(xk,uk,wk). Reformulate the problem into the framework of the basic problem.

1.9 (Multiplicative Cost) In the framework of the basic problem, consider the case where the cost has the multiplicative form E w k k = 0 , 1 , . . . , N − 1 { g N ( x N ) ⋅ g N − 1 ( x N − 1 , u N − 1 , w N − 1 ) … g 0 ( x 0 , u 0 , w 0 ) } \mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,...,N-1}}\left\{g_N(x_N)\cdot g_{N-1}(x_{N-1}, u_{N-1}, w_{N-1})\ldots g_0(x_0,u_0,w_0)\right\} k=0,1,...,N1wkE{gN(xN)gN1(xN1,uN1,wN1)g0(x0,u0,w0)} Develop a DP-like algorithm for this problem assuming that g k ( x k , u k , w k ) ≥ 0 g_k(x_k,u_k,w_k)\ge0 gk(xk,uk,wk)0 for all x k , u k , w k x_k, u_k, w_k xk,uk,wk and k k k.
Solution. The DP-like algorithm for this problem is as follows:
J N ( x N ) = g N ( x N ) J_N(x_N)=g_N(x_N) JN(xN)=gN(xN) J k ( x k ) = E w k { g k ( x k , u k , w k ) J k + 1 ( f k ( x k , u k , w k ) ) } J_k(x_k)=\mathop\mathbf{E}\limits_{w_k}\left\{g_k(x_k,u_k,w_k)J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\} Jk(xk)=wkE{gk(xk,uk,wk)Jk+1(fk(xk,uk,wk))} The proof is standard as above, we may define
J k ∗ ( x k ) = E w k i = k , . . . , N − 1 { g N ( x N ) … g i ( x i , u i , w i ) … g k ( x k , u k , w k ) } J_k^*(x_k)=\mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{i=k,...,N-1}} \left\{g_N(x_N)\ldots g_i(x_i,u_i,w_i)\ldots g_k(x_k,u_k,w_k) \right\} Jk(xk)=i=k,...,N1wkE{gN(xN)gi(xi,ui,wi)gk(xk,uk,wk)} and prove J k ∗ ( x k ) = J k ( x k ) J_k^*(x_k)=J_k(x_k) Jk(xk)=Jk(xk). □ \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\Box

1.10 Assume that we have a vessel whose maximum weight capacity is z z z and whose cargo is to consist of different quantities of N N N different items. Let v i v_i vi denote the value of the i i i-th type of item, w i w_i wi the weight of i i i-th type of item, and x i x_i xi the number of items of type i i i that are loaded in the vessel. The problem is to find the most valuable cargo, i.e., to maximize ∑ i = 1 N x i v i \sum_{i=1}^Nx_iv_i i=1Nxivi subject to the constraints ∑ i = 1 N x i w i ≤ z \sum_{i=1}^Nx_iw_i\leq z i=1Nxiwiz and x i = 0 , 1 , 2 , … x_i=0,1,2,\ldots xi=0,1,2, Formulate this problem in terms of DP.

Solution. We formulate this problem in terms of DP as follows:
\qquad a state x k ′ x&#x27;_k xk is defined by x k ′ = ∑ i = 0 k − 1 x i w i x&#x27;_k=\mathop{\sum}\limits_{i=0}^{k-1}x_iw_i xk=i=0k1xiwi
\qquad an action u k ′ u&#x27;_k uk is taken on the space U k ( x k ′ ) = { 0 , 1 , … , [ z − x k ′ w k ] } U_k(x&#x27;_k)=\left\{0,1,\ldots,\left[\frac{z-x&#x27;_k}{w_k}\right]\right\} Uk(xk)={0,1,,[wkzxk]}
\qquad a random parameter w k ′ w&#x27;_k wk is always be 0
\qquad state transform function is defined as x k + 1 ′ = f k ( x k ′ , u k ′ , w k ′ ) = x k ′ + u k ′ w k x&#x27;_{k+1}=f_k(x&#x27;_k,u&#x27;_k,w&#x27;_k)=x&#x27;_k+u&#x27;_kw_k xk+1=fk(xk,uk,wk)=xk+ukwk
\qquad cost function is defined as g k ( x k ′ , u k ′ , w k ′ ) = − u k ′ v k g_k(x&#x27;_k,u&#x27;_k,w&#x27;_k)=-u&#x27;_kv_k gk(xk,uk,wk)=ukvk
So the problem is reformulated to get an optimal solution to the minimization problem min ⁡ π = { u 1 ′ , … , u N ′ } g k ( x k ′ , u k ′ , w k ′ ) \mathop{\min}\limits_{\pi=\{u&#x27;_1,\ldots,u&#x27;_N\}}g_k(x&#x27;_k,u&#x27;_k,w&#x27;_k) π={u1,,uN}mingk(xk,uk,wk)the problem can be solved by the following DP-algorithm:
J N + 1 ( x N + 1 ′ ) = 0 J_{N+1}(x&#x27;_{N+1})=0\qquad\qquad JN+1(xN+1)=0 J k ( x k ′ ) = min ⁡ u k ′ ∈ U k ( x k ′ ) { − u k ′ v k + J k + 1 ( f k ( x k ′ , u k ′ , w k ′ ) ) } &ThickSpace; , k = 1 , 2 , … , N J_k(x&#x27;_k)=\mathop{\min}\limits_{u&#x27;_k\in U_k(x&#x27;_k)}\{-u&#x27;_kv_k+J_{k+1}(f_k(x&#x27;_k,u&#x27;_k,w&#x27;_k))\}\;,\qquad k=1,2,\ldots,N Jk(xk)=ukUk(xk)min{ukvk+Jk+1(fk(xk,uk,wk))},k=1,2,,N □ \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\Box

1.11 Consider a device consisting of N N N stages connected in series, where each stage consists of a particular component. The components are subject to failure, and to increase the reliablity of the device duplicate components are provided. For j = 1 , 2 , … , N j=1,2,\ldots,N j=1,2,,N, let ( 1 + m j ) (1+m_j) (1+mj) be the number of components for the j j j-th stage, let p j ( m j ) p_j(m_j) pj(mj) be the probability of success operation of the j j j-th stage when ( 1 + m j ) (1+m_j) (1+mj) components are used, and let c j c_j cj denote the cost of a single component at the j j j-th stage. Formulate in terms of DP the problem of finding the number of components at each stage that maximize the realibility of the device expressed by p 1 ( m 1 ) ⋅ p 2 ( m 2 ) … p N ( m N ) p_1(m_1)\cdot p_2(m_2)\ldots p_N(m_N) p1(m1)p2(m2)pN(mN) subject to the cost constraint ∑ j = 1 N c j m j ≤ A \sum_{j=1}^N c_jm_j\leq A j=1NcjmjA, where A &gt; a A&gt;a A>a is given.

1.12 (Minimization over a Subset of Policies) This problem is primarily of theoretical interest. Consider a variant of the basic problem whereby we seek min ⁡ π ∈ Π ~ J k ( x 0 ) \mathop{\min}\limits_{\pi\in\tilde{\Pi}}J_k(x_0) πΠ~minJk(x0) where Π ~ \tilde{\Pi} Π~ is some given subset of the set of sequences { μ 0 , μ 1 , … , μ N − 1 } \{\mu_0,\mu_1,\ldots, \mu_{N-1}\} {μ0,μ1,,μN1} of functions μ k : S k → C k \mu_k:S_k\rightarrow C_k μk:SkCk with μ k ( x k ) ∈ U k ( x k ) \mu_k(x_k)\in U_k(x_k) μk(xk)Uk(xk) for all x k ∈ S k x_k\in S_k xkSk. Assume that π ∗ = { μ 0 ∗ , μ 1 ∗ , … , μ N − 1 ∗ } \pi^*=\{\mu_0^*,\mu_1^*,\ldots, \mu_{N-1}^*\} π={μ0,μ1,,μN1} belongs to Π ~ \tilde{\Pi} Π~ and attains the minimum in the DP algorithm; that is, for all k = 0 , 1 , … , N − 1 k=0,1,\ldots,N-1 k=0,1,,N1 and x k ∈ S k x_k\in S_k xkSk, J k ( x k ) = E w k { g k ( x k , μ k ∗ ( x k ) , w k ) + J k + 1 ( f k ( x k , μ k ∗ ( x k ) , w k ) ) } J_k(x_k)=\mathop{\mathbf{E}}\limits_{w_k}\left\{g_k(x_k,\mu_k^*(x_k),w_k)+J_{k+1}\left(f_k(x_k,\mu_k^*(x_k),w_k)\right)\right\} Jk(xk)=wkE{gk(xk,μk(xk),wk)+Jk+1(fk(xk,μk(xk),wk))} min ⁡ u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k ) + J k + 1 ( f k ( x k , u k , w k ) ) } \mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop{\mathbf{E}}\limits_{w_k}\left\{g_k(x_k,u_k,w_k)+J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\} ukUk(xk)minwkE{gk(xk,uk,wk)+Jk+1(fk(xk,uk,wk))} with J N ( x N ) = g N ( x N ) J_N(x_N)=g_N(x_N) JN(xN)=gN(xN). Assume further that the function J k J_k Jk are real-valued and that the preceding expectations are well-defined and finite. Show that π ∗ \pi^* π is optimal within Π ~ \tilde{\Pi} Π~ and that the corresponding optimal cost is equal to J 0 ( x 0 ) J_0(x_0) J0(x0).

1.13 (Semilinear Systems) Consider a problem involving the system x k + 1 = A k x k + f k ( u k ) + w k x_{k+1}=A_kx_k+f_k(u_k)+w_k xk+1=Akxk+fk(uk)+wk where x k ∈ R n x_k\in\mathfrak{R}^n xkRn, f k f_k fk are given functions, and A k A_k Ak and w k w_k wk are random n × n n\times n n×n matices and n n n-vectors, respectively, with given probability distributions that do not depend on x k , u k x_k, u_k xk,uk or prior values of A k A_k Ak and w k w_k wk. Assume that the cost is of the form E A k , w k k = 0 , 1 , … , N − 1 { c N ′ x n + ∑ k = 0 N − 1 ( c k ′ x k + g k ( μ k ( x k ) ) ) } \mathop{\mathbf{E}}\limits_{\mathop{A_k,w_k}\limits_{k=0,1,\ldots,N-1}}\left\{c&#x27;_Nx_n+\sum_{k=0}^{N-1}\left(c&#x27;_kx_k+g_k\left(\mu_k(x_k)\right)\right)\right\} k=0,1,,N1Ak,wkE{cNxn+k=0N1(ckxk+gk(μk(xk)))}where c k c_k ck are given vectors and g k g_k gk are given functions. Show that if the optimal cost for this problem is finite and the control constraint sets U k ( x k ) U_k(x_k) Uk(xk) are independent of x k x_k xk, then the cost-to-go functions of the DP algorithm are affine (linear plus constraint). Assuming that there is at least one optimal policy, show that there exists an optimal policy that consist of constant functions μ k ∗ \mu_k^* μk that is μ k ∗ ( x k ) = \mu_k^*(x_k)= μk(xk)= constant for all x k ∈ R n x_k\in\mathfrak{R}^n xkRn.

1.14 A farmer annually producing x k x_k xk units of a certain crop stores ( 1 − u k ) x k (1-u_k)x_k (1uk)xk units of his production, where 0 ≤ u k ≤ 1 0\leq u_k\leq 1 0uk1 and invests the remaining u k x k u_kx_k ukxk units, thus increasing the next year’s production to a level x k + 1 x_{k+1} xk+1 given by x k + 1 = x k + w k u k x k , k = 0 , 1 , ⋯ &ThinSpace; , N − 1 x_{k+1}=x_k+w_ku_kx_k,\qquad k=0,1,\cdots,N-1 xk+1=xk+wkukxk,k=0,1,,N1The scalars w k w_k wk are independent random variables with identical probability distributions that do not depend either on x k x_k xk or u k u_k uk. Furthermore, E { w k } = w ‾ &gt; 0 \mathbf{E}\{w_k\}=\overline{w}&gt;0 E{wk}=w>0. The problem is to find the optimal investment policy that maximizes the total expected product stored over N N N years E w k k = 0 , 1 , … , N − 1 { x N + ∑ k = 0 N − 1 ( 1 − u k ) x k } \mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,\ldots,N-1}}\left\{x_N+\sum_{k=0}^{N-1}(1-u_k)x_k\right\} k=0,1,,N1wkE{xN+k=0N1(1uk)xk} Show the optimality of the following policy that consists of constant functions:
(1) If w ‾ &gt; 1 \overline{w}&gt;1 w>1, μ 0 ∗ ( x 0 ) = ⋯ = μ N − 1 ∗ ( x N − 1 ) = 1 \mu_0^*(x_0)=\cdots=\mu_{N-1}^*(x_{N-1})=1 μ0(x0)==μN1(xN1)=1.
(2) If 0 &lt; w ‾ &lt; 1 / N 0&lt;\overline{w}&lt;1/N 0<w<1/N, μ 0 ∗ ( x 0 ) = ⋯ = μ N − 1 ∗ ( x N − 1 ) = 0 \mu_0^*(x_0)=\cdots=\mu_{N-1}^*(x_{N-1})=0 μ0(x0)==μN1(xN1)=0.
(3) If 1 / N ≤ w ‾ ≤ 1 1/N\leq\overline{w}\leq 1 1/Nw1, μ 0 ∗ ( x 0 ) = ⋯ = μ N − k ˉ − 1 ∗ ( x N − k ˉ − 1 ) = 1 \mu_0^*(x_0)=\cdots=\mu_{N-\bar{k}-1}^*(x_{N-\bar{k}-1})=1 μ0(x0)==μNkˉ1(xNkˉ1)=1 μ N − k ˉ ∗ ( x N − k ˉ ) = ⋯ = μ N − 1 ∗ ( x N − 1 ) = 0 \mu_{N-\bar{k}}^*(x_{N-\bar{k}})=\cdots=\mu_{N-1}^*(x_{N-1})=0 μNkˉ(xNkˉ)==μN1(xN1)=0 where k ˉ \bar{k} kˉ is such that 1 / ( k ˉ + 1 ) &lt; w ‾ ≤ 1 / k ˉ 1/{(\bar{k}+1)}&lt;\overline{w}\leq 1/{\bar{k}} 1/(kˉ+1)<w1/kˉ.

Solution. The DP-algorithm can be formed as follows: J N ( x N ) = x N J_N(x_N)=x_N JN(xN)=xN J k ( x k ) = max ⁡ u k E w k { ( 1 − u k ) x k + J k + 1 ( x k + w k u k x k ) } &ThinSpace; , &ThickSpace;&ThickSpace; k = 0 , 1 , ⋯ &ThinSpace; , N − 1 J_k(x_k)=\mathop{\max}\limits_{u_k}\mathop{\mathbf E}\limits_{w_k}\left\{(1-u_k)x_k+J_{k+1}(x_k+w_ku_kx_k)\right\}\,,\;\; k=0,1,\cdots,N-1 Jk(xk)=ukmaxwkE{(1uk)xk+Jk+1(xk+wkukxk)},k=0,1,,N1 (1) In the case w ‾ &gt; 1 \overline{w}&gt;1 w>1 J N − 1 ( x N − 1 ) = max ⁡ u N − 1 E w N − 1 { ( 1 − u N − 1 ) x N − 1 + x N − 1 + w N − 1 u N − 1 x N − 1 } J_{N-1}(x_{N-1})=\mathop{\max}\limits_{u_{N-1}}\mathop{\mathbf E}\limits_{w_{N-1}}\left\{(1-u_{N-1})x_{N-1}+x_{N-1}+w_{N-1}u_{N-1}x_{N-1}\right\} JN1(xN1)=uN1maxwN1E{(1uN1)xN1+xN1+wN1uN1xN1} = max ⁡ u N − 1 E w N − 1 { 2 x N − 1 + ( w N − 1 − 1 ) u N − 1 x N − 1 } =\mathop{\max}\limits_{u_{N-1}}\mathop{\mathbf E}\limits_{w_{N-1}}\left\{2x_{N-1}+(w_{N-1}-1)u_{N-1}x_{N-1}\right\} =uN1maxwN1E{2xN1+(wN11)uN1xN1} = max ⁡ u N − 1 { 2 x N − 1 + ( w ‾ − 1 ) u N − 1 x N − 1 } =\mathop{\max}\limits_{u_{N-1}}\left\{2x_{N-1}+(\overline{w}-1)u_{N-1}x_{N-1}\right\}\qquad\qquad =uN1max{2xN1+(w1)uN1xN1} we obtain that μ N − 1 ∗ ( x N − 1 ) = 1 \mu_{N-1}^*(x_{N-1})=1 μN1(xN1)=1 and J N − 1 ( x N − 1 ) = ( w ‾ + 1 ) x N − 1 J_{N-1}(x_{N-1})=(\overline{w}+1)x_{N-1} JN1(xN1)=(w+1)xN1. By indunction and similar caculations, we can get μ k ∗ ( x k ) = 1 &ThickSpace; , &ThickSpace; J k ( x k ) = ( w ‾ + N − k ) x k &ThickSpace;&ThickSpace; for all  &ThickSpace; k = 0 , 1 , ⋯ &ThinSpace; , N − 1 \mu_k^*(x_k)=1\;,\;J_k(x_k)=(\overline{w}+N-k)x_k\;\;\text{for all }\;k=0,1,\cdots,N-1 μk(xk)=1,Jk(xk)=(w+Nk)xkfor all k=0,1,,N1(2) In the case w ‾ &lt; 1 / N \overline{w}&lt;1/N w<1/N,
J N − 1 ( x N − 1 ) = max ⁡ u N − 1 { 2 x N − 1 + ( w ‾ − 1 ) u N − 1 x N − 1 } = 2 x N − 1 J_{N-1}(x_{N-1})=\mathop{\max}\limits_{u_{N-1}}\left\{2x_{N-1}+(\overline{w}-1)u_{N-1}x_{N-1}\right\}=2x_{N-1} JN1(xN1)=uN1max{2xN1+(w1)uN1xN1}=2xN1 with μ N − 1 ∗ ( x N − 1 ) = 0 \mu_{N-1}^*(x_{N-1})=0 μN1(xN1)=0. And then
J N − 2 ( x N − 2 ) = max ⁡ u N − 2 E w N − 2 { ( 1 − u N − 2 ) x N − 2 + 2 [ x N − 2 + w N − 2 u N − 2 x N − 2 ] } J_{N-2}(x_{N-2})=\mathop{\max}\limits_{u_{N-2}}\mathop{\mathbf E}\limits_{w_{N-2}}\left\{(1-u_{N-2})x_{N-2}+2[x_{N-2}+w_{N-2}u_{N-2}x_{N-2}]\right\} JN2(xN2)=uN2maxwN2E{(1uN2)xN2+2[xN2+wN2uN2xN2]} = max ⁡ u N − 2 { 3 x N − 2 + ( 2 w ‾ − 1 ) u N − 2 x N − 2 } = 3 x N − 2 =\mathop{\max}\limits_{u_{N-2}}\left\{3x_{N-2}+(2\overline{w}-1)u_{N-2}x_{N-2}\right\}=3x_{N-2} =uN2max{3xN2+(2w1)uN2xN2}=3xN2 with μ N − 2 ∗ ( x N − 2 ) = 0 \mu_{N-2}^*(x_{N-2})=0 μN2(xN2)=0. Since w ‾ &lt; 1 / N \overline{w}&lt;1/N w<1/N, we can prove by induction that for all k = 0 , 1 , ⋯ &ThinSpace; , N − 1 k=0,1,\cdots,N-1 k=0,1,,N1, μ k ∗ ( x k ) = 0 &ThickSpace; , &ThickSpace; J k ( x k ) = ( N − k + 1 ) x k \mu_k^*(x_k)=0\;,\;J_k(x_k)=(N-k+1)x_k μk(xk)=0,Jk(xk)=(Nk+1)xk (3) In the case 1 / N ≤ w ‾ ≤ 1 1/N\leq\overline{w}\leq 1 1/Nw1, there is some k = k ˉ k=\bar{k} k=kˉ satisfying 1 / ( k ˉ + 1 ) &lt; w ‾ ≤ 1 / k ˉ 1/{(\bar{k}+1)}&lt;\overline{w}\leq 1/{\bar{k}} 1/(kˉ+1)<w1/kˉ, after which step the induction argument in (2) will stop.
For any k ≥ N − k ˉ k\ge N-\bar{k} kNkˉ, we have J k ( x k ) = max ⁡ u k { ( N − k + 1 ) x k + ( ( N − k ) w ‾ − 1 ) u k x k } J_k(x_k)=\mathop{\max}\limits_{u_k}\left\{(N-k+1)x_k+((N-k)\overline{w}-1)u_kx_k\right\} Jk(xk)=ukmax{(Nk+1)xk+((Nk)w1)ukxk}Since ( N − k ) w ‾ ≤ k ˉ w ‾ ≤ 1 (N-k)\overline{w}\leq\bar{k}\overline{w}\leq 1 (Nk)wkˉw1, then μ k ∗ ( x k ) = 0 \mu_k^*(x_k)=0 μk(xk)=0 and J k ( x k ) = ( N − k + 1 ) x k J_k(x_k)=(N-k+1)x_k Jk(xk)=(Nk+1)xk.
Next for k = N − k ˉ − 1 k= N-\bar{k}-1 k=Nkˉ1
J N − k ˉ − 1 ( x N − k ˉ − 1 ) = max ⁡ u N − k ˉ − 1 { ( k ˉ + 2 ) x N − k ˉ − 1 + ( ( k ˉ + 1 ) w ‾ − 1 ) u N − k ˉ − 1 x N − k ˉ − 1 } J_{N-\bar{k}-1}(x_{N-\bar{k}-1})=\mathop{\max}\limits_{u_{N-\bar{k}-1}}\left\{(\bar{k}+2)x_{N-\bar{k}-1}+((\bar{k}+1)\overline{w}-1)u_{N-\bar{k}-1}x_{N-\bar{k}-1}\right\} JNkˉ1(xNkˉ1)=uNkˉ1max{(kˉ+2)xNkˉ1+((kˉ+1)w1)uNkˉ1xNkˉ1} In this case, ( k ˉ + 1 ) w ‾ − 1 &gt; 0 (\bar{k}+1)\overline{w}-1&gt;0 (kˉ+1)w1>0, yielding μ N − k ˉ − 1 ∗ ( x N − k ˉ − 1 ) = 1 \mu_{N-\bar{k}-1}^*(x_{N-\bar{k}-1})=1 μNkˉ1(xNkˉ1)=1 and J N − k ˉ − 1 ( x N − k ˉ − 1 ) = [ ( k ˉ + 1 ) w ‾ + k ˉ + 1 ] x N − k ˉ − 1 J_{N-\bar{k}-1}(x_{N-\bar{k}-1})=[(\bar{k}+1)\overline{w}+\bar{k}+1]x_{N-\bar{k}-1} JNkˉ1(xNkˉ1)=[(kˉ+1)w+kˉ+1]xNkˉ1.
And then by induction, we can prove that for all k &lt; N − k ˉ − 1 k&lt;N-\bar{k}-1 k<Nkˉ1, μ k ∗ ( x k ) = 1 \mu_k^*(x_k)=1 μk(xk)=1. □ \qquad\qquad\qquad\qquad\Box

1.15 Let x k x_k xk denote the number of educators in a certain country at time k k k and let y k y_k yk denote the number of research scientists at time k k k. New scientists (potential educators or research scientists) are produced during the k k kth period by educators at a rate γ k \gamma_k γk per educator, while educators and research scientists leave the field due to death, retirement, and transfer at a rate δ k \delta_k δk. The scalars γ k \gamma_k γk, k = 0 , 1 , . . . , N − 1 k = 0,1, ... ,N-1 k=0,1,...,N1, are independent identically distributed random variables taking values within a closed and bounded interval of positive numbers. Similarly δ k \delta_k δk, k = 0 , 1 , . . . , N − 1 k = 0,1, ... ,N-1 k=0,1,...,N1, are independent identically distributed and take values in an interval [ δ , δ ′ ] [\delta,\delta&#x27;] [δ,δ] with 0 &lt; δ ≤ δ ′ &lt; 1 0&lt;\delta\leq\delta&#x27;&lt;1 0<δδ<1. By means of incentives, a science policy maker can determine the proportion u k u_k uk of new scientists produced at time k k k who become educators. Thus, the number of research scientists and educators evolves according to the equations x k + 1 = ( 1 − δ k ) x k + u k γ k x k x_{k+1}=(1-\delta_k)x_k+u_k\gamma_k x_k xk+1=(1δk)xk+ukγkxk y k + 1 = ( 1 − δ k ) y k + ( 1 − u k ) γ k x k y_{k+1}=(1-\delta_k)y_k+(1-u_k)\gamma_k x_k yk+1=(1δk)yk+(1uk)γkxk The initial numbers x 0 , y 0 x_0,y_0 x0,y0 are known, and it is required to find a policy { μ 0 ∗ ( x 0 , y 0 ) , ⋯ &ThinSpace; , μ N − 1 ∗ ( x N − 1 , y N − 1 ) } \{\mu_0^*(x_0,y_0),\cdots,\mu_{N-1}^*(x_{N-1},y_{N-1})\} {μ0(x0,y0),,μN1(xN1,yN1)} with 0 &lt; α ≤ μ k ∗ ( x k , y k ) ≤ β &lt; 1 , for all  x k , y k  and  k 0&lt;\alpha\leq \mu_k^*(x_k,y_k)\leq\beta&lt;1,\qquad\text{for all }x_k,y_k\text{ and }k 0<αμk(xk,yk)β<1,for all xk,yk and k which maximizes E γ k , δ k { y N } \mathbf{E}_{\gamma_k,\delta_k}\{y_N\} Eγk,δk{yN} (i.e. the expected final number of research scientists after N N N periods). The scalars α \alpha α and β \beta β are given.
(1) Show that the cost-to-go functions J k ( x k , y k ) J_k(x_k,y_k) Jk(xk,yk) are linear; that is, for some scalars ξ k , ζ k , \xi_k,\zeta_k, ξk,ζk, J k ( x k , y k ) = ξ k x k + ζ k y k J_k(x_k,y_k)=\xi_kx_k+\zeta_ky_k Jk(xk,yk)=ξkxk+ζkyk (2) Derive an optimal policy { μ 0 ∗ , ⋯ &ThinSpace; , μ N − 1 ∗ } \{\mu_0^*,\cdots,\mu_{N-1}^*\} {μ0,,μN1}, under the assumption E { γ k } &gt; E { δ k } \mathbf{E}\{\gamma_k\}&gt;\mathbf{E}\{\delta_k\} E{γk}>E{δk} and show that this optimal policy can consist of constant functions.
(3) Assume that the proportion of new scientists who become educators at time k k k is u k + ϵ k u_k+\epsilon_k uk+ϵk (rather than u k u_k uk), where ϵ k \epsilon_k ϵk are identically distributed independent random variables that are also independent of γ k , δ k \gamma_k,\delta_k γk,δk and take values in the interval [ − α , 1 − β ] [-\alpha,1-\beta] [α,1β]. Derive the form of the cost-to-go functions and the optimal policy.

  • 4
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 2
    评论
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

zte10096334

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值