1.1 Consider the system
x
k
+
1
=
x
k
+
u
k
+
w
k
,
k
=
0
,
1
,
2
,
3
x_{k+1}=x_k+u_k+w_k,\qquad k=0,1,2,3
xk+1=xk+uk+wk,k=0,1,2,3with initial state
x
0
=
5
x_0=5
x0=5, and the cost function
∑
k
=
0
3
(
x
k
2
+
u
k
2
)
\sum_{k=0}^3(x_k^2+u_k^2)
k=0∑3(xk2+uk2)Apply the DP algorithm for the following three cases:
(a)The control constraint set
U
k
(
x
k
)
U_k(x_k)
Uk(xk) is
{
u
∣
0
≤
x
k
+
u
≤
5
,
u
: integer
}
\{u|0\leq x_k+u\leq 5, u \text{: integer}\}
{u∣0≤xk+u≤5,u: integer} for all
x
k
x_k
xk and
k
k
k, and the disturbance
w
k
w_k
wk is equal to zero for all
k
k
k.
(b) The control constraint and the disturbance
w
k
w_k
wk are as in (a), but there is in addition a constraint
x
4
=
5
x_4=5
x4=5 on the final state.
(c)The control constraint is as in part (a) and the disturbance
w
k
w_k
wk takes the values
−
1
-1
−1 and
1
1
1 with equal probability
1
/
2
1/2
1/2 for all
x
k
x_k
xk and
u
k
u_k
uk, except if
x
k
+
u
k
x_k+u_k
xk+uk is equal to
0
0
0 or
5
5
5, in which case
w
k
=
0
w_k=0
wk=0 with probability
1
1
1.
Solution.
(a)
N
=
4
N=4
N=4, we assume that
J
4
(
x
4
)
=
g
4
(
x
4
)
=
0
J_4(x_4)=g_4(x_4)=0
J4(x4)=g4(x4)=0. As the condition, the system equation takes the form
f
k
+
1
=
x
k
+
u
k
+
w
k
,
k
=
0
,
1
,
2
,
3
f_{k+1}=x_k+u_k+w_k,\qquad k=0,1,2,3
fk+1=xk+uk+wk,k=0,1,2,3and the cost function is
g
k
=
x
k
2
+
u
k
2
g_k=x_k^2+u_k^2
gk=xk2+uk2In this case,
w
k
=
0
w_k=0
wk=0 for all
k
k
k, so
J
3
(
x
3
)
=
min
u
3
∈
U
3
(
x
3
)
E
w
3
{
g
3
(
x
3
,
u
3
,
w
3
)
+
J
4
(
f
3
(
x
3
,
u
3
,
w
3
)
)
}
J_3(x_3)=\mathop{\min}\limits_{u_3\in U_3(x_3)} \mathop\mathbf{E}\limits_{w_3}\{g_3(x_3,u_3,w_3)+J_4(f_3(x_3,u_3,w_3))\}
J3(x3)=u3∈U3(x3)minw3E{g3(x3,u3,w3)+J4(f3(x3,u3,w3))}
=
min
u
3
=
−
x
3
,
1
−
x
3
,
.
.
.
,
5
−
x
3
(
x
3
2
+
u
3
2
)
=
x
3
2
=\mathop{\min}\limits_{u_3=-x_3,1-x_3,...,5-x_3}(x_3^2+u_3^2)=x_3^2\qquad\qquad\qquad
=u3=−x3,1−x3,...,5−x3min(x32+u32)=x32i.e.
J
3
(
x
3
)
=
x
3
2
,
μ
3
∗
(
x
3
)
=
0.
(
1
)
\qquad J_3(x_3)=x_3^2, \mu_3^*(x_3)=0.\qquad\qquad\qquad\qquad\qquad\qquad (1)
J3(x3)=x32,μ3∗(x3)=0.(1)
Then
J
2
(
x
2
)
=
min
u
2
=
−
x
2
,
1
−
x
2
,
.
.
.
,
5
−
x
2
(
x
2
2
+
u
2
2
+
(
x
2
+
u
2
)
2
)
J_2(x_2)=\mathop{\min}\limits_{u_2=-x_2,1-x_2,...,5-x_2}(x_2^2+u_2^2+(x_2+u_2)^2)
J2(x2)=u2=−x2,1−x2,...,5−x2min(x22+u22+(x2+u2)2)Let
h
x
2
(
u
2
)
=
x
2
2
+
u
2
2
+
(
x
2
+
u
2
)
2
h_{x_2}(u_2)=x_2^2+u_2^2+(x_2+u_2)^2
hx2(u2)=x22+u22+(x2+u2)2, we have
h
x
2
′
(
u
2
)
=
4
u
2
+
2
x
2
.
h'_{x_2}(u_2)=4u_2+2x_2.
hx2′(u2)=4u2+2x2.
If x 2 > 10 x_2>10 x2>10, h x 2 ( u 2 ) = 4 ( u 2 + x 2 ) − 2 x 2 < 4 ( u 2 + x 2 ) − 20 < 0 h_{x_2}(u_2)=4(u_2+x_2)-2x_2 < 4(u_2+x_2)-20 < 0 hx2(u2)=4(u2+x2)−2x2<4(u2+x2)−20<0, then μ x ∗ ( x 2 ) = 5 − x 2    ,    J x ( x 2 ) = x 2 2 + ( 5 − x 2 ) 2 + 5 2 = 2 x 2 2 − 10 x 2 + 50 \mu_x^*(x_2)=5-x_2\;,\;J_x(x_2)=x_2^2+(5-x_2)^2+5^2=2x_2^2-10x_2+50 μx∗(x2)=5−x2,Jx(x2)=x22+(5−x2)2+52=2x22−10x2+50
If x 2 < 0 x_2<0 x2<0, h x 2 ( u 2 ) = 4 ( u 2 + x 2 ) − 2 x 2 > − 2 x 2 > 0 h_{x_2}(u_2)=4(u_2+x_2)-2x_2>-2x_2>0 hx2(u2)=4(u2+x2)−2x2>−2x2>0, then μ x 2 ( x 2 ) = − x 2    ,    J 2 ( x 2 ) = 2 x 2 2 \mu_x^2(x_2)=-x_2\;,\;J_2(x_2)=2x_2^2 μx2(x2)=−x2,J2(x2)=2x22
If 0 ≤ x 2 ≤ 10 0\leq x_2\leq 10 0≤x2≤10 and x 2 x_2 x2 is even, then μ 2 ∗ ( x 2 ) = − x 2 2 \mu_2^*(x_2)=-\frac{x_2}{2} μ2∗(x2)=−2x2 and J 2 ( x 2 ) = 3 2 x 2 2 . J_2(x_2) = \frac{3}{2}x_2^2. J2(x2)=23x22.
If 0 ≤ x 2 ≤ 10 0\leq x_2\leq 10 0≤x2≤10 and x 2 x_2 x2 is odd, then μ 2 ∗ ( x 2 ) = − x 2 − 1 2    or    − x 2 + 1 2 \mu_2^*(x_2)=-\frac{x_2-1}{2}\;\text{or}\;-\frac{x_2+1}{2} μ2∗(x2)=−2x2−1or−2x2+1 and J 2 ( x 2 ) = 3 2 x 2 2 + 1 2 . J_2(x_2) = \frac{3}{2}x_2^2+\frac{1}{2}. J2(x2)=23x22+21.
So
{
μ
x
2
(
x
2
)
=
−
x
2
  
,
  
J
2
(
x
2
)
=
2
x
2
2
if
  
x
2
<
0
μ
2
∗
(
x
2
)
=
−
x
2
2
  
,
  
J
2
(
x
2
)
=
3
2
x
2
2
if
  
0
≤
x
2
≤
10
  
and
  
x
2
  
is even
μ
2
∗
(
x
2
)
=
−
x
2
−
1
2
  
or
  
−
x
2
+
1
2
  
,
  
J
2
(
x
2
)
=
3
2
x
2
2
+
1
2
if
  
0
≤
x
2
≤
10
  
and
  
x
2
  
is odd
μ
x
∗
(
x
2
)
=
5
−
x
2
  
,
  
J
x
(
x
2
)
=
x
2
2
+
(
5
−
x
2
)
2
+
5
2
=
2
x
2
2
−
10
x
2
+
50
if
  
x
2
>
10
\begin{cases} \mu_x^2(x_2)=-x_2\;,\;J_2(x_2)=2x_2^2 & \qquad\text{if }\; x_2<0 \\ \mu_2^*(x_2)=-\frac{x_2}{2}\;,\;J_2(x_2) = \frac{3}{2}x_2^2 & \qquad\text{if }\;0\leq x_2\leq 10 \;\text{and} \;x_2\; \text{is even}\\ \mu_2^*(x_2)=-\frac{x_2-1}{2}\;\text{or}\;-\frac{x_2+1}{2}\;,\;J_2(x_2) = \frac{3}{2}x_2^2+\frac{1}{2} & \qquad\text{if }\;0\leq x_2\leq 10 \;\text{and} \;x_2\; \text{is odd}\\ \mu_x^*(x_2)=5-x_2\;,\;J_x(x_2)=x_2^2+(5-x_2)^2+5^2=2x_2^2-10x_2+50 & \qquad\text{if }\; x_2>10 \end{cases}
⎩⎪⎪⎪⎨⎪⎪⎪⎧μx2(x2)=−x2,J2(x2)=2x22μ2∗(x2)=−2x2,J2(x2)=23x22μ2∗(x2)=−2x2−1or−2x2+1,J2(x2)=23x22+21μx∗(x2)=5−x2,Jx(x2)=x22+(5−x2)2+52=2x22−10x2+50if x2<0if 0≤x2≤10andx2is evenif 0≤x2≤10andx2is oddif x2>10
In case
k
=
1
k=1
k=1
J
1
(
x
1
)
=
min
u
1
=
−
x
1
,
1
−
x
1
,
.
.
.
,
5
−
x
1
(
x
1
2
+
u
1
2
+
J
2
(
x
1
+
u
1
)
)
      
J_1(x_1)=\mathop{\min}\limits_{u_1=-x_1,1-x_1,...,5-x_1}(x_1^2+u_1^2+J_2(x_1+u_1))\qquad\qquad\qquad\qquad\qquad\;\;\;
J1(x1)=u1=−x1,1−x1,...,5−x1min(x12+u12+J2(x1+u1))
=
min
u
1
=
−
x
1
,
1
−
x
1
,
.
.
.
,
5
−
x
1
(
x
1
2
+
u
1
2
+
3
2
(
x
1
+
u
1
)
2
+
1
2
⋅
1
x
1
+
u
1
  
is odd
)
=\mathop{\min}\limits_{u_1=-x_1,1-x_1,...,5-x_1}\big(x_1^2+u_1^2+\frac{3}{2}(x_1+u_1)^2+\frac{1}{2}\cdot\mathbf{1}_{x_1+u_1\;\text{is odd}}\big)
=u1=−x1,1−x1,...,5−x1min(x12+u12+23(x1+u1)2+21⋅1x1+u1is odd) In this case, we give an caculation in a direct way:
Let
h
x
1
(
u
1
)
=
x
1
2
+
u
1
2
+
3
2
(
x
1
+
u
1
)
2
+
1
2
⋅
1
x
1
+
u
1
  
is odd
h_{x_1}(u_1)=x_1^2+u_1^2+\frac{3}{2}(x_1+u_1)^2+\frac{1}{2}\cdot\mathbf{1}_{x_1+u_1\;\text{is odd}}
hx1(u1)=x12+u12+23(x1+u1)2+21⋅1x1+u1is odd
h
x
1
(
−
x
1
)
=
2
x
1
2
h_{x_1}(-x_1)=2x_1^2
hx1(−x1)=2x12
h
x
1
(
1
−
x
1
)
=
2
x
1
2
−
2
x
1
+
3
h_{x_1}(1-x_1)=2x_1^2-2x_1+3
hx1(1−x1)=2x12−2x1+3
h
x
1
(
2
−
x
1
)
=
2
x
1
2
−
4
x
1
+
10
h_{x_1}(2-x_1)=2x_1^2-4x_1+10
hx1(2−x1)=2x12−4x1+10
h
x
1
(
3
−
x
1
)
=
2
x
1
2
−
6
x
1
+
23
h_{x_1}(3-x_1)=2x_1^2-6x_1+23
hx1(3−x1)=2x12−6x1+23
h
x
1
(
4
−
x
1
)
=
2
x
1
2
−
8
x
1
+
40
h_{x_1}(4-x_1)=2x_1^2-8x_1+40
hx1(4−x1)=2x12−8x1+40
h
x
1
(
5
−
x
1
)
=
2
x
1
2
−
10
x
1
+
63
h_{x_1}(5-x_1)=2x_1^2-10x_1+63
hx1(5−x1)=2x12−10x1+63Then we can get the following result easily
{
μ
1
∗
(
x
1
)
=
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
if
  
x
1
≤
3
2
μ
1
∗
(
x
1
)
=
1
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
−
2
x
1
+
3
if
  
3
2
<
x
1
≤
7
2
μ
1
∗
(
x
1
)
=
2
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
−
4
x
1
+
10
if
  
7
2
<
x
1
≤
13
2
μ
1
∗
(
x
1
)
=
3
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
−
6
x
1
+
23
if
  
13
2
<
x
1
≤
17
2
μ
1
∗
(
x
1
)
=
4
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
−
8
x
1
+
40
if
  
17
2
<
x
1
≤
23
2
μ
1
∗
(
x
1
)
=
5
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
−
10
x
1
+
63
if
  
x
1
>
23
2
\begin{cases} \mu_1^*(x_1)=-x_1\;,\;J_1(x_1)=2x_1^2 & \qquad\text{if }\; x_1\leq\frac{3}{2} \\ \mu_1^*(x_1)=1-x_1\;,\;J_1(x_1)=2x_1^2-2x_1+3 & \qquad\text{if }\; \frac{3}{2}<x_1\leq\frac{7}{2} \\ \mu_1^*(x_1)=2-x_1\;,\;J_1(x_1)=2x_1^2-4x_1+10 & \qquad\text{if }\; \frac{7}{2}<x_1\leq\frac{13}{2} \\ \mu_1^*(x_1)=3-x_1\;,\;J_1(x_1)=2x_1^2-6x_1+23 & \qquad\text{if }\; \frac{13}{2}<x_1\leq\frac{17}{2} \\ \mu_1^*(x_1)=4-x_1\;,\;J_1(x_1)=2x_1^2-8x_1+40 & \qquad\text{if }\; \frac{17}{2}<x_1\leq\frac{23}{2} \\ \mu_1^*(x_1)=5-x_1\;,\;J_1(x_1)=2x_1^2-10x_1+63 & \qquad\text{if }\; x_1 >\frac{23}{2} \\ \end{cases}
⎩⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎧μ1∗(x1)=−x1,J1(x1)=2x12μ1∗(x1)=1−x1,J1(x1)=2x12−2x1+3μ1∗(x1)=2−x1,J1(x1)=2x12−4x1+10μ1∗(x1)=3−x1,J1(x1)=2x12−6x1+23μ1∗(x1)=4−x1,J1(x1)=2x12−8x1+40μ1∗(x1)=5−x1,J1(x1)=2x12−10x1+63if x1≤23if 23<x1≤27if 27<x1≤213if 213<x1≤217if 217<x1≤223if x1>223
For case
k
=
0
 
,
  
x
0
=
5
k=0\,,\; x_0=5
k=0,x0=5,
J
0
(
x
0
)
=
min
u
0
=
−
x
0
,
1
−
x
0
,
.
.
.
,
5
−
x
0
(
x
0
2
+
u
0
2
+
J
1
(
x
0
+
u
0
)
)
=
min
u
0
=
−
5
,
−
4
,
.
.
.
,
0
(
25
+
u
0
2
+
J
1
(
5
+
u
0
)
)
J_0(x_0)=\mathop{\min}\limits_{u_0=-x_0,1-x_0,...,5-x_0}(x_0^2+u_0^2+J_1(x_0+u_0))=\mathop{\min}\limits_{u_0=-5,-4,...,0}(25+u_0^2+J_1(5+u_0))
J0(x0)=u0=−x0,1−x0,...,5−x0min(x02+u02+J1(x0+u0))=u0=−5,−4,...,0min(25+u02+J1(5+u0))Let
h
0
(
u
0
)
=
25
+
u
0
2
+
J
1
(
5
+
u
0
)
h_0(u_0)=25+u_0^2+J_1(5+u_0)
h0(u0)=25+u02+J1(5+u0)we have
h
0
(
−
5
)
=
50
,
  
h
0
(
−
4
)
=
43
,
  
h
0
(
−
3
)
=
41
,
  
h
0
(
−
2
)
=
44
,
  
h
0
(
−
1
)
=
50
,
  
h
0
(
0
)
=
65
h_0(-5)=50,\; h_0(-4)=43,\;h_0(-3)=41,\;h_0(-2)=44,\;h_0(-1)=50,\;h_0(0)=65
h0(−5)=50,h0(−4)=43,h0(−3)=41,h0(−2)=44,h0(−1)=50,h0(0)=65So
μ
0
∗
(
x
0
)
=
−
3
\mu_0^*(x_0)=-3
μ0∗(x0)=−3 and
J
0
(
x
0
)
=
41
J_0(x_0)=41
J0(x0)=41.
(b) In this case ,we still asume that
J
4
(
x
4
)
=
g
4
(
x
4
)
=
0
J_4(x_4)=g_4(x_4)=0
J4(x4)=g4(x4)=0. By the additional condition,
U
3
(
x
3
)
=
{
5
−
x
3
}
U_3(x_3)=\{5-x_3\}
U3(x3)={5−x3}, and
J
3
(
x
3
)
=
min
u
3
∈
U
3
(
x
3
)
(
x
3
2
+
u
3
2
)
=
x
3
2
+
(
5
−
x
3
)
2
J_3(x_3)=\mathop{\min}\limits_{u_3\in U_3(x_3)}(x_3^2+u_3^2)=x_3^2+(5-x_3)^2
J3(x3)=u3∈U3(x3)min(x32+u32)=x32+(5−x3)2Then
J
2
(
x
2
)
=
min
u
2
=
−
x
2
,
1
−
x
2
,
.
.
.
,
5
−
x
2
(
x
2
2
+
u
2
2
+
(
5
−
x
2
−
u
2
)
2
)
J_2(x_2)=\mathop{\min}\limits_{u_2=-x_2,1-x_2,...,5-x_2}(x_2^2+u_2^2+(5-x_2-u_2)^2)
J2(x2)=u2=−x2,1−x2,...,5−x2min(x22+u22+(5−x2−u2)2)Let
h
x
2
(
u
2
)
=
x
2
2
+
u
2
2
+
(
5
−
x
2
−
u
2
)
2
h_{x_2}(u_2)=x_2^2+u_2^2+(5-x_2-u_2)^2
hx2(u2)=x22+u22+(5−x2−u2)2, take some easy caculation
h
x
2
(
−
x
2
)
=
2
x
2
2
+
25
h_{x_2}(-x_2)=2x_2^2+25
hx2(−x2)=2x22+25
h
x
2
(
1
−
x
2
)
=
2
x
2
2
−
2
x
2
+
18
h_{x_2}(1-x_2)=2x_2^2-2x_2+18
hx2(1−x2)=2x22−2x2+18
h
x
2
(
2
−
x
2
)
=
2
x
2
2
−
4
x
1
+
17
h_{x_2}(2-x_2)=2x_2^2-4x_1+17
hx2(2−x2)=2x22−4x1+17
h
x
2
(
3
−
x
2
)
=
2
x
2
2
−
6
x
2
+
22
h_{x_2}(3-x_2)=2x_2^2-6x_2+22
hx2(3−x2)=2x22−6x2+22
h
x
2
(
4
−
x
2
)
=
2
x
2
2
−
8
x
2
+
33
h_{x_2}(4-x_2)=2x_2^2-8x_2+33
hx2(4−x2)=2x22−8x2+33
h
x
2
(
5
−
x
2
)
=
2
x
2
2
−
10
x
2
+
50
h_{x_2}(5-x_2)=2x_2^2-10x_2+50
hx2(5−x2)=2x22−10x2+50and we get
{
μ
2
∗
(
x
2
)
=
−
x
2
  
,
  
J
2
(
x
2
)
=
2
x
2
2
+
25
if
  
x
1
≤
−
7
2
μ
2
∗
(
x
2
)
=
1
−
x
2
  
,
  
J
2
(
x
2
)
=
2
x
2
2
−
2
x
2
+
18
if
  
−
7
2
<
x
1
≤
−
1
2
μ
2
∗
(
x
2
)
=
2
−
x
2
  
,
  
J
2
(
x
2
)
=
2
x
2
2
−
4
x
2
+
17
if
  
−
1
2
<
x
1
≤
5
2
μ
2
∗
(
x
2
)
=
3
−
x
2
  
,
  
J
2
(
x
2
)
=
2
x
2
2
−
6
x
2
+
22
if
  
5
2
<
x
1
≤
11
2
μ
2
∗
(
x
2
)
=
4
−
x
2
  
,
  
J
2
(
x
2
)
=
2
x
2
2
−
8
x
2
+
33
if
  
11
2
<
x
1
≤
17
2
μ
2
∗
(
x
2
)
=
5
−
x
2
  
,
  
J
2
(
x
2
)
=
2
x
2
2
−
10
x
2
+
50
if
  
x
1
>
17
2
\begin{cases} \mu_2^*(x_2)=-x_2\;,\;J_2(x_2)=2x_2^2+25 & \qquad\text{if }\; x_1\leq-\frac{7}{2} \\ \mu_2^*(x_2)=1-x_2\;,\;J_2(x_2)=2x_2^2-2x_2+18 & \qquad\text{if }\;- \frac{7}{2}<x_1\leq-\frac{1}{2} \\ \mu_2^*(x_2)=2-x_2\;,\;J_2(x_2)=2x_2^2-4x_2+17 & \qquad\text{if }\; -\frac{1}{2}<x_1\leq\frac{5}{2} \\ \mu_2^*(x_2)=3-x_2\;,\;J_2(x_2)=2x_2^2-6x_2+22 & \qquad\text{if }\; \frac{5}{2}<x_1\leq\frac{11}{2} \\ \mu_2^*(x_2)=4-x_2\;,\;J_2(x_2)=2x_2^2-8x_2+33 & \qquad\text{if }\; \frac{11}{2}<x_1\leq\frac{17}{2} \\ \mu_2^*(x_2)=5-x_2\;,\;J_2(x_2)=2x_2^2-10x_2+50 & \qquad\text{if }\; x_1 >\frac{17}{2} \\ \end{cases}
⎩⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎧μ2∗(x2)=−x2,J2(x2)=2x22+25μ2∗(x2)=1−x2,J2(x2)=2x22−2x2+18μ2∗(x2)=2−x2,J2(x2)=2x22−4x2+17μ2∗(x2)=3−x2,J2(x2)=2x22−6x2+22μ2∗(x2)=4−x2,J2(x2)=2x22−8x2+33μ2∗(x2)=5−x2,J2(x2)=2x22−10x2+50if x1≤−27if −27<x1≤−21if −21<x1≤25if 25<x1≤211if 211<x1≤217if x1>217
Similarly, for
k
=
1
k=1
k=1 we have
{
μ
1
∗
(
x
1
)
=
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
+
17
if
  
x
1
≤
−
1
2
μ
1
∗
(
x
1
)
=
1
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
−
2
x
1
+
16
if
  
−
1
2
<
x
1
≤
5
2
μ
1
∗
(
x
1
)
=
2
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
−
4
x
1
+
21
if
  
5
2
<
x
1
≤
5
μ
1
∗
(
x
1
)
=
3
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
−
6
x
1
+
31
if
  
5
<
x
1
≤
15
2
μ
1
∗
(
x
1
)
=
4
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
−
8
x
1
+
46
if
  
15
2
<
x
1
≤
21
2
μ
1
∗
(
x
1
)
=
5
−
x
1
  
,
  
J
1
(
x
1
)
=
2
x
1
2
−
10
x
1
+
67
if
  
x
1
>
21
2
\begin{cases} \mu_1^*(x_1)=-x_1\;,\;J_1(x_1)=2x_1^2 +17& \qquad\text{if }\; x_1\leq-\frac{1}{2} \\ \mu_1^*(x_1)=1-x_1\;,\;J_1(x_1)=2x_1^2-2x_1+16 & \qquad\text{if }\; -\frac{1}{2}<x_1\leq\frac{5}{2} \\ \mu_1^*(x_1)=2-x_1\;,\;J_1(x_1)=2x_1^2-4x_1+21 & \qquad\text{if }\; \frac{5}{2}<x_1\leq 5 \\ \mu_1^*(x_1)=3-x_1\;,\;J_1(x_1)=2x_1^2-6x_1+31 & \qquad\text{if }\; 5<x_1\leq\frac{15}{2} \\ \mu_1^*(x_1)=4-x_1\;,\;J_1(x_1)=2x_1^2-8x_1+46 & \qquad\text{if }\; \frac{15}{2}<x_1\leq\frac{21}{2} \\ \mu_1^*(x_1)=5-x_1\;,\;J_1(x_1)=2x_1^2-10x_1+67 & \qquad\text{if }\; x_1 >\frac{21}{2} \\ \end{cases}
⎩⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎧μ1∗(x1)=−x1,J1(x1)=2x12+17μ1∗(x1)=1−x1,J1(x1)=2x12−2x1+16μ1∗(x1)=2−x1,J1(x1)=2x12−4x1+21μ1∗(x1)=3−x1,J1(x1)=2x12−6x1+31μ1∗(x1)=4−x1,J1(x1)=2x12−8x1+46μ1∗(x1)=5−x1,J1(x1)=2x12−10x1+67if x1≤−21if −21<x1≤25if 25<x1≤5if 5<x1≤215if 215<x1≤221if x1>221
For case
k
=
0
 
,
  
x
0
=
5
k=0\,,\; x_0=5
k=0,x0=5
h
0
(
−
5
)
=
66
,
  
h
0
(
−
4
)
=
57
,
  
h
0
(
−
3
)
=
54
,
  
h
0
(
−
2
)
=
56
,
  
h
0
(
−
1
)
=
63
,
  
h
0
(
0
)
=
76
h_0(-5)=66,\; h_0(-4)=57,\;h_0(-3)=54,\;h_0(-2)=56,\;h_0(-1)=63,\;h_0(0)=76
h0(−5)=66,h0(−4)=57,h0(−3)=54,h0(−2)=56,h0(−1)=63,h0(0)=76So
μ
0
∗
(
x
0
)
=
−
3
\mu_0^*(x_0)=-3
μ0∗(x0)=−3 and
J
0
(
x
0
)
=
54
J_0(x_0)=54
J0(x0)=54.
(c)In this case, w k w_k wk is not always be 0 0 0 and satisfies certern distributuion. J 3 ( x 3 ) = min u 3 ∈ U 3 ( x 3 ) E w 3 { g 3 ( x 3 , u 3 , w 3 ) + J 4 ( f 3 ( x 3 , u 3 , w 3 ) ) } = min u 3 ∈ U 3 ( x 3 ) E w 3 ( x 3 2 + u 3 2 ) J_3(x_3)=\mathop{\min}\limits_{u_3\in U_3(x_3)} \mathop\mathbf{E}\limits_{w_3}\{g_3(x_3,u_3,w_3)+J_4(f_3(x_3,u_3,w_3))\}=\mathop{\min}\limits_{u_3\in U_3(x_3)} \mathop\mathbf{E}\limits_{w_3}(x_3^2+u_3^2) J3(x3)=u3∈U3(x3)minw3E{g3(x3,u3,w3)+J4(f3(x3,u3,w3))}=u3∈U3(x3)minw3E(x32+u32)which is not affected by the distritution of w k w_k wk, so the result is the same as case (a):
{ μ 3 ∗ ( x 3 ) = − x 3    ,    J 3 ( x 3 ) = 2 x 3 2 if    x 3 ≤ 1 2 μ 3 ∗ ( x 3 ) = 1 − x 3    ,    J 3 ( x 3 ) = 2 x 3 2 − 2 x 3 + 1 if    1 2 < x 3 ≤ 3 2 μ 3 ∗ ( x 3 ) = 2 − x 3    ,    J 3 ( x 3 ) = 2 x 3 2 − 4 x 3 + 4 if    3 2 < x 3 ≤ 5 2 μ 3 ∗ ( x 3 ) = 3 − x 3    ,    J 3 ( x 3 ) = 2 x 3 2 − 6 x 3 + 9 if    5 2 < x 3 ≤ 7 2 μ 3 ∗ ( x 3 ) = 4 − x 3    ,    J 3 ( x 3 ) = 2 x 3 2 − 8 x 3 + 16 if    7 2 < x 3 ≤ 9 2 μ 3 ∗ ( x 3 ) = 5 − x 3    ,    J 3 ( x 3 ) = 2 x 3 2 − 10 x 3 + 25 if    x 3 > 9 2 \begin{cases} \mu_3^*(x_3)=-x_3\;,\;J_3(x_3)=2x_3^2 & \qquad\text{if }\; x_3\leq\frac{1}{2} \\ \mu_3^*(x_3)=1-x_3\;,\;J_3(x_3)=2x_3^2-2x_3+1 & \qquad\text{if }\; \frac{1}{2}<x_3\leq\frac{3}{2} \\ \mu_3^*(x_3)=2-x_3\;,\;J_3(x_3)=2x_3^2-4x_3+4 & \qquad\text{if }\; \frac{3}{2}<x_3\leq\frac{5}{2} \\ \mu_3^*(x_3)=3-x_3\;,\;J_3(x_3)=2x_3^2-6x_3+9 & \qquad\text{if }\; \frac{5}{2}<x_3\leq\frac{7}{2} \\ \mu_3^*(x_3)=4-x_3\;,\;J_3(x_3)=2x_3^2-8x_3+16 & \qquad\text{if }\; \frac{7}{2}<x_3\leq\frac{9}{2} \\ \mu_3^*(x_3)=5-x_3\;,\;J_3(x_3)=2x_3^2-10x_3+25 & \qquad\text{if }\; x_3 >\frac{9}{2} \\ \end{cases} ⎩⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎧μ3∗(x3)=−x3,J3(x3)=2x32μ3∗(x3)=1−x3,J3(x3)=2x32−2x3+1μ3∗(x3)=2−x3,J3(x3)=2x32−4x3+4μ3∗(x3)=3−x3,J3(x3)=2x32−6x3+9μ3∗(x3)=4−x3,J3(x3)=2x32−8x3+16μ3∗(x3)=5−x3,J3(x3)=2x32−10x3+25if x3≤21if 21<x3≤23if 23<x3≤25if 25<x3≤27if 27<x3≤29if x3>29
Especially, we have
J
3
(
0
)
=
0
,
 
J
3
(
1
)
=
1
,
 
J
3
(
2
)
=
4
,
 
J
3
(
3
)
=
9
,
 
J
3
(
4
)
=
16
,
 
J
3
(
5
)
=
25
,
J_3(0)=0, \,J_3(1)=1,\,J_3(2)=4,\,J_3(3)=9,\,J_3(4)=16,\,J_3(5)=25,
J3(0)=0,J3(1)=1,J3(2)=4,J3(3)=9,J3(4)=16,J3(5)=25, which will be used in the case
k
=
2
k=2
k=2. Now
J
2
(
x
2
)
=
min
u
2
∈
U
2
(
x
2
)
E
w
2
(
x
2
2
+
u
2
2
+
J
3
(
x
2
+
u
2
+
w
2
)
)
J_2(x_2)=\mathop{\min}\limits_{u_2\in U_2(x_2)} \mathop\mathbf{E}\limits_{w_2}(x_2^2+u_2^2+J_3(x_2+u_2+w_2))
J2(x2)=u2∈U2(x2)minw2E(x22+u22+J3(x2+u2+w2))We calculate the expectation of the right side for each of the possible values of
u
2
u_2
u2:
u
2
=
−
x
2
  
:
  
E
{
⋅
}
=
x
2
2
+
(
−
x
2
)
2
+
J
3
(
0
)
=
2
x
2
2
u_2=-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(-x_2)^2+J_3(0)=2x_2^2
u2=−x2:E{⋅}=x22+(−x2)2+J3(0)=2x22
u
2
=
1
−
x
2
  
:
  
E
{
⋅
}
=
x
2
2
+
(
1
−
x
2
)
2
+
1
2
[
J
3
(
0
)
+
J
3
(
2
)
]
=
2
x
2
2
−
2
x
2
+
3
u_2=1-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(1-x_2)^2+\frac{1}{2}[J_3(0)+J_3(2)]=2x_2^2-2x_2+3
u2=1−x2:E{⋅}=x22+(1−x2)2+21[J3(0)+J3(2)]=2x22−2x2+3
u
2
=
2
−
x
2
  
:
  
E
{
⋅
}
=
x
2
2
+
(
2
−
x
2
)
2
+
1
2
[
J
3
(
1
)
+
J
3
(
3
)
]
=
2
x
2
2
−
4
x
2
+
9
u_2=2-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(2-x_2)^2+\frac{1}{2}[J_3(1)+J_3(3)]=2x_2^2-4x_2+9
u2=2−x2:E{⋅}=x22+(2−x2)2+21[J3(1)+J3(3)]=2x22−4x2+9
u
2
=
3
−
x
2
  
:
  
E
{
⋅
}
=
x
2
2
+
(
3
−
x
2
)
2
+
1
2
[
J
3
(
2
)
+
J
3
(
4
)
]
=
2
x
2
2
−
6
x
2
+
19
u_2=3-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(3-x_2)^2+\frac{1}{2}[J_3(2)+J_3(4)]=2x_2^2-6x_2+19
u2=3−x2:E{⋅}=x22+(3−x2)2+21[J3(2)+J3(4)]=2x22−6x2+19
u
2
=
4
−
x
2
  
:
  
E
{
⋅
}
=
x
2
2
+
(
4
−
x
2
)
2
+
1
2
[
J
3
(
3
)
+
J
3
(
5
)
]
=
2
x
2
2
−
8
x
2
+
33
u_2=4-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(4-x_2)^2+\frac{1}{2}[J_3(3)+J_3(5)]=2x_2^2-8x_2+33
u2=4−x2:E{⋅}=x22+(4−x2)2+21[J3(3)+J3(5)]=2x22−8x2+33
u
2
=
5
−
x
2
  
:
  
E
{
⋅
}
=
x
2
2
+
(
5
−
x
2
)
2
+
J
3
(
5
)
=
2
x
2
2
−
10
x
2
+
50
u_2=5-x_2\;:\;\mathbf{E}\{\cdot\}=x_2^2+(5-x_2)^2+J_3(5)=2x_2^2-10x_2+50
u2=5−x2:E{⋅}=x22+(5−x2)2+J3(5)=2x22−10x2+50Hence we have
{ μ 2 ∗ ( x 2 ) = − x 2    ,    J 2 ( x 2 ) = 2 x 2 2 if    x 2 ≤ 3 2 μ 2 ∗ ( x 2 ) = 1 − x 2    ,    J 2 ( x 2 ) = 2 x 2 2 − 2 x 2 + 3 if    3 2 < x 2 ≤ 3 μ 2 ∗ ( x 2 ) = 2 − x 2    ,    J 2 ( x 2 ) = 2 x 2 2 − 4 x 2 + 9 if    3 < x 2 ≤ 5 μ 2 ∗ ( x 2 ) = 3 − x 2    ,    J 2 ( x 2 ) = 2 x 2 2 − 6 x 2 + 19 if    5 < x 2 ≤ 7 μ 2 ∗ ( x 2 ) = 4 − x 2    ,    J 2 ( x 2 ) = 2 x 2 2 − 8 x 2 + 33 if    7 < x 2 ≤ 17 2 μ 2 ∗ ( x 2 ) = 5 − x 2    ,    J 2 ( x 2 ) = 2 x 2 2 − 10 x 2 + 50 if    x 2 > 17 2 \begin{cases} \mu_2^*(x_2)=-x_2\;,\;J_2(x_2)=2x_2^2 & \qquad\text{if }\; x_2\leq\frac{3}{2} \\ \mu_2^*(x_2)=1-x_2\;,\;J_2(x_2)=2x_2^2-2x_2+3 & \qquad\text{if }\; \frac{3}{2}<x_2\leq 3 \\ \mu_2^*(x_2)=2-x_2\;,\;J_2(x_2)=2x_2^2-4x_2+9 & \qquad\text{if }\; 3<x_2\leq5 \\ \mu_2^*(x_2)=3-x_2\;,\;J_2(x_2)=2x_2^2-6x_2+19 & \qquad\text{if }\; 5<x_2\leq7 \\ \mu_2^*(x_2)=4-x_2\;,\;J_2(x_2)=2x_2^2-8x_2+33 & \qquad\text{if }\; 7<x_2\leq\frac{17}{2} \\ \mu_2^*(x_2)=5-x_2\;,\;J_2(x_2)=2x_2^2-10x_2+50 & \qquad\text{if }\; x_2 >\frac{17}{2} \\ \end{cases} ⎩⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎧μ2∗(x2)=−x2,J2(x2)=2x22μ2∗(x2)=1−x2,J2(x2)=2x22−2x2+3μ2∗(x2)=2−x2,J2(x2)=2x22−4x2+9μ2∗(x2)=3−x2,J2(x2)=2x22−6x2+19μ2∗(x2)=4−x2,J2(x2)=2x22−8x2+33μ2∗(x2)=5−x2,J2(x2)=2x22−10x2+50if x2≤23if 23<x2≤3if 3<x2≤5if 5<x2≤7if 7<x2≤217if x2>217
Especially, J 2 ( 0 ) = 0 ,   J 2 ( 1 ) = 2 ,   J 2 ( 2 ) = 7 ,   J 2 ( 3 ) = 15 ,   J 2 ( 4 ) = 25 ,   J 2 ( 5 ) = 39. J_2(0)=0,\,J_2(1)=2,\,J_2(2)=7,\,J_2(3)=15,\,J_2(4)=25,\,J_2(5)=39. J2(0)=0,J2(1)=2,J2(2)=7,J2(3)=15,J2(4)=25,J2(5)=39.
For case
k
=
1
k=1
k=1
J
1
(
x
1
)
=
min
u
1
∈
U
1
(
x
1
)
E
w
1
(
x
1
2
+
u
1
2
+
J
2
(
x
1
+
u
1
+
w
1
)
)
J_1(x_1)=\mathop{\min}\limits_{u_1\in U_1(x_1)} \mathop\mathbf{E}\limits_{w_1}(x_1^2+u_1^2+J_2(x_1+u_1+w_1))
J1(x1)=u1∈U1(x1)minw1E(x12+u12+J2(x1+u1+w1))We calculate the expectation of the right side for each of the possible values of
u
1
u_1
u1:
u
1
=
−
x
1
  
:
  
E
{
⋅
}
=
x
1
2
+
(
−
x
1
)
2
+
J
2
(
0
)
=
2
x
1
2
u_1=-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(-x_1)^2+J_2(0)=2x_1^2
u1=−x1:E{⋅}=x12+(−x1)2+J2(0)=2x12
u
2
=
1
−
x
1
  
:
  
E
{
⋅
}
=
x
1
2
+
(
1
−
x
1
)
2
+
1
2
[
J
2
(
0
)
+
J
2
(
2
)
]
=
2
x
1
2
−
2
x
1
+
9
2
u_2=1-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(1-x_1)^2+\frac{1}{2}[J_2(0)+J_2(2)]=2x_1^2-2x_1+\frac{9}{2}
u2=1−x1:E{⋅}=x12+(1−x1)2+21[J2(0)+J2(2)]=2x12−2x1+29
u
1
=
2
−
x
1
  
:
  
E
{
⋅
}
=
x
1
2
+
(
2
−
x
1
)
2
+
1
2
[
J
2
(
1
)
+
J
2
(
3
)
]
=
2
x
1
2
−
4
x
1
+
25
2
u_1=2-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(2-x_1)^2+\frac{1}{2}[J_2(1)+J_2(3)]=2x_1^2-4x_1+\frac{25}{2}
u1=2−x1:E{⋅}=x12+(2−x1)2+21[J2(1)+J2(3)]=2x12−4x1+225
u
1
=
3
−
x
1
  
:
  
E
{
⋅
}
=
x
1
2
+
(
3
−
x
1
)
2
+
1
2
[
J
2
(
2
)
+
J
2
(
4
)
]
=
2
x
1
2
−
6
x
1
+
25
u_1=3-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(3-x_1)^2+\frac{1}{2}[J_2(2)+J_2(4)]=2x_1^2-6x_1+25
u1=3−x1:E{⋅}=x12+(3−x1)2+21[J2(2)+J2(4)]=2x12−6x1+25
u
1
=
4
−
x
1
  
:
  
E
{
⋅
}
=
x
1
2
+
(
4
−
x
1
)
2
+
1
2
[
J
2
(
3
)
+
J
2
(
5
)
]
=
2
x
1
2
−
8
x
1
+
43
u_1=4-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(4-x_1)^2+\frac{1}{2}[J_2(3)+J_2(5)]=2x_1^2-8x_1+43
u1=4−x1:E{⋅}=x12+(4−x1)2+21[J2(3)+J2(5)]=2x12−8x1+43
u
1
=
5
−
x
1
  
:
  
E
{
⋅
}
=
x
1
2
+
(
5
−
x
1
)
2
+
J
2
(
5
)
=
2
x
1
2
−
10
x
1
+
64
u_1=5-x_1\;:\;\mathbf{E}\{\cdot\}=x_1^2+(5-x_1)^2+J_2(5)=2x_1^2-10x_1+64
u1=5−x1:E{⋅}=x12+(5−x1)2+J2(5)=2x12−10x1+64Hence we have
{ μ 1 ∗ ( x 1 ) = − x 1    ,    J 1 ( x 1 ) = 2 x 1 2 if    x 1 ≤ 9 4 μ 1 ∗ ( x 1 ) = 1 − x 1    ,    J 1 ( x 1 ) = 2 x 1 2 − 2 x 1 + 9 2 if    9 4 < x 1 ≤ 4 μ 1 ∗ ( x 1 ) = 2 − x 1    ,    J 1 ( x 1 ) = 2 x 1 2 − 4 x 1 + 25 2 if    4 < x 1 ≤ 25 4 μ 1 ∗ ( x 1 ) = 3 − x 1    ,    J 1 ( x 1 ) = 2 x 1 2 − 6 x 1 + 25 if    25 4 < x 1 ≤ 9 μ 1 ∗ ( x 1 ) = 4 − x 1    ,    J 1 ( x 1 ) = 2 x 1 2 − 8 x 1 + 43 if    9 < x 1 ≤ 21 2 μ 1 ∗ ( x 1 ) = 5 − x 1    ,    J 1 ( x 1 ) = 2 x 1 2 − 10 x 1 + 64 if    x 1 > 21 2 \begin{cases} \mu_1^*(x_1)=-x_1\;,\;J_1(x_1)=2x_1^2 & \qquad\text{if }\; x_1\leq\frac{9}{4} \\ \mu_1^*(x_1)=1-x_1\;,\;J_1(x_1)=2x_1^2-2x_1+\frac{9}{2} & \qquad\text{if }\; \frac{9}{4}<x_1\leq 4 \\ \mu_1^*(x_1)=2-x_1\;,\;J_1(x_1)=2x_1^2-4x_1+\frac{25}{2}& \qquad\text{if }\; 4<x_1\leq\frac{25}{4} \\ \mu_1^*(x_1)=3-x_1\;,\;J_1(x_1)=2x_1^2-6x_1+25& \qquad\text{if }\; \frac{25}{4}<x_1\leq9 \\ \mu_1^*(x_1)=4-x_1\;,\;J_1(x_1)=2x_1^2-8x_1+43& \qquad\text{if }\; 9<x_1\leq\frac{21}{2} \\ \mu_1^*(x_1)=5-x_1\;,\;J_1(x_1)=2x_1^2-10x_1+64& \qquad\text{if }\; x_1 >\frac{21}{2} \\ \end{cases} ⎩⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎧μ1∗(x1)=−x1,J1(x1)=2x12μ1∗(x1)=1−x1,J1(x1)=2x12−2x1+29μ1∗(x1)=2−x1,J1(x1)=2x12−4x1+225μ1∗(x1)=3−x1,J1(x1)=2x12−6x1+25μ1∗(x1)=4−x1,J1(x1)=2x12−8x1+43μ1∗(x1)=5−x1,J1(x1)=2x12−10x1+64if x1≤49if 49<x1≤4if 4<x1≤425if 425<x1≤9if 9<x1≤221if x1>221
Especially, J 1 ( 0 ) = 0 ,   J 1 ( 1 ) = 2 ,   J 1 ( 2 ) = 8 ,   J 1 ( 3 ) = 15 2 ,   J 1 ( 4 ) = 57 2 ,   J 1 ( 5 ) = 85 2 . J_1(0)=0,\,J_1(1)=2,\,J_1(2)=8,\,J_1(3)=\frac{15}{2},\,J_1(4)=\frac{57}{2},\,J_1(5)=\frac{85}{2}. J1(0)=0,J1(1)=2,J1(2)=8,J1(3)=215,J1(4)=257,J1(5)=285.
The last step, the algorithm takes the form
J
0
(
x
0
)
=
min
u
0
∈
U
0
(
x
0
)
E
w
0
(
x
0
2
+
u
0
2
+
J
1
(
x
0
+
u
0
+
w
0
)
)
=
min
u
0
∈
U
0
(
x
0
)
E
w
0
(
25
+
u
0
2
+
J
1
(
5
+
u
0
+
w
0
)
)
J_0(x_0)=\mathop{\min}\limits_{u_0\in U_0(x_0)} \mathop\mathbf{E}\limits_{w_0}(x_0^2+u_0^2+J_1(x_0+u_0+w_0))=\mathop{\min}\limits_{u_0\in U_0(x_0)} \mathop\mathbf{E}\limits_{w_0}\big(25+u_0^2+J_1(5+u_0+w_0)\big)
J0(x0)=u0∈U0(x0)minw0E(x02+u02+J1(x0+u0+w0))=u0∈U0(x0)minw0E(25+u02+J1(5+u0+w0))
u
0
=
−
5
  
:
  
E
{
⋅
}
=
25
+
25
+
J
1
(
0
)
=
50
u_0=-5\;:\;\mathbf{E}\{\cdot\}=25+25+J_1(0)=50
u0=−5:E{⋅}=25+25+J1(0)=50
u
0
=
−
4
  
:
  
E
{
⋅
}
=
25
+
16
+
1
2
[
J
1
(
0
)
+
J
1
(
2
)
]
=
45
u_0=-4\;:\;\mathbf{E}\{\cdot\}=25+16+\frac{1}{2}[J_1(0)+J_1(2)]=45
u0=−4:E{⋅}=25+16+21[J1(0)+J1(2)]=45
u
0
=
−
3
  
:
  
E
{
⋅
}
=
25
+
9
+
1
2
[
J
1
(
1
)
+
J
1
(
3
)
]
=
38
3
4
u_0=-3\;:\;\mathbf{E}\{\cdot\}=25+9+\frac{1}{2}[J_1(1)+J_1(3)]=38\frac{3}{4}
u0=−3:E{⋅}=25+9+21[J1(1)+J1(3)]=3843
u
0
=
−
2
  
:
  
E
{
⋅
}
=
25
+
4
+
1
2
[
J
1
(
2
)
+
J
1
(
4
)
]
=
47
1
4
u_0=-2\;:\;\mathbf{E}\{\cdot\}=25+4+\frac{1}{2}[J_1(2)+J_1(4)]=47\frac{1}{4}
u0=−2:E{⋅}=25+4+21[J1(2)+J1(4)]=4741
u
1
=
−
1
  
:
  
E
{
⋅
}
=
25
+
1
+
1
2
[
J
1
(
3
)
+
J
1
(
5
)
]
=
51
u_1=-1\;:\;\mathbf{E}\{\cdot\}=25+1+\frac{1}{2}[J_1(3)+J_1(5)]=51
u1=−1:E{⋅}=25+1+21[J1(3)+J1(5)]=51
u
0
=
0
  
:
  
E
{
⋅
}
=
25
+
0
+
J
1
(
5
)
=
67
1
2
u_0=0\;:\;\mathbf{E}\{\cdot\}=25+0+J_1(5)=67\frac{1}{2}
u0=0:E{⋅}=25+0+J1(5)=6721So
μ
0
∗
(
x
0
)
=
−
3
,
J
0
(
x
0
)
=
38
3
4
.
□
\mu_0^*(x_0)=-3,J_0(x_0)=38\frac{3}{4}.\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\Box
μ0∗(x0)=−3,J0(x0)=3843.□
PS: In the book, a hint is given : Alternatively, you may use a terminal cost
g
4
(
x
4
)
g_4(x_4)
g4(x4) equal to a very large number for
x
4
≠
5
x_4\neq 5
x4̸=5. Why a very large number ? My caculation is wrong?
1.2 Carry out the calculations needed to verify that
J
0
(
1
)
=
2.67
J_0(1)=2.67
J0(1)=2.67 and
J
0
(
2
)
=
2.608
J_0(2)=2.608
J0(2)=2.608 in Example 1.3.2.
Solution. The result in the exercise is wrong, even not the same as that in the text.
For
x
0
=
1
x_0=1
x0=1, we have
J
0
(
1
)
=
min
u
0
=
0
,
1
E
w
0
=
0
,
1
,
2
{
u
0
+
(
1
+
u
0
−
w
0
)
2
+
J
1
(
max
(
0
,
1
+
u
0
−
w
0
)
)
)
}
J_0(1)=\min_{u_0=0,1} \mathop{\bf E}\limits_{w_0=0,1,2}\{u_0+(1+u_0-w_0)^2+J_1(\max(0,1+u_0-w_0)))\}
J0(1)=u0=0,1minw0=0,1,2E{u0+(1+u0−w0)2+J1(max(0,1+u0−w0)))}
u
0
=
0
:
  
E
(
.
)
=
E
w
0
=
0
,
1
,
2
{
(
1
−
w
0
)
2
+
J
1
(
max
(
0
,
1
−
w
0
)
)
}
u_0=0: \;\mathbf{E}(.)=\mathop{\mathbf{E}}\limits_{w_0=0,1,2}\{(1-w_0)^2+J_1(\max(0,1-w_0))\}
u0=0:E(.)=w0=0,1,2E{(1−w0)2+J1(max(0,1−w0))}
=
0.1
×
[
1
+
J
1
(
1
)
]
+
0.7
×
J
1
(
0
)
+
0.2
×
[
1
+
J
1
(
0
)
]
\qquad\qquad\qquad=0.1\times[1+J_1(1)]+0.7\times J_1(0)+0.2\times[1+J_1(0)]
=0.1×[1+J1(1)]+0.7×J1(0)+0.2×[1+J1(0)]
=
0.1
×
2.5
+
0.7
×
2.5
+
0.2
×
3.5
\qquad\qquad\qquad=0.1\times2.5+0.7\times 2.5 + 0.2\times 3.5
=0.1×2.5+0.7×2.5+0.2×3.5
=
2.7
\qquad\qquad\qquad=2.7
=2.7
u
0
=
1
:
  
E
(
.
)
=
E
w
0
=
0
,
1
,
2
{
1
+
(
2
−
w
0
)
2
+
J
1
(
max
(
0
,
2
−
w
0
)
)
}
u_0=1: \;\mathbf{E}(.)=\mathop{\mathbf{E}}\limits_{w_0=0,1,2}\{1+(2-w_0)^2+J_1(\max(0,2-w_0))\}
u0=1:E(.)=w0=0,1,2E{1+(2−w0)2+J1(max(0,2−w0))}
=
0.1
×
[
5
+
J
1
(
2
)
]
+
0.7
×
[
2
+
J
1
(
1
)
]
+
0.2
×
[
1
+
J
1
(
0
)
]
\qquad\qquad\qquad=0.1\times[5+J_1(2)]+0.7\times [2+J_1(1)]+0.2\times[1+J_1(0)]
=0.1×[5+J1(2)]+0.7×[2+J1(1)]+0.2×[1+J1(0)]
=
0.1
×
6.68
+
0.7
×
3.5
+
0.2
×
3.5
\qquad\qquad\qquad=0.1\times6.68+0.7\times3.5 + 0.2\times 3.5
=0.1×6.68+0.7×3.5+0.2×3.5
=
3.818
\qquad\qquad\qquad=3.818
=3.818
So
J
0
(
1
)
=
2.7
,
μ
0
∗
(
1
)
=
0.
J_0(1)=2.7, \mu_0^*(1)=0.
J0(1)=2.7,μ0∗(1)=0.
For
x
0
=
2
x_0=2
x0=2, we have
J
0
(
2
)
=
min
u
0
=
0
E
w
0
=
0
,
1
,
2
{
u
0
+
(
2
+
u
0
−
w
0
)
2
+
J
1
(
max
(
0
,
2
+
u
0
−
w
0
)
)
)
}
J_0(2)=\min_{u_0=0} \mathop{\bf E}\limits_{w_0=0,1,2}\{u_0+(2+u_0-w_0)^2+J_1(\max(0,2+u_0-w_0)))\}
J0(2)=u0=0minw0=0,1,2E{u0+(2+u0−w0)2+J1(max(0,2+u0−w0)))}
=
E
w
0
=
0
,
1
,
2
{
(
2
−
w
0
)
2
+
J
1
(
max
(
0
,
2
−
w
0
)
)
}
\qquad\qquad\qquad\qquad=\mathop{\mathbf{E}}\limits_{w_0=0,1,2}\{(2-w_0)^2+J_1(\max(0,2-w_0))\}
=w0=0,1,2E{(2−w0)2+J1(max(0,2−w0))}
=
0.1
×
[
4
+
J
1
(
2
)
]
+
0.7
×
[
1
+
J
1
(
1
)
]
+
0.2
×
J
1
(
0
)
\qquad\qquad\qquad\qquad=0.1\times[4+J_1(2)]+0.7\times[1+J_1(1)]+0.2\times J_1(0)
=0.1×[4+J1(2)]+0.7×[1+J1(1)]+0.2×J1(0)
=
0.1
×
5.68
+
0.7
×
2.5
+
0.2
×
2.5
\qquad\qquad\qquad\qquad=0.1\times5.68+0.7\times2.5+0.2\times 2.5
=0.1×5.68+0.7×2.5+0.2×2.5
=
2.818
\qquad\qquad\qquad\qquad=2.818
=2.818
So
J
0
(
2
)
=
2.818
,
μ
0
∗
(
2
)
=
0.
□
J_0(2)=2.818,\mu_0^*(2)=0.\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\Box
J0(2)=2.818,μ0∗(2)=0.□
1.6 (Discouted Cost per State) In the framework of the basic problem, consider the case where the cost is of the form
E
w
k
k
=
0
,
1
,
.
.
.
,
N
−
1
{
α
N
g
N
(
x
N
)
+
∑
k
=
0
N
−
1
α
k
g
k
(
x
k
,
u
k
,
w
k
)
}
\mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,...,N-1}}\left\{\alpha^Ng_N(x_N)+\sum_{k=0}^{N-1}\alpha^kg_k(x_k,u_k,w_k)\right\}
k=0,1,...,N−1wkE{αNgN(xN)+k=0∑N−1αkgk(xk,uk,wk)}where
α
\alpha
α is a discount factor with
0
<
α
<
1
0<\alpha<1
0<α<1. Show that an alternate form the DP algorithm is given by
V
N
(
x
N
)
=
g
N
(
x
N
)
V_N(x_N)=g_N(x_N)
VN(xN)=gN(xN)
V
k
(
x
k
)
=
min
u
k
∈
U
k
(
x
k
)
E
w
k
{
g
k
(
x
k
,
u
k
,
w
k
)
+
α
V
k
+
1
(
f
k
(
x
k
,
u
k
,
w
k
)
)
}
.
V_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\left\{g_k(x_k,u_k,w_k)+\alpha V_{k+1}\big(f_k(x_k,u_k,w_k)\big)\right\}.
Vk(xk)=uk∈Uk(xk)minwkE{gk(xk,uk,wk)+αVk+1(fk(xk,uk,wk))}.Solution. Using similar argument to Proposition 1.3.1, in the dicounted cost case, the minimizing cost takes the form
V
∗
(
x
0
)
=
min
π
∈
Π
E
w
k
k
=
0
,
1
,
.
.
.
,
N
−
1
{
α
N
g
N
(
x
N
)
+
∑
k
=
0
N
−
1
α
k
g
k
(
x
k
,
u
k
,
w
k
)
}
V^*(x_0)=\mathop{\min}\limits_{\pi\in\Pi}\mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,...,N-1}}\left\{\alpha^Ng_N(x_N)+\sum_{k=0}^{N-1}\alpha^kg_k(x_k,u_k,w_k)\right\}
V∗(x0)=π∈Πmink=0,1,...,N−1wkE{αNgN(xN)+k=0∑N−1αkgk(xk,uk,wk)} Let
V
k
∗
(
x
k
)
V_k^*(x_k)
Vk∗(xk) be the optimal cost for the
(
N
−
k
)
(N-k)
(N−k)-stage problem that starts at state
x
k
x_k
xk and time
k
k
k, and ends at time
N
N
N.
V
k
∗
(
x
k
)
=
min
π
k
E
w
k
,
.
.
.
,
w
N
−
1
{
α
N
−
k
g
N
(
x
N
)
+
∑
i
=
k
N
−
1
α
i
−
k
g
i
(
x
i
,
u
i
,
w
i
)
}
V_k^*(x_k)=\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{w_k,...,w_{N-1}}\left\{\alpha^{N-k}g_N(x_N)+\sum_{i=k}^{N-1}\alpha^{i-k}g_i(x_i,u_i,w_i)\right\}
Vk∗(xk)=πkminwk,...,wN−1E{αN−kgN(xN)+i=k∑N−1αi−kgi(xi,ui,wi)} Claim:
  
V
k
∗
(
x
k
)
=
V
k
(
x
k
)
.
\; V_k^*(x_k)=V_k(x_k).
Vk∗(xk)=Vk(xk).
We prove this cliam by induction on
k
k
k. Firstly,
V
N
∗
(
x
N
)
=
g
N
(
x
N
)
=
V
N
(
x
N
)
V_N^*(x_N)=g_N(x_N)=V_N(x_N)
VN∗(xN)=gN(xN)=VN(xN). Suppose the claim is true for any
k
+
1
,
.
.
.
,
N
k+1, ..., N
k+1,...,N, then
V
k
∗
(
x
k
)
=
min
π
k
E
w
k
,
.
.
.
,
w
N
−
1
{
g
k
(
x
k
,
u
k
,
w
k
)
+
α
N
−
k
g
N
(
x
N
)
+
∑
i
=
k
+
1
N
−
1
α
i
−
k
g
i
(
x
i
,
u
i
,
w
i
)
}
V_k^*(x_k)=\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{w_k,...,w_{N-1}}\left\{g_k(x_k,u_k,w_k)+\alpha^{N-k}g_N(x_N)+\sum_{i=k+1}^{N-1}\alpha^{i-k}g_i(x_i,u_i,w_i)\right\}\qquad\qquad\quad
Vk∗(xk)=πkminwk,...,wN−1E{gk(xk,uk,wk)+αN−kgN(xN)+i=k+1∑N−1αi−kgi(xi,ui,wi)}
=
min
u
k
∈
U
k
(
x
k
)
E
w
k
{
g
k
(
x
k
,
u
k
,
w
k
}
+
α
⋅
min
π
k
E
w
k
,
.
.
.
,
w
N
−
1
{
α
N
−
k
−
1
g
N
(
x
N
)
+
∑
i
=
k
+
1
N
−
1
α
i
−
k
−
1
g
i
(
x
i
,
u
i
,
w
i
)
}
=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha\cdot\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{w_k,...,w_{N-1}}\left\{\alpha^{N-k-1}g_N(x_N)+\sum_{i=k+1}^{N-1}\alpha^{i-k-1}g_i(x_i,u_i,w_i)\right\}
=uk∈Uk(xk)minwkE{gk(xk,uk,wk}+α⋅πkminwk,...,wN−1E{αN−k−1gN(xN)+i=k+1∑N−1αi−k−1gi(xi,ui,wi)}
=
min
u
k
∈
U
k
(
x
k
)
E
w
k
{
g
k
(
x
k
,
u
k
,
w
k
}
+
α
⋅
min
π
k
+
1
E
w
k
+
1
,
.
.
.
,
w
N
−
1
{
α
N
−
k
−
1
g
N
(
x
N
)
+
∑
i
=
k
+
1
N
−
1
α
i
−
k
−
1
g
i
(
x
i
,
u
i
,
w
i
)
}
=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha\cdot\mathop{\min}\limits_{\pi^{k+1}}\mathop\mathbf{E}\limits_{w_{k+1},...,w_{N-1}}\left\{\alpha^{N-k-1}g_N(x_N)+\sum_{i=k+1}^{N-1}\alpha^{i-k-1}g_i(x_i,u_i,w_i)\right\}
=uk∈Uk(xk)minwkE{gk(xk,uk,wk}+α⋅πk+1minwk+1,...,wN−1E{αN−k−1gN(xN)+i=k+1∑N−1αi−k−1gi(xi,ui,wi)}
=
min
u
k
∈
U
k
(
x
k
)
E
w
k
{
g
k
(
x
k
,
u
k
,
w
k
}
+
α
V
k
+
1
∗
(
x
k
+
1
)
=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha V_{k+1}^*(x_{k+1})\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad
=uk∈Uk(xk)minwkE{gk(xk,uk,wk}+αVk+1∗(xk+1)
=
min
u
k
∈
U
k
(
x
k
)
E
w
k
{
g
k
(
x
k
,
u
k
,
w
k
}
+
α
V
k
+
1
(
x
k
+
1
)
=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha V_{k+1}(x_{k+1})\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad
=uk∈Uk(xk)minwkE{gk(xk,uk,wk}+αVk+1(xk+1)
=
min
u
k
∈
U
k
(
x
k
)
E
w
k
{
g
k
(
x
k
,
u
k
,
w
k
}
+
α
⋅
min
u
k
∈
U
k
(
x
k
)
E
w
k
V
k
+
1
(
f
k
(
x
k
,
u
k
,
w
k
)
)
  
=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\{g_k(x_k,u_k,w_k\}+\alpha \cdot\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}V_{k+1}(f_k(x_k,u_k,w_k))\qquad\qquad\qquad\qquad\qquad\qquad\;
=uk∈Uk(xk)minwkE{gk(xk,uk,wk}+α⋅uk∈Uk(xk)minwkEVk+1(fk(xk,uk,wk))
=
min
u
k
∈
U
k
(
x
k
)
E
w
k
{
g
k
(
x
k
,
u
k
,
w
k
)
+
α
V
k
+
1
(
f
k
(
x
k
,
u
k
,
w
k
)
)
}
=
V
k
(
x
k
)
□
=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\left\{g_k(x_k,u_k,w_k)+\alpha V_{k+1}\big(f_k(x_k,u_k,w_k)\big)\right\}=V_k(x_k)\qquad\qquad\qquad\qquad\qquad\qquad\Box
=uk∈Uk(xk)minwkE{gk(xk,uk,wk)+αVk+1(fk(xk,uk,wk))}=Vk(xk)□
1.7 (Exponential Cost Function) In the framework of the basic problem, consider the case where the cost is of the form
E
w
k
k
=
0
,
1
,
.
.
.
,
N
−
1
{
exp
(
g
N
(
x
N
)
+
∑
k
=
0
N
−
1
g
k
(
x
k
,
u
k
,
w
k
)
)
}
\mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,...,N-1}}\left\{\exp\left(g_N(x_N)+\sum_{k=0}^{N-1}g_k(x_k,u_k,w_k)\right)\right\}
k=0,1,...,N−1wkE{exp(gN(xN)+k=0∑N−1gk(xk,uk,wk))}(a)Show that the optimal cost and an optimal policy can be obtained from the DP-like algorithm
J
N
(
x
N
)
=
exp
(
g
N
(
x
N
)
)
J_N(x_N)=\exp\left(g_N(x_N)\right)
JN(xN)=exp(gN(xN))
J
k
(
x
k
)
=
min
u
k
∈
U
k
(
x
k
)
E
w
k
{
J
k
+
1
(
f
k
(
x
k
,
u
k
,
w
k
)
)
exp
(
g
k
(
x
k
,
u
k
,
w
k
)
)
}
J_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop\mathbf{E}\limits_{w_k}\left\{J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\exp\left(g_k(x_k,u_k,w_k)\right)\right\}
Jk(xk)=uk∈Uk(xk)minwkE{Jk+1(fk(xk,uk,wk))exp(gk(xk,uk,wk))} (b)Define the functions
V
k
(
x
k
)
=
ln
J
k
(
x
k
)
V_k(x_k)=\ln J_k(x_k)
Vk(xk)=lnJk(xk). Assume also that
g
k
g_k
gk is a function of
x
k
x_k
xk and
u
k
u_k
uk only (and not of
w
k
w_k
wk). Show that the above algorithm can be rewritten as
V
N
(
x
N
)
=
g
N
(
x
N
)
V_N(x_N)=g_N(x_N)
VN(xN)=gN(xN)
V
k
(
x
k
)
=
min
u
k
∈
U
k
(
x
k
)
{
g
k
(
x
k
,
u
k
)
+
ln
E
w
k
{
exp
(
V
k
+
1
(
f
k
(
x
k
,
u
k
,
w
k
)
)
)
}
}
V_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{g_k(x_k,u_k)+\ln \mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(V_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right)\right\}\right\}
Vk(xk)=uk∈Uk(xk)min{gk(xk,uk)+lnwkE{exp(Vk+1(fk(xk,uk,wk)))}}
Solution. (a) Define
J
k
∗
(
x
k
)
=
min
π
k
E
w
i
i
=
k
,
k
+
1
,
.
.
.
,
N
−
1
{
exp
(
g
N
(
x
N
)
+
∑
i
=
k
N
−
1
g
i
(
x
i
,
u
i
,
w
i
)
)
}
J_k^*(x_k)=\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{\mathop{w_i}\limits_{i=k,k+1,...,N-1}}\left\{\exp\left(g_N(x_N)+\sum_{i=k}^{N-1}g_i(x_i,u_i,w_i)\right)\right\}
Jk∗(xk)=πkmini=k,k+1,...,N−1wiE{exp(gN(xN)+i=k∑N−1gi(xi,ui,wi))} It is easy to check that
J
k
∗
(
x
k
)
=
J
k
(
x
k
)
J_k^*(x_k)=J_k(x_k)
Jk∗(xk)=Jk(xk).
J
k
∗
(
x
k
)
=
min
u
k
E
w
k
{
exp
(
g
k
(
x
k
,
u
k
,
w
k
)
)
}
min
π
k
E
w
i
i
=
k
,
k
+
1
,
.
.
.
,
N
−
1
{
exp
(
g
N
(
x
N
)
+
∑
i
=
k
+
1
N
−
1
g
i
(
x
i
,
u
i
,
w
i
)
)
}
J_k^*(x_k)=\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\mathop{\min}\limits_{\pi^k}\mathop\mathbf{E}\limits_{\mathop{w_i}\limits_{i=k,k+1,...,N-1}}\left\{\exp\left(g_N(x_N)+\sum_{i=k+1}^{N-1}g_i(x_i,u_i,w_i)\right)\right\}
Jk∗(xk)=ukminwkE{exp(gk(xk,uk,wk))}πkmini=k,k+1,...,N−1wiE{exp(gN(xN)+i=k+1∑N−1gi(xi,ui,wi))}
=
min
u
k
E
w
k
{
exp
(
g
k
(
x
k
,
u
k
,
w
k
)
)
}
min
π
k
+
1
x
k
+
1
=
f
k
(
x
k
,
u
k
,
w
k
)
E
w
i
i
=
k
+
1
,
.
.
.
.
,
N
−
1
{
exp
(
g
N
(
x
N
)
+
∑
i
=
k
+
1
N
−
1
g
i
(
x
i
,
u
i
,
w
i
)
)
}
=\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\mathop{\min}\limits_{\mathop{\pi^{k+1}}\limits_{x_{k+1}=f_k(x_k,u_k,w_k)}}\mathop\mathbf{E}\limits_{\mathop{w_i}\limits_{i=k+1,....,N-1}}\left\{\exp\left(g_N(x_N)+\sum_{i=k+1}^{N-1}g_i(x_i,u_i,w_i)\right)\right\}
=ukminwkE{exp(gk(xk,uk,wk))}xk+1=fk(xk,uk,wk)πk+1mini=k+1,....,N−1wiE{exp(gN(xN)+i=k+1∑N−1gi(xi,ui,wi))}
=
min
u
k
E
w
k
{
exp
(
g
k
(
x
k
,
u
k
,
w
k
)
)
}
⋅
min
u
k
E
w
k
J
k
+
1
∗
(
f
k
(
x
k
,
u
k
,
w
k
)
)
)
=\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\cdot \mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}J_{k+1}^*(f_k(x_k,u_k,w_k)))\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad
=ukminwkE{exp(gk(xk,uk,wk))}⋅ukminwkEJk+1∗(fk(xk,uk,wk)))
=
min
u
k
E
w
k
{
exp
(
g
k
(
x
k
,
u
k
,
w
k
)
)
}
⋅
min
u
k
E
w
k
J
k
+
1
(
f
k
(
x
k
,
u
k
,
w
k
)
)
)
=
J
k
(
x
k
)
=\mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\right\}\cdot \mathop{\min}\limits_{u_k}\mathop\mathbf{E}\limits_{w_k}J_{k+1}(f_k(x_k,u_k,w_k)))=J_k(x_k)\qquad\qquad\qquad\qquad\qquad
=ukminwkE{exp(gk(xk,uk,wk))}⋅ukminwkEJk+1(fk(xk,uk,wk)))=Jk(xk) (b) Since
g
k
g_k
gk is a function not of
w
k
w_k
wk, we can rewrite
J
k
(
x
k
)
J_k(x_k)
Jk(xk) as follows:
J
k
(
x
k
)
=
min
u
k
∈
U
k
(
x
k
)
{
exp
(
g
k
(
x
k
,
u
k
,
w
k
)
)
E
w
k
{
J
k
+
1
(
f
k
(
x
k
,
u
k
,
w
k
)
)
}
}
J_k(x_k)=\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{\exp\left(g_k(x_k,u_k,w_k)\right)\mathop\mathbf{E}\limits_{w_k}\left\{J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\}\right\}
Jk(xk)=uk∈Uk(xk)min{exp(gk(xk,uk,wk))wkE{Jk+1(fk(xk,uk,wk))}} So
V
k
(
x
k
)
=
ln
J
k
(
x
k
)
V_k(x_k)=\ln J_k(x_k)\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad
Vk(xk)=lnJk(xk)
=
min
u
k
∈
U
k
(
x
k
)
{
g
k
(
x
k
,
u
k
,
w
k
)
+
ln
E
w
k
{
J
k
+
1
(
f
k
(
x
k
,
u
k
,
w
k
)
)
}
}
=\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{g_k(x_k,u_k,w_k)+\ln\mathop\mathbf{E}\limits_{w_k}\left\{J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\}\right\}\qquad\qquad\qquad
=uk∈Uk(xk)min{gk(xk,uk,wk)+lnwkE{Jk+1(fk(xk,uk,wk))}}
=
min
u
k
∈
U
k
(
x
k
)
{
g
k
(
x
k
,
u
k
,
w
k
)
+
ln
E
w
k
{
exp
(
V
k
+
1
(
f
k
(
x
k
,
u
k
,
w
k
)
)
)
}
}
  
□
=\mathop{\min}\limits_{u_k\in U_k(x_k)}\left\{g_k(x_k,u_k,w_k)+\ln\mathop\mathbf{E}\limits_{w_k}\left\{\exp \left(V_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right)\right\}\right\}\qquad\;\Box
=uk∈Uk(xk)min{gk(xk,uk,wk)+lnwkE{exp(Vk+1(fk(xk,uk,wk)))}}□
1.8 (Terminating Process) In the framework of the basic problem, consider the case where the system evolution terminates at time i i i when a given value w i ˉ \bar{w_i} wiˉ of the disurbance at time i i i occurs, or when a termination decesion u i u_i ui is made by the controller. If termination occurs at time i i i, the resulting cost is T + ∑ k = 0 i g k ( x k , u k , w k ) T+\sum_{k=0}^ig_k(x_k,u_k,w_k) T+k=0∑igk(xk,uk,wk)where T T T is a termination cost. If the process has not terminated up to the final time N N N, the resulting cost is g N ( x N ) + ∑ k = 0 N − 1 g k ( x k , u k , w k ) g_N(x_N)+\sum_{k=0}^{N-1}g_k(x_k,u_k,w_k) gN(xN)+∑k=0N−1gk(xk,uk,wk). Reformulate the problem into the framework of the basic problem.
1.9 (Multiplicative Cost) In the framework of the basic problem, consider the case where the cost has the multiplicative form
E
w
k
k
=
0
,
1
,
.
.
.
,
N
−
1
{
g
N
(
x
N
)
⋅
g
N
−
1
(
x
N
−
1
,
u
N
−
1
,
w
N
−
1
)
…
g
0
(
x
0
,
u
0
,
w
0
)
}
\mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,...,N-1}}\left\{g_N(x_N)\cdot g_{N-1}(x_{N-1}, u_{N-1}, w_{N-1})\ldots g_0(x_0,u_0,w_0)\right\}
k=0,1,...,N−1wkE{gN(xN)⋅gN−1(xN−1,uN−1,wN−1)…g0(x0,u0,w0)} Develop a DP-like algorithm for this problem assuming that
g
k
(
x
k
,
u
k
,
w
k
)
≥
0
g_k(x_k,u_k,w_k)\ge0
gk(xk,uk,wk)≥0 for all
x
k
,
u
k
,
w
k
x_k, u_k, w_k
xk,uk,wk and
k
k
k.
Solution. The DP-like algorithm for this problem is as follows:
J
N
(
x
N
)
=
g
N
(
x
N
)
J_N(x_N)=g_N(x_N)
JN(xN)=gN(xN)
J
k
(
x
k
)
=
E
w
k
{
g
k
(
x
k
,
u
k
,
w
k
)
J
k
+
1
(
f
k
(
x
k
,
u
k
,
w
k
)
)
}
J_k(x_k)=\mathop\mathbf{E}\limits_{w_k}\left\{g_k(x_k,u_k,w_k)J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\}
Jk(xk)=wkE{gk(xk,uk,wk)Jk+1(fk(xk,uk,wk))} The proof is standard as above, we may define
J
k
∗
(
x
k
)
=
E
w
k
i
=
k
,
.
.
.
,
N
−
1
{
g
N
(
x
N
)
…
g
i
(
x
i
,
u
i
,
w
i
)
…
g
k
(
x
k
,
u
k
,
w
k
)
}
J_k^*(x_k)=\mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{i=k,...,N-1}} \left\{g_N(x_N)\ldots g_i(x_i,u_i,w_i)\ldots g_k(x_k,u_k,w_k) \right\}
Jk∗(xk)=i=k,...,N−1wkE{gN(xN)…gi(xi,ui,wi)…gk(xk,uk,wk)} and prove
J
k
∗
(
x
k
)
=
J
k
(
x
k
)
J_k^*(x_k)=J_k(x_k)
Jk∗(xk)=Jk(xk).
□
\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\Box
□
1.10 Assume that we have a vessel whose maximum weight capacity is z z z and whose cargo is to consist of different quantities of N N N different items. Let v i v_i vi denote the value of the i i i-th type of item, w i w_i wi the weight of i i i-th type of item, and x i x_i xi the number of items of type i i i that are loaded in the vessel. The problem is to find the most valuable cargo, i.e., to maximize ∑ i = 1 N x i v i \sum_{i=1}^Nx_iv_i ∑i=1Nxivi subject to the constraints ∑ i = 1 N x i w i ≤ z \sum_{i=1}^Nx_iw_i\leq z ∑i=1Nxiwi≤z and x i = 0 , 1 , 2 , … x_i=0,1,2,\ldots xi=0,1,2,… Formulate this problem in terms of DP.
Solution. We formulate this problem in terms of DP as follows:
\qquad
a state
x
k
′
x'_k
xk′ is defined by
x
k
′
=
∑
i
=
0
k
−
1
x
i
w
i
x'_k=\mathop{\sum}\limits_{i=0}^{k-1}x_iw_i
xk′=i=0∑k−1xiwi
\qquad
an action
u
k
′
u'_k
uk′ is taken on the space
U
k
(
x
k
′
)
=
{
0
,
1
,
…
,
[
z
−
x
k
′
w
k
]
}
U_k(x'_k)=\left\{0,1,\ldots,\left[\frac{z-x'_k}{w_k}\right]\right\}
Uk(xk′)={0,1,…,[wkz−xk′]}
\qquad
a random parameter
w
k
′
w'_k
wk′ is always be 0
\qquad
state transform function is defined as
x
k
+
1
′
=
f
k
(
x
k
′
,
u
k
′
,
w
k
′
)
=
x
k
′
+
u
k
′
w
k
x'_{k+1}=f_k(x'_k,u'_k,w'_k)=x'_k+u'_kw_k
xk+1′=fk(xk′,uk′,wk′)=xk′+uk′wk
\qquad
cost function is defined as
g
k
(
x
k
′
,
u
k
′
,
w
k
′
)
=
−
u
k
′
v
k
g_k(x'_k,u'_k,w'_k)=-u'_kv_k
gk(xk′,uk′,wk′)=−uk′vk
So the problem is reformulated to get an optimal solution to the minimization problem
min
π
=
{
u
1
′
,
…
,
u
N
′
}
g
k
(
x
k
′
,
u
k
′
,
w
k
′
)
\mathop{\min}\limits_{\pi=\{u'_1,\ldots,u'_N\}}g_k(x'_k,u'_k,w'_k)
π={u1′,…,uN′}mingk(xk′,uk′,wk′)the problem can be solved by the following DP-algorithm:
J
N
+
1
(
x
N
+
1
′
)
=
0
J_{N+1}(x'_{N+1})=0\qquad\qquad
JN+1(xN+1′)=0
J
k
(
x
k
′
)
=
min
u
k
′
∈
U
k
(
x
k
′
)
{
−
u
k
′
v
k
+
J
k
+
1
(
f
k
(
x
k
′
,
u
k
′
,
w
k
′
)
)
}
  
,
k
=
1
,
2
,
…
,
N
J_k(x'_k)=\mathop{\min}\limits_{u'_k\in U_k(x'_k)}\{-u'_kv_k+J_{k+1}(f_k(x'_k,u'_k,w'_k))\}\;,\qquad k=1,2,\ldots,N
Jk(xk′)=uk′∈Uk(xk′)min{−uk′vk+Jk+1(fk(xk′,uk′,wk′))},k=1,2,…,N
□
\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\Box
□
1.11 Consider a device consisting of N N N stages connected in series, where each stage consists of a particular component. The components are subject to failure, and to increase the reliablity of the device duplicate components are provided. For j = 1 , 2 , … , N j=1,2,\ldots,N j=1,2,…,N, let ( 1 + m j ) (1+m_j) (1+mj) be the number of components for the j j j-th stage, let p j ( m j ) p_j(m_j) pj(mj) be the probability of success operation of the j j j-th stage when ( 1 + m j ) (1+m_j) (1+mj) components are used, and let c j c_j cj denote the cost of a single component at the j j j-th stage. Formulate in terms of DP the problem of finding the number of components at each stage that maximize the realibility of the device expressed by p 1 ( m 1 ) ⋅ p 2 ( m 2 ) … p N ( m N ) p_1(m_1)\cdot p_2(m_2)\ldots p_N(m_N) p1(m1)⋅p2(m2)…pN(mN) subject to the cost constraint ∑ j = 1 N c j m j ≤ A \sum_{j=1}^N c_jm_j\leq A ∑j=1Ncjmj≤A, where A > a A>a A>a is given.
1.12 (Minimization over a Subset of Policies) This problem is primarily of theoretical interest. Consider a variant of the basic problem whereby we seek min π ∈ Π ~ J k ( x 0 ) \mathop{\min}\limits_{\pi\in\tilde{\Pi}}J_k(x_0) π∈Π~minJk(x0) where Π ~ \tilde{\Pi} Π~ is some given subset of the set of sequences { μ 0 , μ 1 , … , μ N − 1 } \{\mu_0,\mu_1,\ldots, \mu_{N-1}\} {μ0,μ1,…,μN−1} of functions μ k : S k → C k \mu_k:S_k\rightarrow C_k μk:Sk→Ck with μ k ( x k ) ∈ U k ( x k ) \mu_k(x_k)\in U_k(x_k) μk(xk)∈Uk(xk) for all x k ∈ S k x_k\in S_k xk∈Sk. Assume that π ∗ = { μ 0 ∗ , μ 1 ∗ , … , μ N − 1 ∗ } \pi^*=\{\mu_0^*,\mu_1^*,\ldots, \mu_{N-1}^*\} π∗={μ0∗,μ1∗,…,μN−1∗} belongs to Π ~ \tilde{\Pi} Π~ and attains the minimum in the DP algorithm; that is, for all k = 0 , 1 , … , N − 1 k=0,1,\ldots,N-1 k=0,1,…,N−1 and x k ∈ S k x_k\in S_k xk∈Sk, J k ( x k ) = E w k { g k ( x k , μ k ∗ ( x k ) , w k ) + J k + 1 ( f k ( x k , μ k ∗ ( x k ) , w k ) ) } J_k(x_k)=\mathop{\mathbf{E}}\limits_{w_k}\left\{g_k(x_k,\mu_k^*(x_k),w_k)+J_{k+1}\left(f_k(x_k,\mu_k^*(x_k),w_k)\right)\right\} Jk(xk)=wkE{gk(xk,μk∗(xk),wk)+Jk+1(fk(xk,μk∗(xk),wk))} min u k ∈ U k ( x k ) E w k { g k ( x k , u k , w k ) + J k + 1 ( f k ( x k , u k , w k ) ) } \mathop{\min}\limits_{u_k\in U_k(x_k)}\mathop{\mathbf{E}}\limits_{w_k}\left\{g_k(x_k,u_k,w_k)+J_{k+1}\left(f_k(x_k,u_k,w_k)\right)\right\} uk∈Uk(xk)minwkE{gk(xk,uk,wk)+Jk+1(fk(xk,uk,wk))} with J N ( x N ) = g N ( x N ) J_N(x_N)=g_N(x_N) JN(xN)=gN(xN). Assume further that the function J k J_k Jk are real-valued and that the preceding expectations are well-defined and finite. Show that π ∗ \pi^* π∗ is optimal within Π ~ \tilde{\Pi} Π~ and that the corresponding optimal cost is equal to J 0 ( x 0 ) J_0(x_0) J0(x0).
1.13 (Semilinear Systems) Consider a problem involving the system x k + 1 = A k x k + f k ( u k ) + w k x_{k+1}=A_kx_k+f_k(u_k)+w_k xk+1=Akxk+fk(uk)+wk where x k ∈ R n x_k\in\mathfrak{R}^n xk∈Rn, f k f_k fk are given functions, and A k A_k Ak and w k w_k wk are random n × n n\times n n×n matices and n n n-vectors, respectively, with given probability distributions that do not depend on x k , u k x_k, u_k xk,uk or prior values of A k A_k Ak and w k w_k wk. Assume that the cost is of the form E A k , w k k = 0 , 1 , … , N − 1 { c N ′ x n + ∑ k = 0 N − 1 ( c k ′ x k + g k ( μ k ( x k ) ) ) } \mathop{\mathbf{E}}\limits_{\mathop{A_k,w_k}\limits_{k=0,1,\ldots,N-1}}\left\{c'_Nx_n+\sum_{k=0}^{N-1}\left(c'_kx_k+g_k\left(\mu_k(x_k)\right)\right)\right\} k=0,1,…,N−1Ak,wkE{cN′xn+k=0∑N−1(ck′xk+gk(μk(xk)))}where c k c_k ck are given vectors and g k g_k gk are given functions. Show that if the optimal cost for this problem is finite and the control constraint sets U k ( x k ) U_k(x_k) Uk(xk) are independent of x k x_k xk, then the cost-to-go functions of the DP algorithm are affine (linear plus constraint). Assuming that there is at least one optimal policy, show that there exists an optimal policy that consist of constant functions μ k ∗ \mu_k^* μk∗ that is μ k ∗ ( x k ) = \mu_k^*(x_k)= μk∗(xk)= constant for all x k ∈ R n x_k\in\mathfrak{R}^n xk∈Rn.
1.14 A farmer annually producing
x
k
x_k
xk units of a certain crop stores
(
1
−
u
k
)
x
k
(1-u_k)x_k
(1−uk)xk units of his production, where
0
≤
u
k
≤
1
0\leq u_k\leq 1
0≤uk≤1 and invests the remaining
u
k
x
k
u_kx_k
ukxk units, thus increasing the next year’s production to a level
x
k
+
1
x_{k+1}
xk+1 given by
x
k
+
1
=
x
k
+
w
k
u
k
x
k
,
k
=
0
,
1
,
⋯
 
,
N
−
1
x_{k+1}=x_k+w_ku_kx_k,\qquad k=0,1,\cdots,N-1
xk+1=xk+wkukxk,k=0,1,⋯,N−1The scalars
w
k
w_k
wk are independent random variables with identical probability distributions that do not depend either on
x
k
x_k
xk or
u
k
u_k
uk. Furthermore,
E
{
w
k
}
=
w
‾
>
0
\mathbf{E}\{w_k\}=\overline{w}>0
E{wk}=w>0. The problem is to find the optimal investment policy that maximizes the total expected product stored over
N
N
N years
E
w
k
k
=
0
,
1
,
…
,
N
−
1
{
x
N
+
∑
k
=
0
N
−
1
(
1
−
u
k
)
x
k
}
\mathop\mathbf{E}\limits_{\mathop{w_k}\limits_{k=0,1,\ldots,N-1}}\left\{x_N+\sum_{k=0}^{N-1}(1-u_k)x_k\right\}
k=0,1,…,N−1wkE{xN+k=0∑N−1(1−uk)xk} Show the optimality of the following policy that consists of constant functions:
(1) If
w
‾
>
1
\overline{w}>1
w>1,
μ
0
∗
(
x
0
)
=
⋯
=
μ
N
−
1
∗
(
x
N
−
1
)
=
1
\mu_0^*(x_0)=\cdots=\mu_{N-1}^*(x_{N-1})=1
μ0∗(x0)=⋯=μN−1∗(xN−1)=1.
(2) If
0
<
w
‾
<
1
/
N
0<\overline{w}<1/N
0<w<1/N,
μ
0
∗
(
x
0
)
=
⋯
=
μ
N
−
1
∗
(
x
N
−
1
)
=
0
\mu_0^*(x_0)=\cdots=\mu_{N-1}^*(x_{N-1})=0
μ0∗(x0)=⋯=μN−1∗(xN−1)=0.
(3) If
1
/
N
≤
w
‾
≤
1
1/N\leq\overline{w}\leq 1
1/N≤w≤1,
μ
0
∗
(
x
0
)
=
⋯
=
μ
N
−
k
ˉ
−
1
∗
(
x
N
−
k
ˉ
−
1
)
=
1
\mu_0^*(x_0)=\cdots=\mu_{N-\bar{k}-1}^*(x_{N-\bar{k}-1})=1
μ0∗(x0)=⋯=μN−kˉ−1∗(xN−kˉ−1)=1
μ
N
−
k
ˉ
∗
(
x
N
−
k
ˉ
)
=
⋯
=
μ
N
−
1
∗
(
x
N
−
1
)
=
0
\mu_{N-\bar{k}}^*(x_{N-\bar{k}})=\cdots=\mu_{N-1}^*(x_{N-1})=0
μN−kˉ∗(xN−kˉ)=⋯=μN−1∗(xN−1)=0 where
k
ˉ
\bar{k}
kˉ is such that
1
/
(
k
ˉ
+
1
)
<
w
‾
≤
1
/
k
ˉ
1/{(\bar{k}+1)}<\overline{w}\leq 1/{\bar{k}}
1/(kˉ+1)<w≤1/kˉ.
Solution. The DP-algorithm can be formed as follows:
J
N
(
x
N
)
=
x
N
J_N(x_N)=x_N
JN(xN)=xN
J
k
(
x
k
)
=
max
u
k
E
w
k
{
(
1
−
u
k
)
x
k
+
J
k
+
1
(
x
k
+
w
k
u
k
x
k
)
}
 
,
    
k
=
0
,
1
,
⋯
 
,
N
−
1
J_k(x_k)=\mathop{\max}\limits_{u_k}\mathop{\mathbf E}\limits_{w_k}\left\{(1-u_k)x_k+J_{k+1}(x_k+w_ku_kx_k)\right\}\,,\;\; k=0,1,\cdots,N-1
Jk(xk)=ukmaxwkE{(1−uk)xk+Jk+1(xk+wkukxk)},k=0,1,⋯,N−1 (1) In the case
w
‾
>
1
\overline{w}>1
w>1
J
N
−
1
(
x
N
−
1
)
=
max
u
N
−
1
E
w
N
−
1
{
(
1
−
u
N
−
1
)
x
N
−
1
+
x
N
−
1
+
w
N
−
1
u
N
−
1
x
N
−
1
}
J_{N-1}(x_{N-1})=\mathop{\max}\limits_{u_{N-1}}\mathop{\mathbf E}\limits_{w_{N-1}}\left\{(1-u_{N-1})x_{N-1}+x_{N-1}+w_{N-1}u_{N-1}x_{N-1}\right\}
JN−1(xN−1)=uN−1maxwN−1E{(1−uN−1)xN−1+xN−1+wN−1uN−1xN−1}
=
max
u
N
−
1
E
w
N
−
1
{
2
x
N
−
1
+
(
w
N
−
1
−
1
)
u
N
−
1
x
N
−
1
}
=\mathop{\max}\limits_{u_{N-1}}\mathop{\mathbf E}\limits_{w_{N-1}}\left\{2x_{N-1}+(w_{N-1}-1)u_{N-1}x_{N-1}\right\}
=uN−1maxwN−1E{2xN−1+(wN−1−1)uN−1xN−1}
=
max
u
N
−
1
{
2
x
N
−
1
+
(
w
‾
−
1
)
u
N
−
1
x
N
−
1
}
=\mathop{\max}\limits_{u_{N-1}}\left\{2x_{N-1}+(\overline{w}-1)u_{N-1}x_{N-1}\right\}\qquad\qquad
=uN−1max{2xN−1+(w−1)uN−1xN−1} we obtain that
μ
N
−
1
∗
(
x
N
−
1
)
=
1
\mu_{N-1}^*(x_{N-1})=1
μN−1∗(xN−1)=1 and
J
N
−
1
(
x
N
−
1
)
=
(
w
‾
+
1
)
x
N
−
1
J_{N-1}(x_{N-1})=(\overline{w}+1)x_{N-1}
JN−1(xN−1)=(w+1)xN−1. By indunction and similar caculations, we can get
μ
k
∗
(
x
k
)
=
1
  
,
  
J
k
(
x
k
)
=
(
w
‾
+
N
−
k
)
x
k
    
for all
  
k
=
0
,
1
,
⋯
 
,
N
−
1
\mu_k^*(x_k)=1\;,\;J_k(x_k)=(\overline{w}+N-k)x_k\;\;\text{for all }\;k=0,1,\cdots,N-1
μk∗(xk)=1,Jk(xk)=(w+N−k)xkfor all k=0,1,⋯,N−1(2) In the case
w
‾
<
1
/
N
\overline{w}<1/N
w<1/N,
J
N
−
1
(
x
N
−
1
)
=
max
u
N
−
1
{
2
x
N
−
1
+
(
w
‾
−
1
)
u
N
−
1
x
N
−
1
}
=
2
x
N
−
1
J_{N-1}(x_{N-1})=\mathop{\max}\limits_{u_{N-1}}\left\{2x_{N-1}+(\overline{w}-1)u_{N-1}x_{N-1}\right\}=2x_{N-1}
JN−1(xN−1)=uN−1max{2xN−1+(w−1)uN−1xN−1}=2xN−1 with
μ
N
−
1
∗
(
x
N
−
1
)
=
0
\mu_{N-1}^*(x_{N-1})=0
μN−1∗(xN−1)=0. And then
J
N
−
2
(
x
N
−
2
)
=
max
u
N
−
2
E
w
N
−
2
{
(
1
−
u
N
−
2
)
x
N
−
2
+
2
[
x
N
−
2
+
w
N
−
2
u
N
−
2
x
N
−
2
]
}
J_{N-2}(x_{N-2})=\mathop{\max}\limits_{u_{N-2}}\mathop{\mathbf E}\limits_{w_{N-2}}\left\{(1-u_{N-2})x_{N-2}+2[x_{N-2}+w_{N-2}u_{N-2}x_{N-2}]\right\}
JN−2(xN−2)=uN−2maxwN−2E{(1−uN−2)xN−2+2[xN−2+wN−2uN−2xN−2]}
=
max
u
N
−
2
{
3
x
N
−
2
+
(
2
w
‾
−
1
)
u
N
−
2
x
N
−
2
}
=
3
x
N
−
2
=\mathop{\max}\limits_{u_{N-2}}\left\{3x_{N-2}+(2\overline{w}-1)u_{N-2}x_{N-2}\right\}=3x_{N-2}
=uN−2max{3xN−2+(2w−1)uN−2xN−2}=3xN−2 with
μ
N
−
2
∗
(
x
N
−
2
)
=
0
\mu_{N-2}^*(x_{N-2})=0
μN−2∗(xN−2)=0. Since
w
‾
<
1
/
N
\overline{w}<1/N
w<1/N, we can prove by induction that for all
k
=
0
,
1
,
⋯
 
,
N
−
1
k=0,1,\cdots,N-1
k=0,1,⋯,N−1,
μ
k
∗
(
x
k
)
=
0
  
,
  
J
k
(
x
k
)
=
(
N
−
k
+
1
)
x
k
\mu_k^*(x_k)=0\;,\;J_k(x_k)=(N-k+1)x_k
μk∗(xk)=0,Jk(xk)=(N−k+1)xk (3) In the case
1
/
N
≤
w
‾
≤
1
1/N\leq\overline{w}\leq 1
1/N≤w≤1, there is some
k
=
k
ˉ
k=\bar{k}
k=kˉ satisfying
1
/
(
k
ˉ
+
1
)
<
w
‾
≤
1
/
k
ˉ
1/{(\bar{k}+1)}<\overline{w}\leq 1/{\bar{k}}
1/(kˉ+1)<w≤1/kˉ, after which step the induction argument in (2) will stop.
For any
k
≥
N
−
k
ˉ
k\ge N-\bar{k}
k≥N−kˉ, we have
J
k
(
x
k
)
=
max
u
k
{
(
N
−
k
+
1
)
x
k
+
(
(
N
−
k
)
w
‾
−
1
)
u
k
x
k
}
J_k(x_k)=\mathop{\max}\limits_{u_k}\left\{(N-k+1)x_k+((N-k)\overline{w}-1)u_kx_k\right\}
Jk(xk)=ukmax{(N−k+1)xk+((N−k)w−1)ukxk}Since
(
N
−
k
)
w
‾
≤
k
ˉ
w
‾
≤
1
(N-k)\overline{w}\leq\bar{k}\overline{w}\leq 1
(N−k)w≤kˉw≤1, then
μ
k
∗
(
x
k
)
=
0
\mu_k^*(x_k)=0
μk∗(xk)=0 and
J
k
(
x
k
)
=
(
N
−
k
+
1
)
x
k
J_k(x_k)=(N-k+1)x_k
Jk(xk)=(N−k+1)xk.
Next for
k
=
N
−
k
ˉ
−
1
k= N-\bar{k}-1
k=N−kˉ−1
J
N
−
k
ˉ
−
1
(
x
N
−
k
ˉ
−
1
)
=
max
u
N
−
k
ˉ
−
1
{
(
k
ˉ
+
2
)
x
N
−
k
ˉ
−
1
+
(
(
k
ˉ
+
1
)
w
‾
−
1
)
u
N
−
k
ˉ
−
1
x
N
−
k
ˉ
−
1
}
J_{N-\bar{k}-1}(x_{N-\bar{k}-1})=\mathop{\max}\limits_{u_{N-\bar{k}-1}}\left\{(\bar{k}+2)x_{N-\bar{k}-1}+((\bar{k}+1)\overline{w}-1)u_{N-\bar{k}-1}x_{N-\bar{k}-1}\right\}
JN−kˉ−1(xN−kˉ−1)=uN−kˉ−1max{(kˉ+2)xN−kˉ−1+((kˉ+1)w−1)uN−kˉ−1xN−kˉ−1} In this case,
(
k
ˉ
+
1
)
w
‾
−
1
>
0
(\bar{k}+1)\overline{w}-1>0
(kˉ+1)w−1>0, yielding
μ
N
−
k
ˉ
−
1
∗
(
x
N
−
k
ˉ
−
1
)
=
1
\mu_{N-\bar{k}-1}^*(x_{N-\bar{k}-1})=1
μN−kˉ−1∗(xN−kˉ−1)=1 and
J
N
−
k
ˉ
−
1
(
x
N
−
k
ˉ
−
1
)
=
[
(
k
ˉ
+
1
)
w
‾
+
k
ˉ
+
1
]
x
N
−
k
ˉ
−
1
J_{N-\bar{k}-1}(x_{N-\bar{k}-1})=[(\bar{k}+1)\overline{w}+\bar{k}+1]x_{N-\bar{k}-1}
JN−kˉ−1(xN−kˉ−1)=[(kˉ+1)w+kˉ+1]xN−kˉ−1.
And then by induction, we can prove that for all
k
<
N
−
k
ˉ
−
1
k<N-\bar{k}-1
k<N−kˉ−1,
μ
k
∗
(
x
k
)
=
1
\mu_k^*(x_k)=1
μk∗(xk)=1.
□
\qquad\qquad\qquad\qquad\Box
□
1.15 Let
x
k
x_k
xk denote the number of educators in a certain country at time
k
k
k and let
y
k
y_k
yk denote the number of research scientists at time
k
k
k. New scientists (potential educators or research scientists) are produced during the
k
k
kth period by educators at a rate
γ
k
\gamma_k
γk per educator, while educators and research scientists leave the field due to death, retirement, and transfer at a rate
δ
k
\delta_k
δk. The scalars
γ
k
\gamma_k
γk,
k
=
0
,
1
,
.
.
.
,
N
−
1
k = 0,1, ... ,N-1
k=0,1,...,N−1, are independent identically distributed random variables taking values within a closed and bounded interval of positive numbers. Similarly
δ
k
\delta_k
δk,
k
=
0
,
1
,
.
.
.
,
N
−
1
k = 0,1, ... ,N-1
k=0,1,...,N−1, are independent identically distributed and take values in an interval
[
δ
,
δ
′
]
[\delta,\delta']
[δ,δ′] with
0
<
δ
≤
δ
′
<
1
0<\delta\leq\delta'<1
0<δ≤δ′<1. By means of incentives, a science policy maker can determine the proportion
u
k
u_k
uk of new scientists produced at time
k
k
k who become educators. Thus, the number of research scientists and educators evolves according to the equations
x
k
+
1
=
(
1
−
δ
k
)
x
k
+
u
k
γ
k
x
k
x_{k+1}=(1-\delta_k)x_k+u_k\gamma_k x_k
xk+1=(1−δk)xk+ukγkxk
y
k
+
1
=
(
1
−
δ
k
)
y
k
+
(
1
−
u
k
)
γ
k
x
k
y_{k+1}=(1-\delta_k)y_k+(1-u_k)\gamma_k x_k
yk+1=(1−δk)yk+(1−uk)γkxk The initial numbers
x
0
,
y
0
x_0,y_0
x0,y0 are known, and it is required to find a policy
{
μ
0
∗
(
x
0
,
y
0
)
,
⋯
 
,
μ
N
−
1
∗
(
x
N
−
1
,
y
N
−
1
)
}
\{\mu_0^*(x_0,y_0),\cdots,\mu_{N-1}^*(x_{N-1},y_{N-1})\}
{μ0∗(x0,y0),⋯,μN−1∗(xN−1,yN−1)} with
0
<
α
≤
μ
k
∗
(
x
k
,
y
k
)
≤
β
<
1
,
for all
x
k
,
y
k
and
k
0<\alpha\leq \mu_k^*(x_k,y_k)\leq\beta<1,\qquad\text{for all }x_k,y_k\text{ and }k
0<α≤μk∗(xk,yk)≤β<1,for all xk,yk and k which maximizes
E
γ
k
,
δ
k
{
y
N
}
\mathbf{E}_{\gamma_k,\delta_k}\{y_N\}
Eγk,δk{yN} (i.e. the expected final number of research scientists after
N
N
N periods). The scalars
α
\alpha
α and
β
\beta
β are given.
(1) Show that the cost-to-go functions
J
k
(
x
k
,
y
k
)
J_k(x_k,y_k)
Jk(xk,yk) are linear; that is, for some scalars
ξ
k
,
ζ
k
,
\xi_k,\zeta_k,
ξk,ζk,
J
k
(
x
k
,
y
k
)
=
ξ
k
x
k
+
ζ
k
y
k
J_k(x_k,y_k)=\xi_kx_k+\zeta_ky_k
Jk(xk,yk)=ξkxk+ζkyk (2) Derive an optimal policy
{
μ
0
∗
,
⋯
 
,
μ
N
−
1
∗
}
\{\mu_0^*,\cdots,\mu_{N-1}^*\}
{μ0∗,⋯,μN−1∗}, under the assumption
E
{
γ
k
}
>
E
{
δ
k
}
\mathbf{E}\{\gamma_k\}>\mathbf{E}\{\delta_k\}
E{γk}>E{δk} and show that this optimal policy can consist of constant functions.
(3) Assume that the proportion of new scientists who become educators at time
k
k
k is
u
k
+
ϵ
k
u_k+\epsilon_k
uk+ϵk (rather than
u
k
u_k
uk), where
ϵ
k
\epsilon_k
ϵk are identically distributed independent random variables that are also independent of
γ
k
,
δ
k
\gamma_k,\delta_k
γk,δk and take values in the interval
[
−
α
,
1
−
β
]
[-\alpha,1-\beta]
[−α,1−β]. Derive the form of the cost-to-go functions and the optimal policy.