Dynamic Programming and Optimal Control 第三章习题

3.1 Solve the problem of Example 3.2.1 for the case where the cost function is $$(x(T){)}^2+{\int }_0^T(u(t){)}^{2}dt$$Also, calculate the cost-to-go function $J^{\ast }(t,x)$ and verify that it satisfies the HJB equation.
Solution. The scalar system $$\dot{x}(t)=u(t)$$ with the constaint $|u(t)|\le 1$ for all $t\in [0,T]$.

3.2 A young investor has earned in the stock market a large amount if money $S$ and plans to spend it so as to maximize his enjoyment through the rest of his life without working. He estimates that he will live exactly $T$ more than years and that his capital $x(t)$ should be reduced to zero at time $T$, i.e., $x(T)=0$. Also he models the evolution of his capital by the differential equation $$\frac{dx(t)}{dt}=\alpha x(t)-u(t)$$ where $x(0)=S$ is his initial capital, $\alpha >0$ is a given interest rate, and $u(t)\ge 0$ is his rate of expenditure. The total enjoyment he will obtain is given by $${\int }_0^T{e}^{-\beta t}\sqrt{u(t)}dt$$ Here $\beta$ is some positive scalar, which serves to discount future enjoyment. Find the optimal { u ( t ) ∣ t ∈ [ 0 , T ] } \{u(t)|t\in[0,T]\} {u(t)∣t∈[0,T]}.
Solution. We have $$f(x,u)=\alpha x-u,g(x,u)=e^{-\beta t}\sqrt{u}$$giving the Hamiltonian as follows: $$H(x,u,p)=e^{-\beta t}\sqrt{u}+p(\alpha x-u)$$ and the adjoint equation is
$$\dot{p}(t)=-\alpha p(t)$$ yielding $$p(t)=C_1{e}^{-\alpha t}\text{for some constant }{C}_1$$Notice that here $x(T)=0$ is given, so $p(T)=\nabla (h(x^{\ast }(T)))=0$ is not true anymore.
\qquad The optimal control is obtained by maximizing the Hamiltonian with respect to $u$, yielding
$$u^{\ast }(t)=\arg \underset{u}{\max }\left[e^{-\beta t}\sqrt{u}+C_1{e}^{-\alpha t}(\alpha x^{\ast }-u)\right]=\frac{e^{(\alpha -\beta )t}}{2C_1}(3.2.1)$$Then by the differiential equation of the system we get$${\dot{x}}^{\ast }(t)=\alpha x^{\ast }(t)-\frac{e^{(\alpha -\beta )t}}{2C_1}$$Solving this equation, we obtain
$$x^{\ast }(t)=C_2{e}^{\alpha t}+\frac{e^{(\alpha -\beta )t}}{2C_1\beta }\text{for some constant }{C}_2$$And together with the initial condition $x^{\ast }(0)=S$ and the final condition $x^{\ast }(T)=0$, we can get the exact values of $C_1$ and $C_2$. So $u^{\ast }(t)$ in (3.2.1) gives the optimal control. $\square$

3.9 Use the Minimum Principle to solve the linear-quadratic problem of Example 3.2.2.
Solution. The $n$-dimension linear-quadratic system is given by
$$\dot{x}(t)=Ax(t)+Bu(t)$$ where $A$ and $B$ are given matrices, and the quadratic cost
$$x(T{)}^{\prime }{Q}_{T}x(T)+{\int }_0^T\left(x(t{)}^{\prime }Qx(t)+u(t{)}^{\prime }Ru(t)\right)dt$$ where the matrices $Q_T$ and $Q$ are symmetric positive semidefinite, and the matrix $R$ is symmetric positive definite.
\qquad The Hamiltonian here is $$H(x,u,p)=x^{\prime }Qx+u^{\prime }Ru+p^{\prime }(Ax+Bu)$$ and the adjoint equation is
$$\dot{p}(t)=2Qx+A^{\prime }p(t)(1)$$ with the terminal conditon $$p(T)=\nabla h(x^{\ast }(T))=2Q_T{x}^{\ast }(T)$$The optimal control can be obtained by minimizing the Hamiltonian with respect to $u$, yielding
u ∗ ( t ) = arg ⁡ min ⁡ u { x ∗ ( t ) ′ Q x ∗ ( t ) + u ′ R u + p ′ ( A x ∗ ( t ) + B u ) } u^*(t)=\arg\min_{u}\left\{x^*(t)'Qx^*(t)+u'Ru+p'(Ax^*(t)+Bu)\right\} u∗(t)=argumin​{x∗(t)′Qx∗(t)+u′Ru+p′(Ax∗(t)+Bu)} Since ∇ u { x ∗ ( t ) ′ Q x ∗ ( t ) + u ′ R u + p ′ ( A x ∗ ( t ) + B u ) } = 2 R u + B ′ p \nabla_u\{x^*(t)'Qx^*(t)+u'Ru+p'(Ax^*(t)+Bu)\}=2Ru+B'p ∇u​{x∗(t)′Qx∗(t)+u′Ru+p′(Ax∗(t)+Bu)}=2Ru+B′p, we get $$u^{\ast }(t)=-\frac{1}{2}{R}^{-1}{B}^{\prime }p(t)(2)$$ together with the system function leading to $${\dot{x}}^{\ast }(t)=A{x}^{\ast }(t)-\frac{1}{2}B{R}^{-1}{B}^{\prime }p(t)(3)$$ So $p(t)$ can be solved by (1) (But I don’t know the answer!!) , and then $x^{\ast }(t)$ can be solved by (3).

3.11 Use the discrete-time Minimum Principle to solve Exercise 1.14 of Chapter 1, assuming that each $w_k$ is fixed at a known deterministic value.
Solution. Let $w_k=\overline{w}$ for some fixed number $\overline{w}>0$, the system is characterized by $$x_{k+1}=f_k(x_k,u_k)=x_k+\overline{w}{u}_k{x}_k$$ and the cost functiom becomes $$J(u)=x_N+\sum _{k=0}^{N-1}(1-u_k)x_k$$ Then the Hamiltonian function can be written as $$H_k(x_k,u_k,p_{k+1})=(1-u_k)x_k+p_{k+1}(x_k+\overline{w}{u}_k{x}_k)$$ By the Discrete-time Minimum Principle, for $k=0,1,\cdots ,N-1$, we have $$u_k^{\ast }=\arg \underset{u_k}{\max }{H}_k(x_k^{\ast },u_k,p_{k+1})$$ $$=\arg \underset{u_k}{\max }\left[(p_{k+1}\overline{w}-1)u_k{x}_k+(p_{k+1}+1)x_k\right]$$ $$=\{\begin{array}{ll}1,& \text{ if }{p}_{k+1}\overline{w}>1\\ 0,& \text{ if }{p}_{k+1}\overline{w}\le 1\end{array}(3.11.1)$$ On the other hand, for $k=0,1,\cdots ,N-1$, the adjoint equation reads $$p_k={\nabla }_{x_k}{H}_k(x_k^{\ast },u_k^{\ast },p_{k+1})=(p_{k+1}\overline{w}-1)u_k^{\ast }+p_{k+1}+1(3.11.2)$$ with the terminal condition $p_N={\nabla }_{g_N}(x_N^{\ast })=1.$
Combing (3.11.1) with (3.11.2), we can obtain the following argument
$$p_{k+1}\overline{w}>1\Rightarrow {\mu }_k^{\ast }=1\Rightarrow p_k=(\overline{w}+1)p_{k+1}(3.11.3)$$ $$p_{k+1}\overline{w}\le 1\Rightarrow {\mu }_k^{\ast }=0\Rightarrow p_k=p_{k+1}+1(3.11.4)$$ So by induction, we can easily conclude the following optimal control results:
(1) If $\overline{w}>1$, $u_0^{\ast }=\cdots =u_{N-1}^{\ast }=1$.
(2) If $0<\overline{w}<1/N$, $u_0^{\ast }=\cdots =u_{N-1}^{\ast }=0$.
(3) If $1/N\le \overline{w}\le 1$, $$u_0^{\ast }=\cdots =u_{N-\bar{k}-1}^{\ast }=1$$$$u_{N-\bar{k}}^{\ast }=\cdots =u_{N-1}^{\ast }=0$$ where $\bar{k}$ is such that $1/(\bar{k}+1)<\overline{w}\le 1/\bar{k}$. $\square$

3.12 Use the discrete-time Minimum Principle to solve Exercise 1.15 of Chapter 1, assuming that each ${\gamma }_k$ and ${\delta }_k$ are fixed at a known deterministic values.