自适应动态规划（五)-初值不为零的值迭代

初值不为零的值迭代稳定性证明

定理一

问题描述

假设初值为任意半正定函数
$$V_0(x_k)=\Psi (x_k)$$
定义变量 $\underline{\gamma },\overline{\gamma },\underline{\delta }$ 和$\overline{\delta }$ 变量如下：
KaTeX parse error: No such environment: equation at position 8: \begin{̲e̲q̲u̲a̲t̲i̲o̲n̲}̲ 0<\underline{\…
如果对于任意的 $x_k$，上述变量都满足下面
$$\begin{gathered} \underline{\gamma }U(x_k,u_k)\le J^{\ast }(F(x_k,u_k))\le \overline{\gamma }U(x_k,u_k) \\ \underline{\delta }{J}^{\ast }(x_k)\le V_0(x_k)\le \overline{\delta }{J}^{\ast }(x_k) \end{gathered}$$
证明下面的式子成立
$$(1+\frac{\underline{\delta }-1}{(1+{\overline{\gamma }}^{-1}{)}^i})J^{\ast }(x_k)\le V_i(x_k)\le (1+\frac{\overline{\delta }-1}{(1+{\underline{\gamma }}^{-1}{)}^i})J^{\ast }(x_k)$$

证明

这个证明过程分为两步，左边和右边成立证明。

第一步证明左边成立：

用数学归纳法进行证明

一、
V 1 ( x k ) = min ⁡ u k { U ( x k , u k ) + V 0 ( x k + 1 ) } ≥ min ⁡ u k { U ( x k , u k ) + δ ‾ J ∗ ( x k + 1 ) } ≥ min ⁡ u k { U ( x k , u k ) + δ ‾ J ∗ ( x k + 1 ) + δ ‾ − 1 1 + γ ‾ ( γ ‾ U ( x k , u k ) − J ∗ ( x k + 1 ) ) } ≥ min ⁡ u k { ( 1 + γ ‾ δ ‾ − 1 1 + γ ‾ ) U ( x k , u k ) + ( δ ‾ − δ ‾ − 1 1 + γ ‾ ) J ∗ ( x k + 1 ) }           ( 1 ) = ( 1 + γ ‾ δ ‾ − 1 1 + γ ‾ ) min ⁡ u k { U ( x k , u k ) + J ∗ ( x k + 1 ) }             ( 2 ) = ( 1 + γ ‾ δ ‾ − 1 1 + γ ‾ ) J ∗ ( x k ) \begin{aligned} V_1(x_k)&=\min_{u_k}\{U(x_k,u_k)+V_0(x_{k+1})\} \\ &\geq \min_{u_k}\{U(x_k,u_k)+\underline{\delta}J^*(x_{k+1})\} \\ &\geq\min_{u_k}\{U(x_k,u_k)+\underline{\delta}J^*(x_{k+1})+\frac{\underline{\delta}-1}{1+\overline{\gamma}}(\overline{\gamma}U(x_k,u_k)-J^*(x_{k+1}))\} \\ &\geq \min_{u_k}\{(1+\overline{\gamma}\frac{\underline{\delta}-1}{1+\overline{\gamma}})U(x_k,u_k)+(\underline{\delta}-\frac{\underline{\delta}-1}{1+\overline{\gamma}})J^*(x_{k+1})\} ~~~~~~~~~(1)\\ &= (1+\overline{\gamma}\frac{\underline{\delta}-1}{1+\overline{\gamma}})\min_{u_k}\{U(x_k,u_k)+J^*(x_{k+1})\} ~~~~~~~~~~~(2) \\ &=(1+\overline{\gamma}\frac{\underline{\delta}-1}{1+\overline{\gamma}})J^*(x_k) \end{aligned} V1​(xk​)​=uk​min​{U(xk​,uk​)+V0​(xk+1​)}≥uk​min​{U(xk​,uk​)+δ​J∗(xk+1​)}≥uk​min​{U(xk​,uk​)+δ​J∗(xk+1​)+1+γ​δ​−1​(γ​U(xk​,uk​)−J∗(xk+1​))}≥uk​min​{(1+γ​1+γ​δ​−1​)U(xk​,uk​)+(δ​−1+γ​δ​−1​)J∗(xk+1​)}         (1)=(1+γ​1+γ​δ​−1​)uk​min​{U(xk​,uk​)+J∗(xk+1​)}           (2)=(1+γ​1+γ​δ​−1​)J∗(xk​)​
其中(1)由下式推导可得
$$\begin{array}{rl}(1+\overline{\gamma }\frac{\underline{\delta }-1}{1+\overline{\gamma }})U(x_k,u_k)+(\underline{\delta }-\frac{\underline{\delta }-1}{1+\overline{\gamma }})J^{\ast }(x_{k+1})& =U(x_k,u_k)+\overline{\gamma }\frac{\underline{\delta }-1}{1+\overline{\gamma }}U(x_k,u_k)+\underline{\delta }{J}^{\ast }(x_{k+1})-\frac{\underline{\delta }-1}{1+\overline{\gamma }}{J}^{\ast }(x_{k+1})\\ & =U(x_k,u_k)+\underline{\delta }{J}^{\ast }(x_{k+1})+\frac{\underline{\delta }-1}{1+\overline{\gamma }}(\overline{\gamma }U(x_k,u_k)-J^{\ast }(x_{k+1}))\\ & \le U(x_k,u_k)+\underline{\delta }{J}^{\ast }(x_{k+1})\end{array}$$
上式中
$$\begin{gathered} \frac{\underline{\delta }-1}{1+\overline{\gamma }}<0 \\ \overline{\gamma }U(x_k,u_k)-J^{\ast }(x_{k+1})>0 \end{gathered}$$
其中（2）式可由下式推导可以得出
$$\begin{array}{rl}\underline{\delta }-\frac{\underline{\delta }-1}{1+\overline{\gamma }}& =\frac{\underline{\delta }(1+\overline{\gamma })-\underline{\delta }+1}{1+\overline{\gamma }}=\frac{\underline{\delta }+\underline{\delta }\overline{\gamma }-\underline{\delta }+1}{1+\overline{\gamma }}\\ & =\frac{\underline{\delta }\overline{\gamma }+1+\overline{\gamma }-\overline{\gamma }}{1+\overline{\gamma }}=1+\frac{\underline{\delta }\overline{\gamma }-\overline{\gamma }}{1+\overline{\gamma }}=1+\frac{\underline{\delta }-1}{1+{\overline{\gamma }}^{-1}}\end{array}$$
二、

假设结论对$i=l-1,l=1,2,\cdots$ 都成立，则 $i=l$ 可得

由假设得
$$V_{l-1}(x_k)\ge (1+\frac{\underline{\delta }-1}{(1+{\overline{\gamma }}^{-1}{)}^{l-1}})J^{\ast }(x_k)$$

V l = min ⁡ u k { U ( x k , u k ) + V l − 1 ( x k + 1 ) } ≥ min ⁡ u k { U ( x k , u k ) + ( 1 + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l − 1 ) J ∗ ( x k ) + δ ‾ − 1 ( 1 + γ ‾ ) ( 1 + γ ‾ − 1 ) l − 1 ( γ ‾ U ( x k , u k ) − J ∗ ( x k + 1 ) ) } = min ⁡ u k { U ( x k , u k ) + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l U ( x k , u k ) + ( 1 + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l − 1 − δ ‾ − 1 ( 1 + γ ‾ ) ( 1 + γ ‾ − 1 ) l − 1 ) J ∗ ( x k + 1 ) } = min ⁡ u k { U ( x k , u k ) + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l U ( x k , u k ) + ( 1 + ( δ ‾ − 1 ) ( 1 + γ ‾ ) − δ ‾ + 1 ( 1 + γ ‾ ) ( 1 + γ ‾ − 1 ) l − 1 ) J ∗ ( x k + 1 ) } = min ⁡ u k { U ( x k , u k ) + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l U ( x k , u k ) + ( 1 + ( δ ‾ − 1 ) γ ‾ ( 1 + γ ‾ ) ( 1 + γ ‾ − 1 ) l − 1 ) J ∗ ( x k + 1 ) } = min ⁡ u k { U ( x k , u k ) + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l U ( x k , u k ) + ( 1 + ( δ ‾ − 1 ) ( 1 + γ ‾ − 1 ) l ) J ∗ ( x k + 1 ) } = ( 1 + ( δ ‾ − 1 ) ( 1 + γ ‾ − 1 ) l ) min ⁡ u k { U ( x k , u k ) + + J ∗ ( x k + 1 ) } = ( 1 + ( δ ‾ − 1 ) ( 1 + γ ‾ − 1 ) l ) J ∗ ( x k ) \begin{aligned} V_{l}&=\min_{u_k}\{U(x_k,u_k)+V_{l-1}(x_{k+1})\} \\ &\geq \min_{u_k}\{U(x_k,u_k)+(1+\frac{\underline{\delta}-1}{(1+\overline{\gamma}^{-1})^{l-1}})J^*(x_k)+\frac{\underline{\delta}-1}{(1+\overline{\gamma})(1+\overline{\gamma}^{-1})^{l-1}}(\overline{\gamma}U(x_k,u_k)-J^*(x_{k+1}))\} \\ &=\min_{u_k}\{U(x_k,u_k)+\frac{\underline{\delta}-1}{(1+\overline{\gamma}^{-1})^{l}}U(x_k,u_k)+(1+\frac{\underline{\delta}-1}{(1+\overline{\gamma}^{-1})^{l-1}}-\frac{\underline{\delta}-1}{(1+\overline{\gamma})(1+\overline{\gamma}^{-1})^{l-1}})J^*(x_{k+1})\} \\ &=\min_{u_k}\{U(x_k,u_k)+\frac{\underline{\delta}-1}{(1+\overline{\gamma}^{-1})^{l}}U(x_k,u_k)+(1+\frac{(\underline{\delta}-1)(1+\overline{\gamma})-\underline{\delta}+1}{(1+\overline{\gamma})(1+\overline{\gamma}^{-1})^{l-1}})J^*(x_{k+1})\} \\ &=\min_{u_k}\{U(x_k,u_k)+\frac{\underline{\delta}-1}{(1+\overline{\gamma}^{-1})^{l}}U(x_k,u_k)+(1+\frac{(\underline{\delta}-1)\overline{\gamma}}{(1+\overline{\gamma})(1+\overline{\gamma}^{-1})^{l-1}})J^*(x_{k+1})\} \\ &=\min_{u_k}\{U(x_k,u_k)+\frac{\underline{\delta}-1}{(1+\overline{\gamma}^{-1})^{l}}U(x_k,u_k)+(1+\frac{(\underline{\delta}-1)}{(1+\overline{\gamma}^{-1})^{l}})J^*(x_{k+1})\} \\ &=(1+\frac{(\underline{\delta}-1)}{(1+\overline{\gamma}^{-1})^{l}})\min_{u_k}\{U(x_k,u_k)++J^*(x_{k+1})\} \\ &=(1+\frac{(\underline{\delta}-1)}{(1+\overline{\gamma}^{-1})^{l}})J^*(x_k) \end{aligned} Vl​​=uk​min​{U(xk​,uk​)+Vl−1​(xk+1​)}≥uk​min​{U(xk​,uk​)+(1+(1+γ​−1)l−1δ​−1​)J∗(xk​)+(1+γ​)(1+γ​−1)l−1δ​−1​(γ​U(xk​,uk​)−J∗(xk+1​))}=uk​min​{U(xk​,uk​)+(1+γ​−1)lδ​−1​U(xk​,uk​)+(1+(1+γ​−1)l−1δ​−1​−(1+γ​)(1+γ​−1)l−1δ​−1​)J∗(xk+1​)}=uk​min​{U(xk​,uk​)+(1+γ​−1)lδ​−1​U(xk​,uk​)+(1+(1+γ​)(1+γ​−1)l−1(δ​−1)(1+γ​)−δ​+1​)J∗(xk+1​)}=uk​min​{U(xk​,uk​)+(1+γ​−1)lδ​−1​U(xk​,uk​)+(1+(1+γ​)(1+γ​−1)l−1(δ​−1)γ​​)J∗(xk+1​)}=uk​min​{U(xk​,uk​)+(1+γ​−1)lδ​−1​U(xk​,uk​)+(1+(1+γ​−1)l(δ​−1)​)J∗(xk+1​)}=(1+(1+γ​−1)l(δ​−1)​)uk​min​{U(xk​,uk​)++J∗(xk+1​)}=(1+(1+γ​−1)l(δ​−1)​)J∗(xk​)​

因此当$i=l$时成立，证闭。

第二步证明右边成立

一、

同理可得
V 1 ( x k ) = min ⁡ u k { U ( x k , u k ) + V 0 ( x k + 1 ) } ≤ min ⁡ u k { U ( x k , u k ) + δ ‾ J ∗ ( x k + 1 ) } ≥ min ⁡ u k { ( 1 + γ ‾ δ ‾ − 1 1 + γ ‾ ) U ( x k , u k ) + ( δ ‾ − δ ‾ − 1 1 + γ ‾ ) J ∗ ( x k + 1 ) }           ( 1 ) = ( 1 + γ ‾ δ ‾ − 1 1 + γ ‾ ) min ⁡ u k { U ( x k , u k ) + J ∗ ( x k + 1 ) }             ( 2 ) = ( 1 + γ ‾ δ ‾ − 1 1 + γ ‾ ) J ∗ ( x k ) \begin{aligned} V_1(x_k)&=\min_{u_k}\{U(x_k,u_k)+V_0(x_{k+1})\} \\ &\leq \min_{u_k}\{U(x_k,u_k)+\overline{\delta}J^*(x_{k+1})\} \\ &\geq \min_{u_k}\{(1+\underline{\gamma}\frac{\overline{\delta}-1}{1+\underline{\gamma}})U(x_k,u_k)+(\overline{\delta}-\frac{\overline{\delta}-1}{1+\underline{\gamma}})J^*(x_{k+1})\} ~~~~~~~~~(1)\\ &= (1+\underline{\gamma}\frac{\overline{\delta}-1}{1+\underline{\gamma}})\min_{u_k}\{U(x_k,u_k)+J^*(x_{k+1})\} ~~~~~~~~~~~(2) \\ &=(1+\underline{\gamma}\frac{\overline{\delta}-1}{1+\underline{\gamma}})J^*(x_k) \end{aligned} V1​(xk​)​=uk​min​{U(xk​,uk​)+V0​(xk+1​)}≤uk​min​{U(xk​,uk​)+δJ∗(xk+1​)}≥uk​min​{(1+γ​1+γ​δ−1​)U(xk​,uk​)+(δ−1+γ​δ−1​)J∗(xk+1​)}         (1)=(1+γ​1+γ​δ−1​)uk​min​{U(xk​,uk​)+J∗(xk+1​)}           (2)=(1+γ​1+γ​δ−1​)J∗(xk​)​
二、

假设结论对$i=l-1,l=1,2,\cdots$ 都成立，则 $i=l$ 可得
V l = min ⁡ u k { U ( x k , u k ) + V l − 1 ( x k + 1 ) } ≤ min ⁡ u k { U ( x k , u k ) + ( 1 + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l − 1 ) J ∗ ( x k ) + 1 − δ ‾ ( 1 + γ ‾ ) ( 1 + γ ‾ − 1 ) l − 1 ( J ∗ ( x k + 1 ) − γ ‾ U ( x k , u k ) ) } = min ⁡ u k { U ( x k , u k ) + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l U ( x k , u k ) + ( 1 + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l − 1 − δ ‾ − 1 ( 1 + γ ‾ ) ( 1 + γ ‾ − 1 ) l − 1 ) J ∗ ( x k + 1 ) } = min ⁡ u k { U ( x k , u k ) + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l U ( x k , u k ) + ( 1 + ( δ ‾ − 1 ) ( 1 + γ ‾ ) − δ ‾ + 1 ( 1 + γ ‾ ) ( 1 + γ ‾ − 1 ) l − 1 ) J ∗ ( x k + 1 ) } = min ⁡ u k { U ( x k , u k ) + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l U ( x k , u k ) + ( 1 + ( δ ‾ − 1 ) γ ‾ ( 1 + γ ‾ ) ( 1 + γ ‾ − 1 ) l − 1 ) J ∗ ( x k + 1 ) } = min ⁡ u k { U ( x k , u k ) + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l U ( x k , u k ) + ( 1 + ( δ ‾ − 1 ) ( 1 + γ ‾ − 1 ) l ) J ∗ ( x k + 1 ) } = ( 1 + ( δ ‾ − 1 ) ( 1 + γ ‾ − 1 ) l ) min ⁡ u k { U ( x k , u k ) + + J ∗ ( x k + 1 ) } = ( 1 + ( δ ‾ − 1 ) ( 1 + γ ‾ − 1 ) l ) J ∗ ( x k ) \begin{aligned} V_{l}&=\min_{u_k}\{U(x_k,u_k)+V_{l-1}(x_{k+1})\} \\ &\leq \min_{u_k}\{U(x_k,u_k)+(1+\frac{\overline{\delta}-1}{(1+\underline{\gamma}^{-1})^{l-1}})J^*(x_k)+\frac{1-\overline{\delta}}{(1+\underline{\gamma})(1+\underline{\gamma}^{-1})^{l-1}}(J^*(x_{k+1})-\underline{\gamma}U(x_k,u_k))\} \\ &=\min_{u_k}\{U(x_k,u_k)+\frac{\overline{\delta}-1}{(1+\underline{\gamma}^{-1})^{l}}U(x_k,u_k)+(1+\frac{\overline{\delta}-1}{(1+\underline{\gamma}^{-1})^{l-1}}-\frac{\overline{\delta}-1}{(1+\underline{\gamma})(1+\underline{\gamma}^{-1})^{l-1}})J^*(x_{k+1})\} \\ &=\min_{u_k}\{U(x_k,u_k)+\frac{\overline{\delta}-1}{(1+\underline{\gamma}^{-1})^{l}}U(x_k,u_k)+(1+\frac{(\overline{\delta}-1)(1+\underline{\gamma})-\overline{\delta}+1}{(1+\underline{\gamma})(1+\underline{\gamma}^{-1})^{l-1}})J^*(x_{k+1})\} \\ &=\min_{u_k}\{U(x_k,u_k)+\frac{\overline{\delta}-1}{(1+\underline{\gamma}^{-1})^{l}}U(x_k,u_k)+(1+\frac{(\overline{\delta}-1)\underline{\gamma}}{(1+\underline{\gamma})(1+\underline{\gamma}^{-1})^{l-1}})J^*(x_{k+1})\} \\ &=\min_{u_k}\{U(x_k,u_k)+\frac{\overline{\delta}-1}{(1+\underline{\gamma}^{-1})^{l}}U(x_k,u_k)+(1+\frac{(\overline{\delta}-1)}{(1+\underline{\gamma}^{-1})^{l}})J^*(x_{k+1})\} \\ &=(1+\frac{(\overline{\delta}-1)}{(1+\underline{\gamma}^{-1})^{l}})\min_{u_k}\{U(x_k,u_k)++J^*(x_{k+1})\} \\ &=(1+\frac{(\overline{\delta}-1)}{(1+\underline{\gamma}^{-1})^{l}})J^*(x_k) \end{aligned} Vl​​=uk​min​{U(xk​,uk​)+Vl−1​(xk+1​)}≤uk​min​{U(xk​,uk​)+(1+(1+γ​−1)l−1δ−1​)J∗(xk​)+(1+γ​)(1+γ​−1)l−11−δ​(J∗(xk+1​)−γ​U(xk​,uk​))}=uk​min​{U(xk​,uk​)+(1+γ​−1)lδ−1​U(xk​,uk​)+(1+(1+γ​−1)l−1δ−1​−(1+γ​)(1+γ​−1)l−1δ−1​)J∗(xk+1​)}=uk​min​{U(xk​,uk​)+(1+γ​−1)lδ−1​U(xk​,uk​)+(1+(1+γ​)(1+γ​−1)l−1(δ−1)(1+γ​)−δ+1​)J∗(xk+1​)}=uk​min​{U(xk​,uk​)+(1+γ​−1)lδ−1​U(xk​,uk​)+(1+(1+γ​)(1+γ​−1)l−1(δ−1)γ​​)J∗(xk+1​)}=uk​min​{U(xk​,uk​)+(1+γ​−1)lδ−1​U(xk​,uk​)+(1+(1+γ​−1)l(δ−1)​)J∗(xk+1​)}=(1+(1+γ​−1)l(δ−1)​)uk​min​{U(xk​,uk​)++J∗(xk+1​)}=(1+(1+γ​−1)l(δ−1)​)J∗(xk​)​
因此当$i=l$时成立，证闭。

这里限定了初值只能取到比最优代价函数小的初值。

定理二

重新定义
$$0\le \underline{\delta }\le 1\le \overline{\delta }\le \infty$$
证明下面的式子成立
$$(1+\frac{\underline{\delta }-1}{(1+{\overline{\gamma }}^{-1}{)}^i})J^{\ast }(x_k)\le V_i(x_k)\le (1+\frac{\overline{\delta }-1}{(1+{\overline{\gamma }}^{-1}{)}^i})J^{\ast }(x_k)$$

证明

左边的证明跟上面是一样，只需要证明右边的即可。
V 1 ( x k , u k ) = min ⁡ u k { U ( x k , u k ) + V 0 ( x k + 1 ) } ≤ min ⁡ u k { U ( x k , u k ) + δ ‾ J ∗ ( x k + 1 ) + δ ‾ − 1 1 + γ ‾ ( γ ‾ U ( x k , u k ) − J ∗ ( x k + 1 ) ) } = min ⁡ u k { ( 1 + δ ‾ − 1 1 + γ ‾ − 1 ) U ( x k , u k ) + ( 1 + δ ‾ − 1 1 + γ ‾ − 1 ) J ∗ ( x k ) } = ( 1 + δ ‾ − 1 1 + γ ‾ − 1 ) J ∗ ( x k ) \begin{aligned} V_1(x_k,u_k)&=\min_{u_k}\{U(x_k,u_k)+V_0(x_{k+1})\} \\ &\leq\min_{u_k}\{U(x_k,u_k)+\overline{\delta}J^*(x_{k+1})+\frac{\overline{\delta}-1}{1+\overline{\gamma}}(\overline{\gamma}U(x_k,u_k)-J^*(x_{k+1}))\} \\ &=\min_{u_k}\{(1+\frac{\overline{\delta}-1}{1+\overline{\gamma}^{-1}})U(x_k,u_k)+(1+\frac{\overline{\delta}-1}{1+\overline{\gamma}^{-1}})J^*(x_k)\} \\ &=(1+\frac{\overline{\delta}-1}{1+\overline{\gamma}^{-1}})J^*(x_k) \\ \end{aligned} V1​(xk​,uk​)​=uk​min​{U(xk​,uk​)+V0​(xk+1​)}≤uk​min​{U(xk​,uk​)+δJ∗(xk+1​)+1+γ​δ−1​(γ​U(xk​,uk​)−J∗(xk+1​))}=uk​min​{(1+1+γ​−1δ−1​)U(xk​,uk​)+(1+1+γ​−1δ−1​)J∗(xk​)}=(1+1+γ​−1δ−1​)J∗(xk​)​
假设结论对$i=l-1,l=1,2,\cdots$ 都成立，则 $i=l$ 可得
V l ( x k ) = min ⁡ u k { U ( x k , u k ) + V l − 1 ( x k + 1 ) } ≤ min ⁡ u k { U ( x k , u k ) + ( 1 + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l − 1 ) J ∗ ( x k + 1 ) + δ ‾ − 1 ( 1 + γ ‾ ) ( 1 + γ ‾ − 1 ) l − 1 ( γ ‾ U ( x k , u k ) − J ∗ ( x k + 1 ) ) } = ( 1 + δ ‾ − 1 ( 1 + γ ‾ − 1 ) l ) J ∗ ( x k ) \begin{aligned} V_{l}(x_k)&=\min_{u_k}\{U(x_k,u_k)+V_{l-1}(x_{k+1})\} \\ &\leq \min_{u_k}\{U(x_k,u_k)+(1+\frac{\overline{\delta}-1}{(1+\overline{\gamma}^{-1})^{l-1}})J^*(x_{k+1})+\frac{\overline{\delta}-1}{(1+\overline{\gamma})(1+\overline{\gamma}^{-1})^{l-1}}(\overline{\gamma}U(x_k,u_k)-J^*(x_{k+1}))\} \\ &=(1+\frac{\overline{\delta}-1}{(1+\overline{\gamma}^{-1})^{l}})J^*(x_k) \end{aligned} Vl​(xk​)​=uk​min​{U(xk​,uk​)+Vl−1​(xk+1​)}≤uk​min​{U(xk​,uk​)+(1+(1+γ​−1)l−1δ−1​)J∗(xk+1​)+(1+γ​)(1+γ​−1)l−1δ−1​(γ​U(xk​,uk​)−J∗(xk+1​))}=(1+(1+γ​−1)lδ−1​)J∗(xk​)​
因此当$i=l$时成立，证闭。

定理三

重新定义
$$1\le \underline{\delta }\le \overline{\delta }\le \infty$$
满足
$$(1+\frac{\underline{\delta }-1}{(1+{\underline{\gamma }}^{-1}{)}^i})J^{\ast }(x_k)\le V_i(x_k)\le (1+\frac{\overline{\delta }-1}{(1+{\overline{\gamma }}^{-1}{)}^i})J^{\ast }(x_k)$$
跟定理一的证明是一样的。

定理四

重新定义
$$0\le \underline{\delta }\le \overline{\delta }\le \infty$$
使下式成立
$$\underset{i\to \infty }{\lim }{V}_i(x_k)=J^{\ast }(x_k)$$
证明：

左边的不等式
lim ⁡ i → ∞ { ( 1 + δ ‾ − 1 ( 1 + γ ‾ − 1 ) i ) J ∗ ( x k ) } = lim ⁡ i → ∞ { ( 1 + δ ‾ − 1 ( 1 + γ ‾ − 1 ) i ) J ∗ ( x k ) } = J ∗ ( x k ) \lim_{i\rightarrow\infty}\{(1+\frac{\underline{\delta}-1}{(1+\overline{\gamma}^{-1})^{i}})J^*(x_k)\}=\lim_{i\rightarrow\infty}\{(1+\frac{\underline{\delta}-1}{(1+\underline{\gamma}^{-1})^{i}})J^*(x_k)\}=J^*(x_k) i→∞lim​{(1+(1+γ​−1)iδ​−1​)J∗(xk​)}=i→∞lim​{(1+(1+γ​−1)iδ​−1​)J∗(xk​)}=J∗(xk​)
右边不等式
lim ⁡ i → ∞ { ( 1 + δ ‾ − 1 ( 1 + γ ‾ − 1 ) i ) J ∗ ( x k ) } = lim ⁡ i → ∞ { ( 1 + δ ‾ − 1 ( 1 + γ ‾ − 1 ) i ) J ∗ ( x k ) } = J ∗ ( x k ) \lim_{i\rightarrow\infty}\{(1+\frac{\overline{\delta}-1}{(1+\underline{\gamma}^{-1})^{i}})J^*(x_k)\}=\lim_{i\rightarrow\infty}\{(1+\frac{\overline{\delta}-1}{(1+\overline{\gamma}^{-1})^{i}})J^*(x_k)\}=J^*(x_k) i→∞lim​{(1+(1+γ​−1)iδ−1​)J∗(xk​)}=i→∞lim​{(1+(1+γ​−1)iδ−1​)J∗(xk​)}=J∗(xk​)
上面的证明可以得出，收敛条件和初值是无关的，因此没有必要获取变量 $\underline{\gamma },\overline{\gamma },\underline{\delta }$ 和$\overline{\delta }$ 变量。

定理五

如果
$$V_1(x_k)\le V_0(x_k)$$
则
$$V_{i+1}(x_k)\le V_i(x_k)$$

证明

用数学归纳法进行证明：

令$i=1$，可得
V 2 ( x k ) = min ⁡ u k { U ( x k , u k ) + V 1 ( x k + 1 ) } ≤ min ⁡ u k { U ( x k , u k ) + V 0 ( x k + 1 ) } = V 1 ( x k ) \begin{aligned} V_2(x_k)&=\min_{u_k}\{U(x_k,u_k)+V_1(x_{k+1})\} \\ &\leq \min_{u_k}\{U(x_k,u_k)+V_0(x_{k+1})\} \\ &=V_1(x_k) \end{aligned} V2​(xk​)​=uk​min​{U(xk​,uk​)+V1​(xk+1​)}≤uk​min​{U(xk​,uk​)+V0​(xk+1​)}=V1​(xk​)​
假设结论对$i=l-1,l=1,2,\cdots$ 都成立，则 $i=l$ 可得
V l + 1 ( x k ) = min ⁡ u k { U ( x k , u k ) + V l ( x k + 1 ) } ≤ min ⁡ u k { U ( x k , u k ) + V l − 1 ( x k + 1 ) } = V l ( x k ) \begin{aligned} V_{l+1}(x_k)&=\min_{u_k}\{U(x_k,u_k)+V_l(x_{k+1})\} \\ &\leq \min_{u_k}\{U(x_k,u_k)+V_{l-1}(x_{k+1})\} \\ &=V_l(x_k) \end{aligned} Vl+1​(xk​)​=uk​min​{U(xk​,uk​)+Vl​(xk+1​)}≤uk​min​{U(xk​,uk​)+Vl−1​(xk+1​)}=Vl​(xk​)​

定理六

如果
$$V_1(x_k)\ge V_0(x_k)$$
则
$$V_{i+1}(x_k)\ge V_i(x_k)$$

证明

用数学归纳法进行证明：

令$i=1$，可得
V 2 ( x k ) = min ⁡ u k { U ( x k , u k ) + V 1 ( x k + 1 ) } ≥ min ⁡ u k { U ( x k , u k ) + V 0 ( x k + 1 ) } = V 1 ( x k ) \begin{aligned} V_2(x_k)&=\min_{u_k}\{U(x_k,u_k)+V_1(x_{k+1})\} \\ &\geq \min_{u_k}\{U(x_k,u_k)+V_0(x_{k+1})\} \\ &=V_1(x_k) \end{aligned} V2​(xk​)​=uk​min​{U(xk​,uk​)+V1​(xk+1​)}≥uk​min​{U(xk​,uk​)+V0​(xk+1​)}=V1​(xk​)​
假设结论对$i=l-1,l=1,2,\cdots$ 都成立，则 $i=l$ 可得
V l + 1 ( x k ) = min ⁡ u k { U ( x k , u k ) + V l ( x k + 1 ) } ≥ min ⁡ u k { U ( x k , u k ) + V l − 1 ( x k + 1 ) } = V l ( x k ) \begin{aligned} V_{l+1}(x_k)&=\min_{u_k}\{U(x_k,u_k)+V_l(x_{k+1})\} \\ &\geq \min_{u_k}\{U(x_k,u_k)+V_{l-1}(x_{k+1})\} \\ &=V_l(x_k) \end{aligned} Vl+1​(xk​)​=uk​min​{U(xk​,uk​)+Vl​(xk+1​)}≥uk​min​{U(xk​,uk​)+Vl−1​(xk+1​)}=Vl​(xk​)​

推论

如果$V_1(x_k)\le V_0(x_k)$，则$V_i(x_k)\ge J^{\ast }(x_k)$

由定理五可知
$$V_i(x_k)\ge V_{i+1}(x_k)\ge V_{i+2}(x_k)\ge \cdots \ge$$
对于$l\ge i$
$$V_i(x_k)\ge V_l(x_K)$$
可得
$$V_i(x_k)\ge \underset{l\to \infty }{\lim }{V}_l(x_k)=J^{\ast }(x_k)$$
同理可得果$V_1(x_k)\ge V_0(x_k)$，则$V_i(x_k)\le J^{\ast }(x_k)$

上述推论，反过来是不成立的。知道初值的大小，是不能决定单调性的。

当满足一些条件也是成立的。

定理七

初值是一个半正定函数$\Psi (x_k)$，$\overline{v}(x_k)$是容许控制律，则值函数是不增的函数，$\Psi (x_k)$满足
$$\Psi (x_k)=U(x_k,\overline{v}(x_k))+\Psi (x_{k+1})$$
证明：
V 1 ( x k ) = U ( x k , v 0 ( x k ) ) + V 0 ( x k + 1 ) = min ⁡ u k { U ( x k , u k ) + Ψ ( x k + 1 } ≤ U ( x k , v ‾ ( x k ) ) + Ψ ( x k + 1 ) = Ψ ( x k ) \begin{aligned} V_1(x_k)&=U(x_k,v_0(x_k))+V_0(x_{k+1}) \\ &=\min_{u_k}\{U(x_k,u_k)+\Psi(x_{k+1}\} \\ &\leq U(x_k,\overline{v}(x_k))+\Psi(x_{k+1}) \\ &=\Psi(x_k) \end{aligned} V1​(xk​)​=U(xk​,v0​(xk​))+V0​(xk+1​)=uk​min​{U(xk​,uk​)+Ψ(xk+1​}≤U(xk​,v(xk​))+Ψ(xk+1​)=Ψ(xk​)​
通过数学归纳法，就可以证明值函数是单调不增的。

通过上面的证明可以看出，如果任意定义一个初始值函数，不满足迭代公式时，$U(x_k,\overline{v}(x_k))+\Psi (x_{k+1})$ 的值的大小是未知的，因此比较公式会变成$V_1(x_k) $ 和 $U(x_k,\overline{v}(x_k))+\Psi (x_{k+1})$ 进行比较，这才是真正能和$J^{\ast }(x_k)$进行比较的量，决定当前的单调性的地方。从公式中看出，此时不仅与初值有关，而且和当前控制网络的控制律有关。其实，也可以看做，当前的值迭代问题，又变成了策略迭代了。

如果此时的初始值函数是恒为零的函数，$V_1(x_k)$的值就是效用函数，肯定大于零，大于当前的初始值函数。

定理八

终止条件是
$$|V_{i+1}(x_k)-V_i(x_k)|\le \epsilon$$
控制律$v_i(x_k)$是最终一致有界（UUB）。

定义：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-e576hTLL-1604646023919)(C:\Users\Wandering\AppData\Roaming\Typora\typora-user-images\image-20201023152548119.png)]

证明

首先当初始值函数$\Psi (x_k)$ 是半正定的，则 $V_i(x_k)$ 就是正定的。
$$\begin{gathered} |V_{i+1}(x_k)-V_i(x_k)|\le \epsilon \to |U(x_k,v_i(x_k)+V_i(x_{k+1}))-V_i(x_k)|\le \epsilon \\ -U(x_k,v_i(x_k))-\epsilon \le \Delta V_i(x_k)=V_i(x_{k+1})-V_i(x_k)\le -U(x_k,v_i(x_k))+\epsilon \end{gathered}$$
我们可以很容易的证明当$-U(x_k,v_i(x_k))-\epsilon \le \Delta V_i(x_k)$， $V_i(x_k)$是一个李雅普诺夫函数，此时系统是渐进稳定的。只需要分析$0\le \Delta V_i(x_k)\le -U(x_k,v_i(x_k))+\epsilon$ 这一种情形。

由于$V_i(x_k)$ 是正定的，因此一定存在$\alpha (||x_k||)$ 和$\beta (||x_k||)$ 满足下面的式子
$$0<\alpha (||x_k||)\le V_i(x_k)\le \beta (||x_k||)$$
定义一个新的状态空间
Ω x k = { x k ∣ x k ∈ R n  and   U ( x k , v i ( x k ) ) ≤ ε } \Omega_{x_k}=\{x_k|x_k\in R^n~\text{and}~~U(x_k,v_i(x_k))\leq\varepsilon\} Ωxk​​={xk​∣xk​∈Rn and  U(xk​,vi​(xk​))≤ε}
由于$U(x_k,v_i(x_k))$ 是正定函数，因此$||x_k||$ 是有限的，$||x_k||$ 是欧几里得范式。定义
ϱ = sup ⁡ x k ∈ Ω x k { ∣ ∣ x k ∣ ∣ } \varrho=\sup_{x_k\in \Omega_{x_k}}\{||x_k||\} ϱ=xk​∈Ωxk​​sup​{∣∣xk​∣∣}
由于$\epsilon$是有限的，$\varrho$ 是有限的，对于任意的$\varrho$ 满足上式，总存在一个有限的 $\Gamma$，$||\Gamma ||\ge ||\varrho ||$ ，满足
$$\alpha (||\Gamma ||)\ge \beta (||\varrho ||)$$
$ϵ$ 满足$ϵ\ge ||\Gamma ||$，存在 $\delta (ϵ)$，使 $\delta (ϵ)\ge ||\varrho ||$ 成立，满足$\beta (\delta )\le \alpha (ϵ)$。因此存在状态$x_k$，$||\varrho ||\le ||x_k||\le \delta (ϵ)$，满足
$$\alpha (ϵ)\ge \beta (\delta )\ge V_i(x_k)$$
当$||x_k||\ge ||\varrho ||$，可得
$$V_i(x_{k+1})-V_i(x_k)\le 0$$
因此对于任意 $x_k$ 满足 $||\varrho ||\le ||x_k||\le \delta (ϵ)$ ,总存在一个 $T>0$ 满足
$$\alpha (ϵ)\ge \beta (\delta )\ge V_i(x_k)\ge V_i(x_{k+T}\ge \alpha (||x_{k+T}||))$$
可以得到$ϵ>||x_{k+T}||$。因此对于任意的 $x_k$，满足$||x_k||\ge ||\varrho ||$ 存在$T=1,2,\cdots$，使$||x_{k+T}||\le ||\varrho ||$ 成立。当$||\Gamma ||\ge ||\varrho ||$，我们可以获得$||x_{k+T}||\le ||\Gamma ||$,因此定理得证。

这里$0\le \Delta V_i(x_k)\le -U(x_k,v_i(x_k))+\epsilon$ 可以看出$U(x_k,v_i(x_k))\le \epsilon$ ，对效用函数进行了限制，当不满足这个不等式时，系统是渐进稳定的，因此系统在这个条件下状态向量有界。

定理九

当满足条件
$$V_{i+1}(x_k)-V_i(x_k)<U(x_k,v_i(x_k))$$
此时的控制律$v_i(x_k)$是容许控制律。

存在$-\infty <\theta <1$
$$\begin{gathered} V_{i+1}(x_k)-V_i(x_k)<\theta U(x_k,v_i(x_k)) \\ U(x_k,v_i(x_k))+V_i(x_{k+1})-V_i(x_k)<\theta U(x_k,v_i(x_k)) \\ {V}_i(x_{k+1})-V_i(x_k)<(\theta -1)U(x_k,v_i(x_k)) \end{gathered}$$
因此 $V_i(x_{k+1})-V_i(x_k)<0$，由李雅普诺夫稳定判据可知，系统是稳定的，故当前的控制是稳定的控制。
$$\{\begin{array}{cll}{V}_i(x_{k+1})-V_i(x_k)& <& (\theta -1)U(x_k,v_i(x_k))\\ V_i(x_{k+2})-V_i(x_{k+1})& <& (\theta -1)U(x_{k+1},v_i(x_{k+1}))\\ V_i(x_{k+3})-V_i(x_{k+2})& <& (\theta -1)U(x_{k+2},v_i(x_{k+2}))\\ \cdot & & \\ \cdot & & \\ \cdot & & \\ V_i(x_{k+N})-V_i(x_{k+N-1})& <& (\theta -1)U(x_{k+N-1},v_i(x_{k+N-1}))\end{array}$$
可得
$$V_i(x_{k+N})-V_i(x_k)<(\theta -1)\sum _{j=0}^{N}U(x_{k+j},v_i(x_{k+j}))$$
当$v_i(x_k)$是稳定的控制，$N\to \infty$时，$x_N\to 0,V_i(x_{k+N})\to 0$，因此可得
$$V_i(x_k)>(1-\theta )\sum _{j=0}^{N}U(x_{k+j},v_i(x_{k+j}))$$
对于有限的状态$x_k$，值函数$V_i(x_k)$都是有限的，因此可以知${\sum }_{j=0}^{N}U(x_{k+j},v_i(x_{k+j}))$ 是有限的，则控制律 $v_i(x_k)$ 是容许控制律。

定理十

存在一个有限的常数$N>0$ 满足
$$V_{N+1}(x_k)-V_N(x_k)<U(x_k,v_N(x_k))$$
证明

利用反证法进行证明，对于$N=0,1,\cdots$，任意${\overline{x}}_k\in R^n$，满足
$$V_{N+1}({\overline{x}}_k)-V_N({\overline{x}}_k)\ge U({\overline{x}}_k,v_N({\overline{x}}_k))$$
当 $N\to \infty$，根据定理四我们可以得到${\lim }_{N\to }(V_{N+1}({\overline{x}}_k)-V_N({\overline{x}}_k))=0$ ，根据上式就可以得出
$$\underset{N\to }{\lim }(V_{N+1}({\overline{x}}_k)-V_N({\overline{x}}_k))=U({\overline{x}}_k,v_N({\overline{x}}_k))=0$$
对任意${\overline{x}}_k\in R^n$ 都成立。这与效用函数$U(x_k,u_k)$是正定的相矛盾。所以定理得证。

定理十一

1. $V_{i+1}(x_k)+V_{i+j}(x_k)\ge V_i(x_k)+V_{i+j+1}(x_k)$
2. $V_j(x_k)\ge \frac{1}{2}(V_{j+1}(x_k)+V_{j-1}(x_k))$
3. $\Delta V_j(x_k)=V_j(X_K)-V_{j-1}(x_k)$，对于所有的$j>i$，$\Delta V_j(x_k)\ge \Delta V_{j+1}(x_k)$

则$v_j(x_k)$ 是容许的控制。

值函数满足 $V_{i+1}(x_k)-V_i(x_k)<U(x_k,v_i(x_k))$

证明
$$(V_{i+j+1}(x_k)-V_{i+j}(x_k))-(V_{i+1}(x_k)-V_i(x_k))\le 0$$
可得
$$\begin{gathered} (V_{i+j+1}(x_k)-U(x_k,v_{i+j}(x_k)))-V_{i+j}(x_k)\le V_{i+1}(x_k)-U(x_k,v_i(x_k))-V_i(x_k)+U(x_k,v_i(x_k)) \\ {V}_{i+j+1}(x_{k+1})-V_{i+j}(x_k)\le V_i(x_{k+1})-V_i(x_k)+U(x_k,v_i(x_k))<0 \end{gathered}$$
由定理九可知，此控制是容许的控制。

后面这几条定理，都是为了得到$V_i(x_{k+1})-V_i(x_k)<0$的结论，当这个条件满足了，就是李雅普诺夫函数满足了，因此此时肯定是一个稳定的控制。