凸优化基础知识-强凸性&Hessian矩阵的上下限

---

9.1 Unconstrained minimization problems

In this chapter, we discuss methods for solving the unconstrained optimization problem
$$P9.1:f(x)$$ where $f:R_n\to R$ is convex and twice continuously differentiable (which implies that domf is open). We denote the optimal value, ${\inf }_{x}f(x)=f(x^{\star })$, as $p^{\star }$.

Since $f$ is differentiable and convex, a necessary and sufficient condition for a point $x^{\star }$ to be optimal is
$$P9.2:\nabla f(x^{\ast })=0.$$ Thus, solving the unconstrained minimization problem $(9.1)$ is the same as finding a solution of $(9.2)$, which is a set of n equations in the n variables $x_1,...,x_n$. In a few special cases, we can find a solution to the problem $(9.1)$ by
analytically solving the optimality equation $(9.2)$, but usually the problem must be solved by an iterative algorithm. By this, we mean an algorithm that computes a sequence of points $x^{(0)},x^{(1)},...\in \mathbf{d}\mathbf{o}\mathbf{m}f$ with $f(x(k))\to p^{\star }$ as $k\to \infty$.

---

9.1.2 Strong Convexity and implications

Lower bound on ${\nabla }^{2}f(x)$ (Hessian Matrix)

We assume that the obejctive function is strongly convex on $\mathcal{S}$, which means there exists an $m>0$ such that $$P9.7:{\nabla }^{2}f(x)⪰m\mathbf{I}$$ for all $x\in \mathcal{S}$.

Strong convexity has some interesting consequences. For $x,y\in \mathcal{S}$, we have $$f(y)=f(x)+\nabla f(x{)}^T(y-x)+\frac{1}{2}(y-x{)}^T{\nabla }^{2}f(z)(y-z),$$ for some $z$ on the line segment $[x,y]$.

By the strong convexity assumption $(9.7)$, the last term on the righthand side is at least $\frac{m}{2}\Vert y-x{\Vert }_2^2$, so we have the inequality $$P9.8:f(y)\ge f(x)+\nabla f(x{)}^T(y-x)+\frac{m}{2}\Vert y-x{\Vert }_2^2$$ for all $x$ and $y$ in $\mathcal{S}$. When $m=0$, we recover the basic inequality characterizing convexity; for $m>0$, we obtain a better lower bound on $f(y)$ than follows from convexity alone.

Then, we will show the inequality (9.8) can be used to bound $f(x)-p^{\ast }$, which is the suboptimality of the point $x$, in terms of $\Vert \nabla f(x){\Vert }_2$. the righthand side of (9.8) is a convex quadratic function of $y$ (for fixed x). Setting the gradient with respect to $y$ equal to zeros, we find that $\tilde{y}=x-\frac{1}{m}\nabla f(x)$ minimizes the righthand side. Therefore we have $$\begin{array}{rl}f(y)& \ge f(x)+\nabla f(x{)}^T(y-x)+\frac{1}{m}\Vert y-x{\Vert }_2^2\\ & \ge f(x)+\nabla f(x{)}^T(\tilde{y}-x)+\frac{1}{m}\Vert \tilde{y}-x{\Vert }_2^2\\ & =f(x)-\frac{1}{2m}\Vert \nabla f(x){\Vert }_2^2\end{array}$$ Since this holds fpr any $y\in S$, we have
$$P9.9:p^{\ast }\ge f(x)-\frac{1}{2m}\Vert \nabla f(x){\Vert }_2^2,$$ which can be rewritten as
$$\Vert f(x)-p^{\ast }{\Vert }_2\le \frac{1}{2m}\Vert \nabla f(x){\Vert }_2^2.$$
We can also derive a bound on $\Vert x-x^{\ast }{\Vert }_2,$ the distance between $x$ and any optimal point $x^{\ast }$, in terms of $\Vert \nabla f(x){\Vert }_2:$
$$P9.11:\Vert x-x^{\ast }\Vert \le \frac{2}{m}\Vert \nabla f(x){\Vert }_2.$$

To see this, we apply (9.8) with $y=x^{\ast }$ to obtain
$$\begin{array}{rl}{p}^{\ast }=f(x^{\ast })& \ge f(x)+\nabla f(x{)}^T(x^{\ast }-x)+\frac{m}{2}\Vert x^{\ast }-x{\Vert }_2^2\\ & \ge f(x)+\Vert \nabla f(x){\Vert }_2\Vert x^{\ast }-x{\Vert }_2+\frac{m}{2}\Vert x^{\ast }-x{\Vert }_2^2,\end{array}$$ where we use the Cauchy-Schwarz inequality in the second inequality, $<x^{\ast },x>+\Vert x^{\ast }-x\Vert \ge 0$. Since $p^{\ast }\le f(x)$, we must have
$$-\Vert \nabla f(x){\Vert }_2\Vert x^{\ast }-x{\Vert }_2+\frac{m}{2}\Vert x^{\ast }-x{\Vert }_2^2\le 0,$$ from which $(9.11)$ follows.

Upper bound on ${\nabla }^{2}f(x)$ (Hessian Matrix)

The inequality (9.8) implies that the sublevel sets contained in $S$ are bounded, so in particular, $S$ is bounded. Therefore the maximum eigenvalue of ${\nabla }^{2}f(x)$, which is a continuous function of $x$ on $S$, is bounded above on $S$, i.e., there exists a constant $M$ such that $${\nabla }^{2}f(x)⪯M\mathbf{I}$$ for all $x\in \mathcal{S}$. This upper bound on the Hessian implies for any $x,y\in \mathcal{S},$
$$f(y)\le f(x)+\nabla f(x{)}^T(y-x)+\frac{M}{2}\Vert y-x{\Vert }_2^2,$$ which is analogous to $(9.8)$. Minimizing each side over $y$ yieldes
$$p^{\ast }\le f(x)-\frac{1}{2M}\Vert \nabla f(x){\Vert }_2^2$$ which can be rewritten as $$\Vert f(x)-p^{\ast }\Vert \ge \frac{1}{2M}\Vert \nabla f(x){\Vert }_2^2,$$ the counterpart to $(9.9).$

---