[First order method] Gradient descent tools

1. Lipschitz gradient

If $\nabla f(x)$ L-is lipschitz continuous, then we have

$$\frac{1}{2L}\|\nabla f(x)\|_2^2\leq f(x) - f(x^*)\leq \frac{L}{2}\|x-x^*\|_2^2$$

The right hand is because that

$$f(x)\leq f(x^*)+\nabla f(x^*)\cdot (x-x^*)+\frac{L}{2}\|x-x^*\|_2^2 =\frac{L}{2}\|x-x^*\|_2^2$$
The left hand is because that
$$\begin{align} f(x^*)\leq f(y)\leq &\inf_y f(x)+\nabla f(x)\cdot (y-x)+\frac{L}{2}\|y-x\|_2^2 \\ &= f(x) - \frac{1}{2L}\|\nabla f(x)\|_2^2 \end{align}$$

This tells that function with lipschitz derivative is upper bounded by a quadratic function.

2. Strong continuous

If $f(x)$ is m-strong continuous, then we have

$$\frac{m}{2}\|x-x^*\|_2^2\leq f(x)-f(x^*)\leq \frac{1}{2m}\|\nabla f(x)\|_2^2$$

The left hand is because that

$$f(x)\geq f(x^*)+\nabla f(x^*)\cdot (x-x^*)+\frac{m}{2}\|x-x^*\|_2= \frac{m}{2}\|x-x^*\|_2$$

The left hand is because that $\forall y$

$$\begin{align} f(y)&\geq \inf_y f(x) + \nabla f(x)\cdot (y-x)+\frac{m}{2}\|x-y\|_2^2\\ &=f(x)+\frac{1}{2m}\|\nabla f(x)\|_2^2 \end{align}$$
so
$$f(x^*)\geq f(x)+ \frac{1}{2m}\|\nabla f(x)\|_2^2$$

This tells that function with strong convexity is lower bounded by a quadratic function.

3. Co-coercivity of gradient

3.1 Lipschitz of gradient

If $f(x)$ is convex and $\frac{L}{2} \|x\|_2^2 - f(x)$ is convex($\nabla f(x)$ is lipschitz continuous), then we have

$$0\leq(\nabla f(x) - \nabla f(y))\cdot (x-y)\leq L\|x-y\|_2^2$$

which can be rewritten as

$$((Lx-\nabla f(x)) - (Ly -\nabla f(y)))\cdot (x-y)\geq 0$$

which says that

$$g(z)=\frac{L}{2}\|z\|_2^2 - f(z)$$
with increasing derivative $Lz - \nabla f(z)$, is convex. So both of
$$\begin{align} g(z)+f(x) = \frac{L}{2}\|z\|_2^2 -\underbrace{ f(z) +\nabla f(x)\cdot z}_{\text{$f_x(z)$}}\\ g(z)+f(y)= \frac{L}{2}\|z\|_2^2 -\underbrace{ f(z) +\nabla f(y)\cdot z}_{\text{$f_y(z)$}} \end{align}$$
are convex, then both of $f_x(z)$ and $f_y(z)$ are L-lipschitz. So
$$\begin{align} f(y) - f(x) -\nabla f(x)\cdot(y-x) &=(f(y) + \nabla f(x)\cdot y) - (f(x) - \nabla f(x)\cdot x)\\ &=f_x(y)-f_x(x)\\ &\geq \frac{1}{2L}\|\nabla f_x(y)\|_2^2\\ &=\frac{1}{2L}\|\nabla f(y) - \nabla f(x)\|_2^2 \end{align}$$
the same
$$f(x) - f(y) -\nabla f(y)\cdot(x-y) \geq\frac{1}{2L}\|\nabla f(y) - \nabla f(x)\|_2^2$$

combining these two, we get the co-coercivity of L-lipschitz gradient is

$$(\nabla f(x) - \nabla f(y))\cdot (x-y) \geq \frac{1}{L}\|\nabla f(y) - \nabla f(x)\|_2^2$$

3.2 Strong convex

If $f(x)$ is m-strong convex, then we have

$$h(x) = f(x) - \frac{m}{2} \|x\|_2^2$$ is convex.

And from theorem in blog
http://blog.csdn.net/comeyan/article/details/50541596#2-strong-convex
there exists a $M$ such that $\nabla f(x)$ is M-lipschitz. So we have

$$\begin{align} \|\nabla h(x) - \nabla h(y)\|_2 &= \|\nabla f(x) - mx - \nabla f(y)+m(y)\|_2\\ &\leq (M+m)\|x-y\|_2 \end{align}$$

So $\nabla h(x)$ is M+m lipschitz continuous. From last subsection, we know that

$$\begin{align} &(\nabla h(x) - \nabla h(y))\cdot (x-y) \geq \frac{1}{M+m}\|\nabla h(y) - \nabla h(x)\|_2^2\\ &\Rightarrow (\nabla f(x) - mx - \nabla f(z) +my)\cdot(x-y)\geq \frac{1}{M+m}\|\nabla f(x) - mx - \nabla f(z) +my\|_2^2\\ &\Rightarrow (\nabla f(x) - \nabla f(y))\cdot (x-y)\geq \frac{1}{M+m}\|\nabla f(x) - \nabla f(y)\|_2^2+\frac{mM}{M+m}\|x-y\|_2^2 \end{align}$$