Optimization inequalities cheatsheet

This article is from http://fa.bianp.net/blog/2017/optimization-inequalities-cheatsheet/, just record it.

Most proofs in optimization consist in using inequalities for a particular function class in some creative way. This is a cheatsheet with inequalities that I use most often. It considers class of functions that are convex, strongly convex and $L$-smooth.
Setting. $f$ is a function $\mathbb{R}^p \to \mathbb{R}$. Below are a set of inequalities that are verified when $f$ belongs to a particular class of functions and $x, y \in \mathbb{R}^p$ are arbitrary elements in its domain.
$f$ is $L$-smooth. This is the class of functions that are differentiable and its gradient is Lipschitz continuous.

1. $\|\nabla f(y) - \nabla f(x) \| \leq \|x - y\|$
2. $|f(x) - f(y) - \langle \nabla f(x), y - x\rangle| \leq \frac{L}{2}\|y - x\|^2$
3. $\|\nabla^2 f(x)\| \leq L\qquad \text{ (assuming $f$ is twice differentiable)}$

$f$ is convex.

1. $f(x) \leq f(y) + \langle \nabla f(x), x - y\rangle$
   • $0 \leq \langle \nabla f(x) - \nabla f(y), x - y\rangle$
   • $f(\mathbb{E}X) \leq \mathbb{E}[f(X)]$ where $X$ is a random variable (Jensen’s inequality).

$f$ is both $L$-smooth and convex:

1. $\frac{1}{L}\|\nabla f(x) - \nabla f(y)\|^2 \leq \langle \nabla f(x) - \nabla f(y), x - y\rangle$
2. $0 \leq f(y) - f(x) - \langle \nabla f(x), y - x\rangle \leq \frac{L}{2}\|x - y\|^2$
3. $f(x) \leq f(y) + \langle \nabla f(x), x - y\rangle - \frac{1}{2 L}\|\nabla f(x) - \nabla f(y)\|^2$

4. $f$ is $\mu$-strongly convex. Set of functions $f$ such that $f - \frac{\mu}{2}\|\cdot\|^2$ is convex. It includes the set of convex functions with $\mu=0$.

   1. $f(x) \leq f(y) + \langle \nabla f(x), x - y \rangle - \frac{\mu}{2}\|x - y\|^2$
   2. $\frac{\mu}{2}\|x - y\|^2 \leq \langle \nabla f(x) - \nabla f(y), x - y\rangle$
   3. $\frac{\mu}{2}\|x-x^*\|^2\leq f(x^*) - f(x)$

   $f$ is both $L$-smooth and $\mu$-strongly convex.

   $\frac{\mu L}{\mu + L}\|x - y\|^2 + \frac{1}{\mu + L}\|\nabla f(x) - \nabla f(y)\|^2 \leq \langle \nabla f(x) - \nabla f(y), x - y\rangle$

   Most of these inequalities appear in the Book: “Introductory lectures on convex optimization: A basic course” by Nesterov, Yurii (2013, Springer Science & Business Media). Another good source (and freely available for download) is the book “Convex Optimization” by
   Stephen Boyd and Lieven Vandenberghe.