Optimization Week 2: Convex Functions

1 Definition of convex function

1.1 Basic definition

A function $f:{\mathbb{R}}^d\to \mathbb{R}$ is convex if for $\forall$ $x_1,x_2$ and $\forall$ $\lambda \in [0,1]$:
$$f(\lambda x_1+(1-\lambda )x_2)\le \lambda f(x_1)+(1-\lambda )f(x_2)$$

1.2 Definition of differentiable function

• If $f$ is differentiable, $f$ is convex $\iff$ $\forall$ $x$ and $y$, $f(y)\ge f(x)+\nabla f(x{)}^T(y-x)$
• If $f$ is twice differentiable, $f$ is convex $\iff$ $\forall$ $x$, ${\nabla }^{2}f(x)\ge 0$

1.3 Definition with monotone gradient

• If $f$ is differentiable, $f$ is convex $\iff$ $\forall$ $x$ and $y$, $(\nabla f(x)-\nabla f(y))(x-y)\ge 0$

2 Operations preserving convexity

2.1 Non-negative sum

$$f=a_1{f}_1(x)+a_2{f}_2(x)$$

2.2 Affine map

$$f=f_1(Ax+b)$$

2.3 Pointwise max

f = max ⁡ { f 1 ( x ) , f 2 ( x ) } f=\max\{f_1(x),f_2(x)\} f=max{f1​(x),f2​(x)}

2.4 Minimizing out a subset of vars

$$f(x)=\underset{y}{\min }{f}_1(x,y)$$

3 Examples

3.1 Linear, Exponential, Negative entrophy (use hessian)

$$f(x)=ax+b$$ $$f(x)=e^{a^{T}x}$$ $$f(x)=x\log x$$

3.2 All norms

$$f(x)=|x{|}_p,p\ge 1$$

3.3 Maximum singular value of $X$

$$f(X)=\text{max }\lambda \text{ of }X$$

3.4 Pointwise max: Sum of largest n components, Piece wise linear functions