Convex Optimization 读书笔记 (2)

Chapter3: Convex Functions

3.1 Basic properties and examples

3.1.1 Definition

A function $f:{\mathbf{R}}^n\to \mathbf{R}$ is a convex function if $\mathbf{d}\mathbf{o}\mathbf{m}f$ is convex set and for $x,y\in \mathbf{d}\mathbf{o}\mathbf{m}f,\theta \in [0,1]$, we have
$$f(\theta x+(1-\theta )y)\le \theta f(x)+(1-\theta )f(y)$$

$f$ is concave if $-f$ is convex.

$f$ is convex if and only if for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$, and for all $v$, the function $g(t)=f(x+tv)$ is convex.

3.1.2 Extended-value extensions

If f is convex define its extended-value extension f ~ : R n → R ∪ { ∞ } \tilde{f}:\mathbf{R^n}\rightarrow\mathbf{R}\cup \{\infty\} f~​:Rn→R∪{∞}:
$$\tilde{f}=\{\begin{array}{rc}f(x)& x\in \mathbf{d}\mathbf{o}\mathbf{m}f\\ \infty & x\notin \mathbf{d}\mathbf{o}\mathbf{m}f\end{array}$$

3.1.3 First-order conditions

Suppose $f$ is a convex, then
$$f(y)\ge f(x)+\nabla f(x{)}^T(y-x)$$
holds for all $x,y\in \mathbf{d}\mathbf{o}\mathbf{m}f$.

3.1.4 Second-order conditions

$f$ is convex if and only if $\mathbf{d}\mathbf{o}\mathbf{m}f$ is convex and for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$, ${\nabla }^{2}f⪰0$.

3.1.5 Examples

Powers of absolute value. $|x{|}^p$ for $p\ge 1$ is convex.

Negative entropy. $xlogx$ (either on ${\mathbf{R}}_{++}$, or on ${\mathbf{R}}_{+}$, defined as $0$ for $x=0$) is convex.

Norms. Every norm on ${\mathbf{R}}^n$ is convex.

Log-sum-exp. The function $f(x)=\log (e^{x_1}+\cdot \cdot \cdot +e^{x_n})$ is convex on R .

Geometric mean. The geometric mean $f(x)=({\prod }_i^n{x}_i{)}^{\frac{1}{n}}$ is concave on $\mathbf{d}\mathbf{o}\mathbf{m}f={\mathbf{R}}_{++}^n$.

Log-determinant. The function$ f (X ) = \log \det X $ is concave on $\mathbf{d}\mathbf{o}\mathbf{m}f={\mathbf{S}}_{++}^n$.

3.1.6 Sublevel sets

The $\alpha$-sublevel set of a function $f:{\mathbf{R}}^n\to \mathbf{R}$ is
{ x ∣ x ∈ d o m f ; f ( x ) ≤ α } \{ x\mid x\in \mathbf{dom}f;f(x)\leq \alpha \} {x∣x∈domf;f(x)≤α}
Sublevel sets of a convex function are convex.

3.1.7 Epigraph

A Epigraph of a function $f:{\mathbf{R}}^n\to \mathbf{R}$ is
e p i f = { ( x , t ) ∣ x ∈ d o m f ; f ( x ) ≤ t } \mathbf{epi}f=\{ (x,t) \mid x\in \mathbf{dom}f;f(x)\leq t \} epif={(x,t)∣x∈domf;f(x)≤t}
is a subset of ${\mathbf{R}}^{n+1}$.

3.1.8 Jensen’s inequality and extensions

The basic inequality can be extended to
$$f({\theta }_1{x}_1+\cdots +{\theta }_n{x}_n)\le {\theta }_{1}f(x_1)+\cdots +{\theta }_{n}f(x_n)$$
where ${\sum }_i{\theta }_i=1,{\theta }_i>0,x_i\in \mathbf{d}\mathbf{o}\mathbf{m}f,f$ is convex function. For intergral, that is
$$f\left({\int }_{S}p(x)x\right)\le {\int }_{S}p(x)f(x)$$
It is the expectation inequality
$$f(\mathbb{E}(x))\le \mathbb{E}(f(x))$$

3.1.9 Inequalities

3.2 Operations that preserve convexity

3.2.1 Nonnegative weighted sums

Suppose $f_i$ is convex and $w_i>0$,
$$gx(x)=\sum _i{w}_i{f}_i(x)$$
is convex.

3.2.2 Composition with an affine mapping

Suppose $f:{\mathbf{R}}^n\to \mathbf{R},A\in {\mathbf{R}}^{n\times m},b\in {\mathbf{R}}^n$, define $g:{\mathbf{R}}^m\to \mathbf{R}$:
g ( x ) = f ( A x + b ) , x ∈ { x ∣ A x + b ∈ d o m f } g(x)=f(Ax+b),x\in \{ x\mid Ax+b\in \mathbf{dom}f \} g(x)=f(Ax+b),x∈{x∣Ax+b∈domf}
its convexity is same with $f$.

3.2.3 Pointwise maximum and supremum

If $f_1$ and $f_2$ are convex functions then their pointwise maximum $f$, defined by
f ( x ) = max ⁡ { f 1 ( x ) , f 2 ( x ) } , d o m f = d o m f 1 ∩ d o m f 2 f(x)=\max\{ f_1(x),f_2(x) \}, \mathbf{dom}f=\mathbf{dom}f_1\cap\mathbf{dom}f_2 f(x)=max{f1​(x),f2​(x)},domf=domf1​∩domf2​
is convex.

For each $y\in \mathcal{A},f(x,y)$ is convex in $x$, the pointwise supremum
$$g(x)=\underset{y\in \mathcal{A}}{\sup }f(x,y)$$
is convex. Where d o m   g = { x ∣ ( x , y ) ∈ d o m   g , sup ⁡ y ∈ A f ( x , y ) < ∞ } \mathbf{dom}\space g=\{ x\mid (x,y)\in \mathbf{dom} \space g, \sup_{\mathcal{y\in A}}f(x,y) <\infty \} dom g={x∣(x,y)∈dom g,supy∈A​f(x,y)<∞}.

3.2.4 Composition

3.2.5 Minimization

If $f$ is convex in $(x,y)$, and $C$ is a convex nonempty set, then the function
$$g(x)=\underset{y\in C}{\inf }f(x,y)$$
is convex.

3.2.6 Perspective of a function

If $f:{\mathbf{R}}^n\to \mathbf{R},$ the perspective of $f$ is $g:{\mathbf{R}}^{n+1}\to \mathbf{R}$:
$$g(x,t)=tf(x/t)$$
with domian
d o m g = { ( x , t ) ∣ x / t ∈ d o m f , t > 0 } \mathbf{dom}g=\{ (x,t)\mid x/t\in \mathbf{dom}f,t>0\} domg={(x,t)∣x/t∈domf,t>0}
It is convex if $f$ is convex.

3.3 The conjugate function

3.3.1 Definition and examples

If KaTeX parse error: Undefined control sequence: \mbox at position 37: …rrow\mathbf{R},\̲m̲b̲o̲x̲{the function }… is called conjugate function if
$$f^{\ast }(y)=\underset{x\in \mathbf{d}\mathbf{o}\mathbf{m}f}{\sup }(y^{T}x-f(x))$$
The domain of the conjugate function consists of for which the supremum is finite.

3.3.2 Basic properties

Fenchel’s inequality
$$f(x)+f^{\ast }(y)\ge x^{T}y$$
Conjugate of the conjugate

The conjugate of the conjugate of a convex function is the original function.

Differentiable functions

Let $z\in {\mathbf{R}}^n,y=\nabla f(z)$
$$f^{\ast }(y)=z^T\nabla f(z)-f(z)$$
Scaling and composition with affine transformation

Conjugate of $g(x)=af(x)+b$ is $g^{\ast }(y)=a{f}^{\ast }(y/a)-b$.

Conjugate of $g(x)=f(Ax+b)$ is $g^{\ast }(y)=f^{\ast }(A^{-1}y)-b^T{A}^{-T}y$.

Sums of independent functions

If $f(x,y)=f_1(x)+f_2(y)$, then $f^{\ast }(u,v)=f_1^{\ast }(u)+f_2^{\ast }(v)$.

3.4 Quasiconvex functions

3.4.1 Definition and examples

A function $f:{\mathbf{R}}^n\to \mathbf{R}$ is called quasiconvex (or unimodal) if its domain and all its sublevel sets
S α = { x ∈ d o m f ∣ f ( x ) < α } S_{\alpha}=\{ x\in \mathbf{dom} f \mid f(x)<\alpha \} Sα​={x∈domf∣f(x)<α}
Are convex.

3.4.2 Basic properties

The extension of Jenson’s equality is: A function $f$ is quasiconvex if $\mathbf{d}\mathbf{o}\mathbf{m}f$ is convex and for $x,y\in \mathbf{d}\mathbf{o}\mathbf{m}f$, f ( θ x + ( 1 − θ ) y ) ≤ max ⁡ { f ( x ) , f ( y ) } f(\theta x+(1-\theta)y)\leq\max\{f(x),f(y)\} f(θx+(1−θ)y)≤max{f(x),f(y)}

3.4.3 Differentiable quasiconvex functions

First-order conditions

Suppose $f:{\mathbf{R}}^n\to \mathbf{R}$ is differentiable. Then $f$ is quasiconvex if and only if dom $f$ is convex and for all $x,y\in \mathbf{d}\mathbf{o}\mathbf{m}f$
$$f(y)\le f(x)\Rightarrow \nabla f(x{)}^T(y-x)\le 0$$
Second-order conditions

If f is quasiconvex, then for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$ , and all $y\in {\mathbf{R}}^n$, we have
$$y^T\nabla f(x)=0\Rightarrow y^T{\nabla }^{2}f(x)y\ge 0$$

3.4.4 Operations that preserve quasiconvexity

Nonnegative weighted maximum

If $w_i>0,f_i(x)$ is quasiconvex,
f = max ⁡ { ∑ i w i f ( x ) } f=\max\{\sum_iw_if(x) \} f=max{i∑​wi​f(x)}
is quasi convex.
f ( x ) = sup ⁡ y ∈ C { w ( y ) f ( x , y ) } f(x)=\sup_{y\in C}\{ w(y)f(x,y) \} f(x)=y∈Csup​{w(y)f(x,y)}
is quasi convex.

Minimization

If $f(x,y)$ is quasiconvex jointly in $x$ and $y$ and $C$ is a convex set, then the function
$$g(x)=\underset{y\in C}{\inf }f(x,y)$$
is quasi convex.

3.4.5 Representation via family of convex functions

We seek a family of convex functions ${\varphi }_t:{\mathbf{R}}^n\to \mathbf{R}$, indexed by $t\in \mathbf{R}$, with
$$f(x)\le t\iff {\varphi }_t\le 0$$

3.5 Log-concave and log-convex functions

3.5.1 Definition

A function $f:{\mathbf{R}}^n\to \mathbf{R}$ is logarithmically concave or log-concave if $ f(x) > 0$ for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$ and $\log f$ is concave.

if for all $x,y\in \mathbf{d}\mathbf{o}\mathbf{m}f$ and $0\le \theta \le 1$,we have
$$f(\theta x+(1-\theta )y)\le f(x{)}^{\theta }f(y{)}^{1-\theta }$$

3.5.2 Properties

Twice differentiable log-convex/concave functions

We conclude that $f$ is log-convex if and only if for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$,
$$f(x){\nabla }^{2}f(x)⪰\nabla f(x)\nabla f(x{)}^T$$
and log-concave if and only if for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$,
$$f(x){\nabla }^{2}f(x)⪯\nabla f(x)\nabla f(x{)}^T$$
Multiplication, addition, and integration

Log-convexity and log-concavity are closed under multiplication and positive scaling.

The sum of two log-convex functions is log-convex. If $f(x,y)$ is log-convex in $x$ for every $y$, then
$$g(x)=\int f(x,y)dy$$
is log-convex.

Integration of log-concave functions

If $f:{\mathbf{R}}^n\times {\mathbf{R}}^m\to \mathbf{R}$ is log-concave, then $g(x)=\int f(x,y)dy$ is log-concave.

3.6 Convexity with respect to generalized inequalities

3.6.1 Monotonicity with respect to a generalized inequality

Suppose $K\subseteq {\mathbf{R}}^n$ is a proper cone with associated generalized inequality ${⪯}_K$. A function $f:{\mathbf{R}}^n\to \mathbf{R}$ is called $K$-nondecreasing if
$$x{⪯}_{K}y⟹f(x)\le f(y)$$
and $K$-increasing if
$$x{⪯}_{K}y,x\ne y⟹f(x)<f(y)$$

3.6.2 Convexity with respect to a generalized inequality

Suppose $K\subseteq {\mathbf{R}}^m$ is a proper cone with associated generalized inequality ${⪯}_K$. We say $f:{\mathbf{R}}^n\to {\mathbf{R}}^m$ is $K$-convex if for all KaTeX parse error: Undefined control sequence: \mbox at position 7: x, y, \̲m̲b̲o̲x̲{and}\space 0 ≤…,
$$f(\theta x+(1-\theta )y){⪯}_K\theta f(x)+(1-\theta )f(y)$$
Differentiable K-convex functions

A differentiable function $f$ is $K$-convex if and only if its domain is convex, and for all $x,y\in \mathbf{d}\mathbf{o}\mathbf{m}f$,
$$f(y){⪰}_{K}f(x)+Df(x)(y-x)$$
Here $Df(x)\in {\mathbf{R}}^{m\times n}$ is the Jacobian matrix of $f$ at $x$.