凸优化学习笔记2

Chapter 3 Convex functions 凸函数的相关内容

3.1 Basic properties and examples

Definition(定义1): $\mathbf{d}\mathbf{o}\mathbf{m}f$ is a convex set, $0\le \theta \le 1$, $f(\theta x+(1-\theta )y)\le \theta f(x)+(1-\theta )f(y)$.

• $f$ is concave if $-f$ is convex.

• (定义2)$f$ is convex $\iff$ for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$ and all $v$, $g(t)=f(x+tv)$ is convex, t ∈ { t ∣ x + t v ∈ d o m f } t\in\{t|x+tv\in\mathbf{dom}f\} t∈{t∣x+tv∈domf}).

• extended-value extension:
  $$\tilde{f}(x)=\{\begin{array}{ll}f(x)& x\in \mathbf{d}\mathbf{o}\mathbf{m}f\\ \infty & x\notin \mathbf{d}\mathbf{o}\mathbf{m}f.\end{array}$$

  • indicator function:
    $${\tilde{I}}_C(x)=\{\begin{array}{ll}0& x\in C\\ \infty & x\notin C.\end{array}$$

• (定义3) $f$ is differentiable, $f$ is convex $\iff$ $\mathbf{d}\mathbf{o}\mathbf{m}f$ is convex, $f(y)\ge f(x)+\nabla f(x{)}^{\mathrm{T}}(y-x)$ for all $x,y\in \mathbf{d}\mathbf{o}\mathbf{m}f$.

• (定义4)$f$ is twice differentiable, $f$ is convex $\iff$ $\mathbf{d}\mathbf{o}\mathbf{m}f$ is convex, for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$, ${\nabla }^{2}f(x)⪰0$.

• Examples

  • Exponential. $e^{ax}$ is convex on $\mathbb{R}$, for any $a\in \mathbb{R}$.
  • Powers. $x^a$ is convex on ${\mathbb{R}}_{++}$ when $a\ge 1$ or $a\le 0$, and concave for $0\le a\le 1$.
  • Powers of absolute value. $|x{|}^p$, for $p\ge 1$, is convex on $\mathbb{R}$.
  • Logarithm. $\log x$ is concave on ${\mathbb{R}}_{++}$.
  • Negative entropy. $x\log x$ (either on ${\mathbb{R}}_{++}$ or on ${\mathbb{R}}_{+}$, defined as 0 for $x=0$) is convex.
  • Norms. Every norm on ${\mathbb{R}}^n$ is convex.
  • Max function. f ( x ) = m a x { x 1 , … , x n } f(x)=max\{x_1,\ldots,x_n\} f(x)=max{x1​,…,xn​} is convex on ${\mathbb{R}}^n$.
  • Quadratic-over-linear function. The function $f(x,y)=x^2/y$, with d o m f = R × R + + = { ( x , y ) ∈ R 2 ∣ y > 0 } \mathbf{dom}f=\mathbf{R}\times\mathbf{R}_{++}=\{(x,y)\in\mathbb{R}^2\vert y>0\} domf=R×R++​={(x,y)∈R2∣y>0}, is convex.
  • Log-sum-exp. The function $f(x)=\log (e^{x_1}+\cdots +e^{x_n})$ is convex on ${\mathbb{R}}^n$.
  • Geometric mean. The geometric mean $f(x)=({\prod }_{i=1}^n{x}_i{)}^{1/n}$ is concave on $\mathbf{d}\mathbf{o}\mathbf{m}f={\mathbb{R}}_{++}^n$.
  • Log-determinant. The function $f(X)=\log \det X$ is concave on $\mathbf{d}\mathbf{o}\mathbf{m}f={\mathbf{S}}_{++}^n$.

• Sublevel sets: $\alpha$-sublevel set C α = { x ∈ d o m f ∣ f ( x ) ≤ α } C_\alpha=\{x\in\mathbf{dom}f\vert f(x)\leq\alpha\} Cα​={x∈domf∣f(x)≤α}. Sublevel sets of a convex function are convex for any value of $\alpha$.

  • If $f$ is concave, then its $\alpha$-superlevel set { x ∈ d o m f ∣ f ( x ) ≥ α } \{x\in\mathbf{dom}f\vert f(x)\geq\alpha\} {x∈domf∣f(x)≥α} is a convex set.

• Epigraph: graph { ( x , f ( x ) ) ∣ x ∈ d o m f } \{(x,f(x))\vert x\in\mathbf{dom}f\} {(x,f(x))∣x∈domf}, epigraph e p i f = { ( x , t ) ∣ x ∈ d o m f , f ( x ) ≤ t } \mathbf{epi}f=\{(x,t)\vert x\in\mathbf{dom}f,f(x)\leq t\} epif={(x,t)∣x∈domf,f(x)≤t}, hypograph h y p o f = { ( x , t ) ∣ t ≤ f ( x ) } \mathbf{hypo}f=\{(x,t)\vert t\leq f(x)\} hypof={(x,t)∣t≤f(x)}.

  • A function is convex $\iff$ its epigraph is a convex set.
  • A function is concave $\iff$ its hypograph is a convex set.

• Jensen’s inequality: $f(\mathbf{E}x)\le \mathbf{E}f(x)$, $x$ is a random variable such that $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$ with probability one, and $f$ is convex.

  • $a^{\theta }{b}^{1-\theta }\le \theta a+(1-\theta )b$ for $a,b\ge 0$ and $0\le \theta \le 1$, by convexity of $-\log x$.
    • $\sum _{i=1}^n{x}_i{y}_i\le {\left(\sum _{i=1}^n|x_i{|}^p\right)}^{1/p}{\left(\sum _{i=1}^n|y_i{|}^q\right)}^{1/q}$ for $p>1$, $1/p+1/q=1$ and $x,y\in {\mathbb{R}}^n$, by taking $a=\frac{|x_i{|}^p}{{\sum }_{j=1}^n|x_i{|}^p}$, $b=\frac{|y_i{|}^q}{{\sum }_{j=1}^n|y_i{|}^q}$, $\theta =1/p$ and summing over $i$.

3.2 Operations that preserve convexity

• nonnegative weighted sums: a nonnegative weighted sum of convex functions $f=w_1{f}_1+\cdots +w_m{f}_m$ is convex.
  • a nonnegative weighted sum of concave functions is concave.
• composition with an affine mapping： $f:{\mathbb{R}}^n\to \mathbb{R}$, $A\in \mathbb{R}$, $b\in {\mathbb{R}}^n$, $g(x)=f(Ax+b)$, then if $f$ is convex, so is $g$; if $f$ is concave, so is $g$.
• pointwise maximum and supremum: if $f_1,\dots ,f_m$ are convex, then their pointwise maximum f ( x ) = max ⁡ { f 1 ( x ) , … , f m ( x ) } f(x)=\max\{f_1(x),\ldots,f_m(x)\} f(x)=max{f1​(x),…,fm​(x)} is also convex.
  • If for each $y\in \mathcal{A}$, $f(x,y)$ is convex in $x$, then $g(x)=\underset{y\in \mathcal{A}}{\sup }f(x,y)$ is convex in $x$.
  • The pointwise infimum of a set of concave functions is a concave function.
  • support function: S C ( x ) = sup ⁡ { x T y ∣ y ∈ C } S_C(x)=\sup\{x^\mathrm{T}y\vert y\in C\} SC​(x)=sup{xTy∣y∈C} , d o m S C = { x ∣ sup ⁡ y ∈ C x T y < ∞ } \mathbf{dom}S_C=\{x\vert \sup_{y\in C}x^\mathrm{T}y< \infty\} domSC​={x∣supy∈C​xTy<∞} is convex.
  • almost every convex function can be expressed as the pointwise supremum of a family of affine functions.
• composition: $f(x)=h(g(x))$
  • scalar composition: $h:\mathbb{R}\to \mathbb{R}$, $g:{\mathbb{R}}^n\to \mathbb{R}$
    • $f$ is convex if $h$ is convex, $\tilde{h}$ is nondecreasing, and $g$ is convex
    • $f$ is convex if $h$ is convex, $\tilde{h}$ is nonincreasing, and $g$ is concave
    • $f$ is concave if $h$ is concave, $\tilde{h}$ is nondecreasing, and $g$ is concave
    • $f$ is concave if $h$ is concave, $\tilde{h}$ is nonincreasing, and $g$ is convex
  • vector composition: $h:{\mathbb{R}}^k\to \mathbb{R}$, $g_i:{\mathbb{R}}^n\to \mathbb{R}$: similar derivations
• minimization: $f$ is convex in $(x,y)$, $C$ is a convex nonempty set $\Rightarrow$ $g(x)=\underset{y\in C}{\inf }f(x,y)$ is convex in $x$.
• perspective: $g(x,t)=tf(x/t)$. $f$ is a convex function $\Rightarrow$ $g$ is convex. Similarly, if $f$ is concave, then so is $g$.

3.3 The conjugate function

• definition: $f^{\ast }(y)=\underset{x\in \mathbf{d}\mathbf{o}\mathbf{m}f}{\sup }(y^{\mathrm{T}}x-f(x))$, a convex function.
• basic properties:
  • Fenchel’s inequality: $f(x)+f^{\ast }(y)\ge x^{\mathrm{T}}y$.
  • if $f$ is convex, and $f$ is closed (i.e., $\mathbf{e}\mathbf{p}\mathbf{i}f$ is a closed set), then $f^{\ast \ast }=f$.
  • $f$ is convex and differentiable with $\mathbf{d}\mathbf{o}\mathbf{m}f={\mathbb{R}}^n$, let $z\in {\mathbb{R}}^n$ be arbitrary and define $y=\nabla f(z)$, then $f^{\ast }(y)=z^{\mathrm{T}}\nabla f(z)-f(z)$.
  • $g(x)=af(x)+b$, $a>0$, $b\in \mathbb{R}$, then $g^{\ast }(y)=a{f}^{\ast }(y/a)-b$; $g(x)=f(Ax+b)$, $A\in {\mathbb{R}}^{n\times n}$ nonsingular, $b\in {\mathbb{R}}^n$, then $g^{\ast }(y)=f^{\ast }(A^{-\mathrm{T}}y)-b^{\mathrm{T}}{A}^{-\mathrm{T}}y$.
  • If $f(u,v)=f_1(u)+f_2(v)$, $f_1$ and $f_2$ are convex functions with conjugates $f_1^{\ast }$ and $f_2^{\ast }$, respectively, then $f^{\ast }(w,z)=f_1^{\ast }(w)+f_2^{\ast }(z)$.

3.4 Quasiconvex functions

• definition(定义1): $\mathbf{d}\mathbf{o}\mathbf{m}f$ and all sublevel sets S α = { x ∈ d o m f ∣ f ( x ) ≤ α } S_\alpha=\{x\in\mathbf{dom}f\vert f(x)\leq\alpha\} Sα​={x∈domf∣f(x)≤α} for $\alpha \in \mathbb{R}$ are convex.
  • a function is quasiconcave if $-f$ is quasiconvex, i.e., every superlevel set { x ∣ f ( x ) ≥ α } \{x\vert f(x)\geq\alpha\} {x∣f(x)≥α} is convex.
  • a function that is both quasiconvex and quasiconcave is called quasilinear.
• basic properties:
  • (定义2)$f$ is quasiconvex $\iff$ $\mathbf{d}\mathbf{o}\mathbf{m}f$ is convex, $x,y\in \mathbf{d}\mathbf{o}\mathbf{m}f$, $0\le \theta \le 1$, f ( θ x + ( 1 − θ ) y ) ≤ max ⁡ { f ( x ) , f ( y ) } f(\theta x+(1-\theta)y)\leq\max\{f(x),f(y)\} f(θx+(1−θ)y)≤max{f(x),f(y)}.
  • $f$ is quasiconvex $\iff$ its restriction to any line intersecting its domain is quasiconvex.
  • $f:\mathbb{R}\to \mathbb{R}$ is quasiconvex $\iff$ (at least one condition holds)
    • $f$ is nondecreasing
    • $f$ is nonincreasing
    • $\exists c\in \mathbf{d}\mathbf{o}\mathbf{m}f$, for $t\le c$ and $t\in \mathbf{d}\mathbf{o}\mathbf{m}f$, $f$ is nonincreasing; for $t\ge c$ and $t\in \mathbf{d}\mathbf{o}\mathbf{m}f$, $f$ is nondecreasing
• (定义3)$f$ is differentiable, $f$ is quasiconvex $\iff$ $\mathbf{d}\mathbf{o}\mathbf{m}f$ is convex, $x,y\in \mathbf{d}\mathbf{o}\mathbf{m}f$, $f(y)\le f(x)\Rightarrow \nabla f^{\mathrm{T}}(x)(y-x)\le 0$.
• (定义4)$f$ is twice differentiable:
  • $f$ is quasiconvex $\Rightarrow$ $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$, $y\in {\mathbb{R}}^n$, $y^{\mathrm{T}}\nabla f(x)=0\Rightarrow y^{\mathrm{T}}{\nabla }^{2}f(x)y\ge 0$.
  • $y^{\mathrm{T}}\nabla f(x)=0\Rightarrow y^{\mathrm{T}}{\nabla }^{2}f(x)y>0$ for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$ and all $y\in {\mathbb{R}}^n$, $y\ne 0$, then $f$ is quasiconvex.
• operations that preserve quasiconvexity
  • nonnegative weighted maximum: f = max ⁡ { w 1 f 1 , … , w m f m } f=\max\{w_1f_1,\ldots,w_mf_m\} f=max{w1​f1​,…,wm​fm​} with $w_i\ge 0$ and $f_i$ quasiconvex, is quasiconvex.
  • composition:
    • $g:{\mathbb{R}}^n\to \mathbb{R}$ is quasiconvex, $h:\mathbb{R}\to \mathbb{R}$ is nondecreasing, $f=h\circ g$ is quasiconvex.
    • $f$ is quasiconvex, $g(x)=f(Ax+b)$ is quasiconvex.
    • $f$ is quasiconvex, $\tilde{g}(x)=f((Ax+b)/(c^{\mathrm{T}}x+d))$ is quasiconvex on { x ∣ c T x + d > 0 , ( A x + b ) / ( c T x + d ) ∈ d o m f } \{x\vert c^\mathrm{T}x+d>0,(Ax+b)/(c^\mathrm{T}x+d)\in\mathbf{dom}f\} {x∣cTx+d>0,(Ax+b)/(cTx+d)∈domf}.
  • minimization: $f(x,y)$ is quasiconvex jointly in $x$ and $y$, $C$ is a convex set, then $g(x)=\underset{y\in C}{\inf }f(x,y)$ is quasiconvex.
• representation via family of convex functions: $f(x)\le t\iff {\varphi }_t(x)\le 0$.

3.5 Log-concave and log-convex functions

• definition(定义1):
  • log-concave: $f(x)>0$ for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$ and $\log f$ is concave.
  • log-convex: $f(x)>0$ for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$ and $\log f$ is convex.
• (定义2) $f:{\mathbb{R}}^n\to \mathbb{R}$ with convex domain and $f(x)>0$ for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$ is log-concave $\iff$ for all $x,y\in \mathbf{d}\mathbf{o}\mathbf{m}f$ and $0\le \theta \le 1$, $f(\theta x+(1-\theta )y)\ge f^{\theta }(x)f^{1-\theta }(y)$.
• properties:
  • (定义3) $f$ is twice differentiable with $\mathrm{d}\mathrm{o}\mathrm{m}f$ convex. $f$ is log-convex $\iff$ for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$, $f(x){\nabla }^{2}f(x)⪰\nabla f(x){\nabla }^{\mathrm{T}}f(x)$; $f$ is log-concave $\iff$ for all $x\in \mathbf{d}\mathbf{o}\mathbf{m}f$, $f(x){\nabla }^{2}f(x)⪯\nabla f(x){\nabla }^{\mathrm{T}}f(x)$.
  • Log-convexity and log-concavity are closed under multiplication and positive scaling.
  • Log-convexity is preserved under sums. If $f(x,y)$ is log-convex in $x$ for each $y\in C$, $g(x)={\int }_{C}f(x,y)\mathrm{d}y$ is log-convex.
  • If $f:{\mathbb{R}}^n\times {\mathbb{R}}^m\to \mathbb{R}$ is log-concave, then $g(x)=\int f(x,y)\mathrm{d}y$ is a log-concave function of $x$.

3.6 Convexity with respect to generalized inequalities

• K-nondecreasing: $f:{\mathbb{R}}^n\to \mathbb{R}$, $x{⪯}_{K}y⟹$ $f(x)\le f(y)$.

• K-increasing: $f:{\mathbb{R}}^n\to \mathbb{R}$, $x{⪯}_{K}y,x\ne y⟹$ $f(x)<f(y)$.

• $f$ is differentiable with convex domain is K-nondecreasing $\iff$ $\nabla f(x){⪰}_K^{\ast }0$.

• $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$ is K-convex: for all $x$, $y$, and $0\le \theta \le 1$, $f(\theta x+(1-\theta )y){⪯}_K\theta f(x)+(1-\theta )f(y)$.