Convex Optimization 读书笔记 (2)

Chapter3: Convex Functions

3.1 Basic properties and examples

3.1.1 Definition

A function f : R n → R f:\mathbf{R}^n\rightarrow\mathbf{R} f:RnR is a convex function if d o m   f \mathbf{dom}\space f dom f is convex set and for x , y ∈ d o m   f , θ ∈ [ 0 , 1 ] x,y\in \mathbf{dom}\space f,\theta\in[0,1] x,ydom f,θ[0,1], we have
f ( θ x + ( 1 − θ ) y ) ≤ θ f ( x ) + ( 1 − θ ) f ( y ) f(\theta x+(1-\theta)y)\leq \theta f(x)+(1-\theta)f(y) f(θx+(1θ)y)θf(x)+(1θ)f(y)

f f f is concave if − f -f f is convex.

f f f is convex if and only if for all x ∈ d o m   f x\in \mathbf{dom}\space f xdom f, and for all v v v, the function g ( t ) = f ( x + t v ) g(t)=f(x+tv) g(t)=f(x+tv) is convex.

3.1.2 Extended-value extensions

If f is convex define its extended-value extension f ~ : R n → R ∪ { ∞ } \tilde{f}:\mathbf{R^n}\rightarrow\mathbf{R}\cup \{\infty\} f~:RnR{}:
f ~ = { f ( x ) x ∈ d o m   f ∞ x ∉ d o m   f \tilde{f}=\left\{ \begin{array}{rcl} f(x) &x \in \mathbf{dom}\space f \\ \infty &x \notin \mathbf{dom}\space f \end{array}\right. f~={f(x)xdom fx/dom f

3.1.3 First-order conditions

Suppose f f f is a convex, then
f ( y ) ≥ f ( x ) + ∇ f ( x ) T ( y − x ) f(y)\geq f(x)+\nabla f(x)^T(y-x) f(y)f(x)+f(x)T(yx)
holds for all x , y ∈ d o m f x,y\in \mathbf{dom}f x,ydomf.

3.1.4 Second-order conditions

f f f is convex if and only if d o m f \mathbf{dom}f domf is convex and for all x ∈ d o m f x\in \mathbf{dom}f xdomf, ∇ 2 f ⪰ 0 \nabla^2f\succeq0 2f0.

3.1.5 Examples

Powers of absolute value. ∣ x ∣ p |x|^p xp for p ≥ 1 p\geq1 p1 is convex.

Negative entropy. x l o g x x log x xlogx (either on R + + \mathbf{R}_{++} R++, or on R + \mathbf{R}_+ R+, defined as 0 0 0 for x = 0 x = 0 x=0) is convex.

Norms. Every norm on R n \mathbf{R}^n Rn is convex.

Log-sum-exp. The function f ( x ) = log ⁡ ( e x 1 + ⋅ ⋅ ⋅ + e x n ) f(x) = \log(e^{x_1} +···+e^{x_n} ) f(x)=log(ex1++exn) is convex on R .

Geometric mean. The geometric mean f ( x ) = ( ∏ i n x i ) 1 n f(x) = (\prod_i^{n}x_i)^{\frac{1}{n}} f(x)=(inxi)n1 is concave on d o m f = R + + n \mathbf{dom}f=\mathbf{R}_{++}^n domf=R++n.

Log-determinant. The function$ f (X ) = \log \det X $ is concave on d o m f = S + + n \mathbf{dom}f=\mathbf{S}_{++}^n domf=S++n.

3.1.6 Sublevel sets

The α \alpha α-sublevel set of a function f : R n → R f:\mathbf{R}^n\rightarrow\mathbf{R} f:RnR is
{ x ∣ x ∈ d o m f ; f ( x ) ≤ α } \{ x\mid x\in \mathbf{dom}f;f(x)\leq \alpha \} {xxdomf;f(x)α}
Sublevel sets of a convex function are convex.

3.1.7 Epigraph

A Epigraph of a function f : R n → R f:\mathbf{R}^n\rightarrow\mathbf{R} f:RnR is
e p i f = { ( x , t ) ∣ x ∈ d o m f ; f ( x ) ≤ t } \mathbf{epi}f=\{ (x,t) \mid x\in \mathbf{dom}f;f(x)\leq t \} epif={(x,t)xdomf;f(x)t}
is a subset of R n + 1 \mathbf{R}^{n+1} Rn+1.

3.1.8 Jensen’s inequality and extensions

The basic inequality can be extended to
f ( θ 1 x 1 + ⋯ + θ n x n ) ≤ θ 1 f ( x 1 ) + ⋯ + θ n f ( x n ) f(\theta_1x_1+\cdots+\theta_nx_n)\leq \theta_1f(x_1)+\cdots+\theta_nf(x_n) f(θ1x1++θnxn)θ1f(x1)++θnf(xn)
where ∑ i θ i = 1 , θ i > 0 , x i ∈ d o m f , f \sum_i\theta_i=1,\theta_i>0,x_i \in \mathbf{dom}f,f iθi=1,θi>0,xidomf,f is convex function. For intergral, that is
f ( ∫ S p ( x ) x ) ≤ ∫ S p ( x ) f ( x ) f\left(\int_Sp(x)x\right)\leq \int_Sp(x)f(x) f(Sp(x)x)Sp(x)f(x)
It is the expectation inequality
f ( E ( x ) ) ≤ E ( f ( x ) ) f(\mathbb{E}(x))\leq\mathbb{E}(f(x)) f(E(x))E(f(x))

3.1.9 Inequalities

3.2 Operations that preserve convexity

3.2.1 Nonnegative weighted sums

Suppose f i f_i fi is convex and w i > 0 w_i>0 wi>0,
g x ( x ) = ∑ i w i f i ( x ) gx(x) = \sum_iw_if_i(x) gx(x)=iwifi(x)
is convex.

3.2.2 Composition with an affine mapping

Suppose f : R n → R , A ∈ R n × m , b ∈ R n f:\mathbf{R}^n\rightarrow\mathbf{R}, A\in \mathbf{R}^{n\times m},b\in \mathbf{R}^{n} f:RnR,ARn×m,bRn, define g : R m → R g:\mathbf{R}^m\rightarrow\mathbf{R} g:RmR:
g ( x ) = f ( A x + b ) , x ∈ { x ∣ A x + b ∈ d o m f } g(x)=f(Ax+b),x\in \{ x\mid Ax+b\in \mathbf{dom}f \} g(x)=f(Ax+b),x{xAx+bdomf}
its convexity is same with f f f.

3.2.3 Pointwise maximum and supremum

If f 1 f_1 f1 and f 2 f_2 f2 are convex functions then their pointwise maximum f f f, defined by
f ( x ) = max ⁡ { f 1 ( x ) , f 2 ( x ) } , d o m f = d o m f 1 ∩ d o m f 2 f(x)=\max\{ f_1(x),f_2(x) \}, \mathbf{dom}f=\mathbf{dom}f_1\cap\mathbf{dom}f_2 f(x)=max{f1(x),f2(x)},domf=domf1domf2
is convex.

For each y ∈ A , f ( x , y ) y\in \mathcal{A},f(x,y) yA,f(x,y) is convex in x x x, the pointwise supremum
g ( x ) = sup ⁡ y ∈ A f ( x , y ) g(x)=\sup_{\mathcal{y\in A}}f(x,y) g(x)=yAsupf(x,y)
is convex. Where d o m   g = { x ∣ ( x , y ) ∈ d o m   g , sup ⁡ y ∈ A f ( x , y ) < ∞ } \mathbf{dom}\space g=\{ x\mid (x,y)\in \mathbf{dom} \space g, \sup_{\mathcal{y\in A}}f(x,y) <\infty \} dom g={x(x,y)dom g,supyAf(x,y)<}.

3.2.4 Composition

3.2.5 Minimization

If f f f is convex in ( x , y ) (x,y) (x,y), and C C C is a convex nonempty set, then the function
g ( x ) = inf ⁡ y ∈ C f ( x , y ) g(x)=\inf_{y\in C}f(x,y) g(x)=yCinff(x,y)
is convex.

3.2.6 Perspective of a function

If f : R n → R , f:\mathbf{R}^n\rightarrow\mathbf{R}, f:RnR, the perspective of f f f is g : R n + 1 → R g:\mathbf{R}^{n+1}\rightarrow\mathbf{R} g:Rn+1R:
g ( x , t ) = t f ( x / t ) g(x,t)=tf(x/t) g(x,t)=tf(x/t)
with domian
d o m g = { ( x , t ) ∣ x / t ∈ d o m f , t > 0 } \mathbf{dom}g=\{ (x,t)\mid x/t\in \mathbf{dom}f,t>0\} domg={(x,t)x/tdomf,t>0}
It is convex if f f f is convex.

3.3 The conjugate function

3.3.1 Definition and examples

If KaTeX parse error: Undefined control sequence: \mbox at position 37: …rrow\mathbf{R},\̲m̲b̲o̲x̲{the function }… is called conjugate function if
f ∗ ( y ) = sup ⁡ x ∈ d o m f ( y T x − f ( x ) ) f^*(y) = \sup_{x\in \mathbf{dom}f}(y^Tx-f(x)) f(y)=xdomfsup(yTxf(x))
The domain of the conjugate function consists of for which the supremum is finite.

3.3.2 Basic properties

Fenchel’s inequality
f ( x ) + f ∗ ( y ) ≥ x T y f(x)+f^*(y)\geq x^Ty f(x)+f(y)xTy
Conjugate of the conjugate

The conjugate of the conjugate of a convex function is the original function.

Differentiable functions

Let z ∈ R n , y = ∇ f ( z ) z\in \mathbf{R}^n,y=\nabla f(z) zRn,y=f(z)
f ∗ ( y ) = z T ∇ f ( z ) − f ( z ) f^*(y)=z^T\nabla f(z)-f(z) f(y)=zTf(z)f(z)
Scaling and composition with affine transformation

Conjugate of g ( x ) = a f ( x ) + b g(x)=af(x)+b g(x)=af(x)+b is g ∗ ( y ) = a f ∗ ( y / a ) − b g^*(y)=af^*(y/a)-b g(y)=af(y/a)b.

Conjugate of g ( x ) = f ( A x + b ) g(x)=f(Ax+b) g(x)=f(Ax+b) is g ∗ ( y ) = f ∗ ( A − 1 y ) − b T A − T y g^*(y)=f^*(A^{-1}y)-b^TA^{-T}y g(y)=f(A1y)bTATy.

Sums of independent functions

If f ( x , y ) = f 1 ( x ) + f 2 ( y ) f(x,y)=f_1(x)+f_2(y) f(x,y)=f1(x)+f2(y), then f ∗ ( u , v ) = f 1 ∗ ( u ) + f 2 ∗ ( v ) f^*(u,v)=f_1^*(u)+f_2^*(v) f(u,v)=f1(u)+f2(v).

3.4 Quasiconvex functions

3.4.1 Definition and examples

A function f : R n → R f:\mathbf{R}^n\rightarrow\mathbf{R} f:RnR is called quasiconvex (or unimodal) if its domain and all its sublevel sets
S α = { x ∈ d o m f ∣ f ( x ) < α } S_{\alpha}=\{ x\in \mathbf{dom} f \mid f(x)<\alpha \} Sα={xdomff(x)<α}
Are convex.

3.4.2 Basic properties

The extension of Jenson’s equality is: A function f f f is quasiconvex if d o m f \mathbf{dom}f domf is convex and for x , y ∈ d o m f x,y\in \mathbf{dom}f x,ydomf, f ( θ x + ( 1 − θ ) y ) ≤ max ⁡ { f ( x ) , f ( y ) } f(\theta x+(1-\theta)y)\leq\max\{f(x),f(y)\} f(θx+(1θ)y)max{f(x),f(y)}

3.4.3 Differentiable quasiconvex functions

First-order conditions

Suppose f : R n → R f:\mathbf{R}^n\rightarrow\mathbf{R} f:RnR is differentiable. Then f f f is quasiconvex if and only if dom f f f is convex and for all x , y ∈ d o m f x, y ∈ \mathbf{dom}f x,ydomf
f ( y ) ≤ f ( x ) ⇒ ∇ f ( x ) T ( y − x ) ≤ 0 f(y)\leq f(x) \Rightarrow\nabla f(x)^T(y-x)\leq0 f(y)f(x)f(x)T(yx)0
Second-order conditions

If f is quasiconvex, then for all x ∈ d o m f x ∈ \mathbf{dom}f xdomf , and all y ∈ R n y ∈ \mathbf{R}^n yRn, we have
y T ∇ f ( x ) = 0 ⇒ y T ∇ 2 f ( x ) y ≥ 0 y^T\nabla f(x)=0\Rightarrow y^T \nabla ^2f(x)y\geq0 yTf(x)=0yT2f(x)y0

3.4.4 Operations that preserve quasiconvexity

Nonnegative weighted maximum

If w i > 0 , f i ( x ) w_i>0,f_i(x) wi>0,fi(x) is quasiconvex,
f = max ⁡ { ∑ i w i f ( x ) } f=\max\{\sum_iw_if(x) \} f=max{iwif(x)}
is quasi convex.
f ( x ) = sup ⁡ y ∈ C { w ( y ) f ( x , y ) } f(x)=\sup_{y\in C}\{ w(y)f(x,y) \} f(x)=yCsup{w(y)f(x,y)}
is quasi convex.

Minimization

If f ( x , y ) f(x,y) f(x,y) is quasiconvex jointly in x x x and y y y and C C C is a convex set, then the function
g ( x ) = inf ⁡ y ∈ C f ( x , y ) g(x) = \inf_{y\in C}f(x,y) g(x)=yCinff(x,y)
is quasi convex.

3.4.5 Representation via family of convex functions

We seek a family of convex functions ϕ t : R n → R \phi_t : \mathbf{R}^n\rightarrow\mathbf{R} ϕt:RnR, indexed by t ∈ R t ∈ \mathbf{R} tR, with
f ( x ) ≤ t ⇔ ϕ t ≤ 0 f(x)\leq t \Leftrightarrow\phi_t\leq0 f(x)tϕt0

3.5 Log-concave and log-convex functions

3.5.1 Definition

A function f : R n → R f : \mathbf{R}^n\rightarrow\mathbf{R} f:RnR is logarithmically concave or log-concave if $ f(x) > 0$ for all x ∈ d o m f x ∈ \mathbf{dom}f xdomf and log ⁡ f \log f logf is concave.

if for all x , y ∈ d o m f x, y∈\mathbf{dom}f x,ydomf and 0 ≤ θ ≤ 1 0≤θ≤1 0θ1,we have
f ( θ x + ( 1 − θ ) y ) ≤ f ( x ) θ f ( y ) 1 − θ f(\theta x+(1-\theta)y)\leq f(x)^{\theta}f(y)^{1-\theta} f(θx+(1θ)y)f(x)θf(y)1θ

3.5.2 Properties

Twice differentiable log-convex/concave functions

We conclude that f f f is log-convex if and only if for all x ∈ d o m f x ∈ \mathbf{dom} f xdomf,
f ( x ) ∇ 2 f ( x ) ⪰ ∇ f ( x ) ∇ f ( x ) T f(x)\nabla^2f(x) \succeq \nabla f(x)\nabla f(x)^T f(x)2f(x)f(x)f(x)T
and log-concave if and only if for all x ∈ d o m f x ∈ \mathbf{dom} f xdomf,
f ( x ) ∇ 2 f ( x ) ⪯ ∇ f ( x ) ∇ f ( x ) T f(x)\nabla^2f(x) \preceq \nabla f(x)\nabla f(x)^T f(x)2f(x)f(x)f(x)T
Multiplication, addition, and integration

Log-convexity and log-concavity are closed under multiplication and positive scaling.

The sum of two log-convex functions is log-convex. If f ( x , y ) f(x,y) f(x,y) is log-convex in x x x for every y y y, then
g ( x ) = ∫ f ( x , y ) d y g(x)=\int f(x,y)dy g(x)=f(x,y)dy
is log-convex.

Integration of log-concave functions

If f : R n × R m → R f : \mathbf{R}^n \times \mathbf{R}^m\rightarrow\mathbf{R} f:Rn×RmR is log-concave, then g ( x ) = ∫ f ( x , y ) d y g(x)=\int f(x,y)dy g(x)=f(x,y)dy is log-concave.

3.6 Convexity with respect to generalized inequalities

3.6.1 Monotonicity with respect to a generalized inequality

Suppose K ⊆ R n K ⊆ \mathbf{R}^n KRn is a proper cone with associated generalized inequality ⪯ K \preceq_K K. A function f : R n → R f : \mathbf{R}^n\rightarrow\mathbf{R} f:RnR is called K K K-nondecreasing if
x ⪯ K y ⟹ f ( x ) ≤ f ( y ) x\preceq_Ky \Longrightarrow f(x)\leq f(y) xKyf(x)f(y)
and K K K-increasing if
x ⪯ K y , x ≠ y ⟹ f ( x ) < f ( y ) x\preceq_Ky,x\neq y \Longrightarrow f(x)< f(y) xKy,x=yf(x)<f(y)

3.6.2 Convexity with respect to a generalized inequality

Suppose K ⊆ R m K ⊆ \mathbf{R}^m KRm is a proper cone with associated generalized inequality ⪯ K \preceq_K K. We say f : R n → R m f : \mathbf{R}^n\rightarrow\mathbf{R}^m f:RnRm is K K K-convex if for all KaTeX parse error: Undefined control sequence: \mbox at position 7: x, y, \̲m̲b̲o̲x̲{and}\space 0 ≤…,
f ( θ x + ( 1 − θ ) y ) ⪯ K θ f ( x ) + ( 1 − θ ) f ( y ) f(\theta x+(1-\theta)y)\preceq_K \theta f(x)+(1-\theta)f(y) f(θx+(1θ)y)Kθf(x)+(1θ)f(y)
Differentiable K-convex functions

A differentiable function f f f is K K K-convex if and only if its domain is convex, and for all x , y ∈ d o m f x, y∈\mathbf{dom}f x,ydomf,
f ( y ) ⪰ K f ( x ) + D f ( x ) ( y − x ) f(y)\succeq_Kf(x)+Df(x)(y-x) f(y)Kf(x)+Df(x)(yx)
Here D f ( x ) ∈ R m × n Df(x)\in\mathbf{R}^{m\times n} Df(x)Rm×n is the Jacobian matrix of f f f at x x x.

©️2020 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页