凸优化学习笔记1

学习教材主要是Boyd的《Convex Optimization》

Chapter 2 convex sets，主要是convex sets的一些数学定义，以及一些常见的convex sets。

2.1 Affine and convex sets

• lines: $y=\theta x_1+(1-\theta )x_2=x_2+\theta (x_1-x_2)$, $\theta \in \mathbb{R}$, $x_1,x_2\in {\mathbb{R}}^n,x_1\ne x_2$

• affine sets: $x_1,x_2\in C$, $\theta \in \mathbb{R}$, $\theta x_1+(1-\theta )x_2\in C$

  • affine hull: a f f C = { θ 1 x 1 + ⋯ + θ k x k ∣ x 1 , … , x k ∈ C , θ 1 + ⋯ + θ k = 1 } \mathbf{aff} C=\{\theta_1 x_1+\dots+\theta_k x_k|x_1,\ldots,x_k\in C, \theta_1+\dots+\theta_k=1\} affC={θ1​x1​+⋯+θk​xk​∣x1​,…,xk​∈C,θ1​+⋯+θk​=1}, affine dimension of $C$ is the dimension of its affine hull
  • relative interior: r e l i n t C = { x ∈ C ∣ B ( x , r ) ∩ a f f C ⊆ C    for some    r > 0 } \mathbf{relint}C=\{x\in C|B(x,r)\cap\mathbf{aff}C\subseteq C\;\text{for some}\;r>0\} relintC={x∈C∣B(x,r)∩affC⊆Cfor somer>0}, B ( x , r ) = { y ∣ ∥ y − x ∥ ≤ r } B(x,r)=\{y|\Vert y-x\Vert\leq r\} B(x,r)={y∣∥y−x∥≤r}
  • relative boundary: $\mathbf{c}\mathbf{l}C\backslash \mathbf{r}\mathbf{e}\mathbf{l}\mathbf{i}\mathbf{n}\mathbf{t}C$,

• convex sets: $x_1,x_2\in C$, $0\le \theta \le 1$, $\theta x_1+(1-\theta )x_2\in C$

  • convex hull: c o n v C = { θ 1 x 1 + ⋯ + θ k x k ∣ x i ∈ C , θ i ≥ 0 , i = 1 , … , k , θ 1 + ⋯ + θ k = 1 } \mathbf{conv}C=\{\theta_1x_1+\dots+\theta_kx_k|x_i\in C, \theta_i\geq 0, i=1,\ldots,k,\theta_1+\dots+\theta_k=1\} convC={θ1​x1​+⋯+θk​xk​∣xi​∈C,θi​≥0,i=1,…,k,θ1​+⋯+θk​=1}
  • $p:{\mathbb{R}}^n\to \mathbb{R}$, $p(x)\ge 0$, $x\in C$, $C\subseteq {\mathbb{R}}^n$ is convex, ${\int }_{C}p(x)\mathrm{d}x=1$,${\int }_{C}p(x)x\mathrm{d}x\in C$

• cones: $x\in C$, $\theta \ge 0$, $\theta x\in C$

  • convex cone: $x_1,x_2\in C$, ${\theta }_1,{\theta }_2\ge 0$, ${\theta }_1{x}_1+{\theta }_2{x}_2\in C$
  • conic hull: { θ 1 x 1 + ⋯ + θ k x k ∣ x i ∈ C , θ i ≥ 0 , i = 1 , … , k } \{\theta_1x_1+\dots+\theta_kx_k |x_i\in C, \theta_i\geq 0, i=1,\ldots,k\} {θ1​x1​+⋯+θk​xk​∣xi​∈C,θi​≥0,i=1,…,k}

affine$\to$convex

2.2 Some important examples

• hyperplane: { x ∣ a T x = b } \{x|a^\mathrm{T}x=b\} {x∣aTx=b}, $a\in \mathbb{R}$, $a\ne 0$, $b\in \mathbb{R}$, affine set

• halfspace: { x ∣ a T x ≤ b } \{x|a^\mathrm{T}x\leq b\} {x∣aTx≤b}, convex

• Euclidean ball: B ( x c , r ) = { x ∣ ∥ x − x c ∥ 2 ≤ r } = { x c + r u ∣ ∥ u ∥ 2 ≤ 1 } B(x_c,r)=\{x|\Vert x-x_c\Vert_2\leq r\}=\{x_c+ru|\Vert u\Vert_2\leq 1\} B(xc​,r)={x∣∥x−xc​∥2​≤r}={xc​+ru∣∥u∥2​≤1}, convex

  • ellipsoid: E = { x ∣ ( x − x c ) T P − 1 ( x − x c ) ≤ 1 } = { x c + A u ∣ ∥ u ∥ 2 ≤ 1 } \mathcal{E}=\{x|(x-x_c)^\mathrm{T}P^{-1}(x-x_c)\leq 1\}=\{x_c+Au|\Vert u\Vert_2\leq 1\} E={x∣(x−xc​)TP−1(x−xc​)≤1}={xc​+Au∣∥u∥2​≤1} , convex

• norm ball: { x ∣ ∥ x − x c ∥ ≤ r } \{x|\Vert x-x_c\Vert\leq r\} {x∣∥x−xc​∥≤r}, convex; norm cone C = { ( x , t ) ∣ ∥ x ∥ ≤ t } ⊆ R n + 1 C=\{(x,t)|\Vert x\Vert\leq t\}\subseteq\mathbb{R}^{n+1} C={(x,t)∣∥x∥≤t}⊆Rn+1, convex

  • second-order cone:
    C = { ( x , t ) ∈ R n + 1 ∣ ∥ x ∥ 2 ≤ t } = { [ x t ] ∣ [ x t ] T [ I 0 0 − 1 ] [ x t ] ≤ 0 , t ≥ 0 } \begin{aligned} C&=\{(x,t)\in\mathbb{R}^{n+1}|\Vert x\Vert_2\leq t\}\\ &=\left\{\begin{bmatrix} x \\ t \end{bmatrix}\left |\begin{bmatrix} x \\ t \end{bmatrix}^\mathrm{T}\begin{bmatrix} I & 0 \\ 0 &-1 \end{bmatrix}\begin{bmatrix} x \\ t \end{bmatrix}\leq 0,\quad t\geq 0\right.\right\} \end{aligned} C​={(x,t)∈Rn+1∣∥x∥2​≤t}={[xt​]∣∣∣∣∣​[xt​]T[I0​0−1​][xt​]≤0,t≥0}​

• polyhedra: P = { x ∣ a j T x ≤ b j ,    j = 1 , … , m ,    c j T x = d j ,    j = 1 , … , p } = { x ∣ A x ⪯ b , C x = d } \mathcal{P}=\{x|a_j^\mathrm{T}x\leq b_j,\;j=1,\ldots,m,\;c_j^\mathrm{T}x=d_j,\;j=1,\ldots,p\}=\{x|Ax\preceq b,Cx=d\} P={x∣ajT​x≤bj​,j=1,…,m,cjT​x=dj​,j=1,…,p}={x∣Ax⪯b,Cx=d}, convex

  • $k$-dimensional simplexes: C = c o n v { v 0 , … , v k } = { θ 0 v 0 + ⋯ + θ k v k ∣ θ ⪰ 0 , 1 T θ = 1 } C=\mathbf{conv}\{v_0,\ldots,v_k\}=\{\theta_0v_0+\cdots+\theta_kv_k|\theta\succeq0,\mathbf{1}^\mathrm{T}\theta=1\} C=conv{v0​,…,vk​}={θ0​v0​+⋯+θk​vk​∣θ⪰0,1Tθ=1}, $v_1-v_0,\dots ,v_k-v_0$ linearly independent

• positive semidefinite cone: S + n = { X ∈ S n ∣ X ⪰ 0 } \mathbf{S}_+^n=\{X\in\mathbf{S}^n|X\succeq 0\} S+n​={X∈Sn∣X⪰0}, the set of symmetric positive semidefinite matrices, where S n = { X ∈ R n × n ∣ X = X T } \mathbf{S}^n=\{X\in\mathbb{R}^{n\times n}|X=X^\mathrm{T}\} Sn={X∈Rn×n∣X=XT}.

2.3 Operations that preserve convexity

• intersection: if $S_1$ and $S_2$ are convex, then $S_1\bigcap S_2$ is convex
• affine functions: if $S\subseteq {\mathbb{R}}^n$ is convex, $f(x)=Ax+b$, then f ( S ) = { f ( x ) ∣ x ∈ S } f(S)=\{f(x)|x\in S\} f(S)={f(x)∣x∈S} and f − 1 ( S ) = { x ∣ f ( x ) ∈ S } f^{-1}(S)=\{x|f(x)\in S\} f−1(S)={x∣f(x)∈S} are convex
• perspective function: $P(z,t)=z/t$, $t>0$. If $C$ is convex, then P ( C ) = { P ( x ) ∣ x ∈ C } P(C)=\{P(x)|x\in C\} P(C)={P(x)∣x∈C} and P − 1 ( C ) = { ( x , t ) ∈ R n + 1 ∣ x / t ∈ C , t > 0 } P^{-1}(C)=\{(x,t)\in\mathbb{R}^{n+1}|x/t\in C, t>0\} P−1(C)={(x,t)∈Rn+1∣x/t∈C,t>0} are convex
• linear-fractional function: $f(x)=(Ax+b)/(c^{\mathrm{T}}x+d)$, $c^{\mathrm{T}}x+d>0$. If $C$ is convex, then $f(C)$ and $f^{-1}(C)$ are convex

2.4 Generalized inequalities

• proper cone: a cone $K\subseteq {\mathbb{R}}^n$ is convex, closed, solid and pointed.
  • $x{⪯}_{K}y\iff y-x\in K$; $x{\prec }_{K}y\iff y-x\in \mathbf{i}\mathbf{n}\mathbf{t}K$;
• minimum element: $S\subseteq x+K$; minimal element: ( x − K ) ⋂ S = { x } (x-K)\bigcap S=\{x\} (x−K)⋂S={x}

2.5 Separating and supporting hyperplanes

• separating hyperplane theorem: Suppose $C$ and $D$ are nonempty disjoint convex sets, i.e., $C\bigcap D=\varnothing$. Then there exist $a\ne 0$ and $b$ such that $a^{\mathrm{T}}x\le b$ for all $x\in C$ and $a^{\mathrm{T}}x\ge b$ for all $x\in D$.
• supporting hyperplane: $C\subseteq {\mathbb{R}}^n$, $x_0\in \mathbf{b}\mathbf{d}C=\mathbf{c}\mathbf{l}C\backslash \mathbf{i}\mathbf{n}\mathbf{t}C$, if $a\ne 0$ satisfies $a^{\mathrm{T}}x\le a^{\mathrm{T}}{x}_0$ for all $x\in C$, then the hyperplane { x ∣ a T x = a T x 0 } \{x|a^\mathrm{T}x=a^\mathrm{T}x_0\} {x∣aTx=aTx0​} is called a supporting hyperplane to $C$ at the point $x_0$.
• supporting hyperplane theorem: for any nonempty convex set $C$ and any $x_0\in \mathbf{b}\mathbf{d}C$, there exists a supporting hyperplane to $C$ at $x_0$.

2.6 Dual cones and generalized inequalities

• dual cone: Let $K$ be a cone, the set K ∗ = { y ∣ x T y ≥ 0    f o r    a l l    x ∈ K } K^*=\{y|x^\mathrm{T}y\geq 0\;{\rm for\;all}\;x\in K\} K∗={y∣xTy≥0forallx∈K} is called the dual cone of $K$.
• minimum element: $x$ is the minimum element of $S$, with respect to the generalized inequality ${⪯}_K$, if and only if for all $\lambda {\succ }_{K^{\ast }}0$, $x$ is the unique minimizer of ${\lambda }^{\mathrm{T}}z$ over $z\in S$.
• minimal element: if $\lambda {\succ }_{K^{\ast }}0$ and $x$ minimizes ${\lambda }^{\mathrm{T}}z$ over $z\in S$, then $x$ is minimal.
  • If $S$ is convex, for any minimal element $x$ there exists a nonzero $\lambda {⪰}_{K^{\ast }}0$ such that $x$ minimizes ${\lambda }^{\mathrm{T}}z$ over $z\in S$.