Convex Optimization 读书笔记 (1)

Chapter2 Convex Set

2.1 Affine, Convex Set

2.1.1 Line, Line Segment

Suppose $x_1,x_2$ are two points in ${\mathbf{R}}^n$,

$y=\theta x_1+(1-\theta )x_2,\theta \in \mathbf{R}$

is a line. If $\theta \in [0,1]$, $y$ is a line segment.

2.1.2 Affine Set

A set $C\subseteq {\mathbf{R}}^n$ is affine if for any $x_1,x_2\in C,\theta \in \mathbf{R}$, $\theta x_1+(1-\theta )x_2\in C$.

Affine combination means for $x_1,x_2,\dots ,x_k$ and ${\sum }_i{\theta }_i=1$, ${\sum }_i^k{\theta }_i{x}_i\in C$. If $x_i\in C$, then its affine combination $\in C$.

A affine set $C$ can be expressed as C = V + x 0 = { v + x 0 ∣ v ∈ V } C=V+x_0=\{v+x_0|v\in V\} C=V+x0​={v+x0​∣v∈V} ( a subspace plus an offset). The dimension of an affine set is the dimension of subspace $V=C-x_0,x_0\in C$.

Affine hull of a set $C$ is the set of all affine combination of $x_i\in C$, denoted as
a f f   C = { ∑ i k θ i x i ∣ x i ∈ C , ∑ i θ i = 1 } {\rm \mathbf{aff}}\space C=\{\sum_i^k{\theta_ix_i}|x_i \in C, \sum_i\theta_i=1\} aff C={i∑k​θi​xi​∣xi​∈C,i∑​θi​=1}
affine hull is the smallest affine set that contains $C$, which is if set $S$ is a affine set that contains $C$, the $\mathbf{a}\mathbf{f}\mathbf{f}C\subseteq S$.

2.1.3 Affine dimension, relative interior

Affine dimension of a set $C$ is the dimension of its affine hul,

Relative interior is
r e l i n t   C = { x ∈ C ∣ B ( x , r ) ∩ a f f   C ⊆ C   f o r   s o m e   r > 0 } \mathbf{relint}\space C=\{x\in C\mid B(x,r)\cap\mathbf{aff}\space C\subseteq {\rm C \space for\space some \space} r>0\} relint C={x∈C∣B(x,r)∩aff C⊆C for some r>0}
where B ( x , r ) = { y ∣ ∣ ∣ y − x ∣ ∣ ≤ r } B(x,r)=\{ y\mid ||{y-x}|| \leq r \} B(x,r)={y∣∣∣y−x∣∣≤r}

2.1.4 Convex sets

Set $C$ is convex if for $x_1,x_2\in C,\theta \in [0,1]$, we have $\theta x_1+(1-\theta )x_2\in C$.

A convex combination means the form ${\sum }_i^k{\theta }_i{x}_i$, where ${\sum }_i{\theta }_i=1,{\theta }_i>0$.

Convex hull of a set $C$ is the set of all convex combination of $x_i\in C$, denoted as
c o n v   C = { ∑ i k θ i x i ∣ x i ∈ C , ∑ i θ i = 1 , θ i > 0 } {\rm \mathbf{conv}}\space C=\{\sum_i^k{\theta_ix_i}|x_i \in C, \sum_i\theta_i=1,\theta_i>0\} conv C={i∑k​θi​xi​∣xi​∈C,i∑​θi​=1,θi​>0}
Convex hull is the smallest convex set that contains $C$, which is if set $S$ is a convex set that contains $C$, the $\mathbf{c}\mathbf{o}\mathbf{n}\mathbf{v}C\subseteq S$.

2.1.5 Cones

A set $C$ is a cone if for $x\in C,\theta >0$, we have $\theta x\in C$. A set is convex cone if for $x_1,x_2\in C;{\theta }_1,{\theta }_2\ge 0$, we have ${\theta }_1{x}_1+{\theta }_2{x}_2\in C$.

A conic combination of $x_i\in C$ is ${\sum }_i{\theta }_i{x}_i$ with ${\theta }_i\ge 0$.

2.2 Important examples

2.2.1 hyperplane, halfspaces

A hyperplane is a set of form
{ x ∣ a T x = b } \{x\mid a^Tx=b\} {x∣aTx=b}
where $a\in {\mathbf{R}}^n,a\ne 0,b\in \mathbf{R}$. It can be represented as
{ x ∣ a T ( x − x 0 ) = 0 } = x 0 + a ⊥ \{ x \mid a^T(x-x_0) =0 \}=x_0 + a^{\bot} {x∣aT(x−x0​)=0}=x0​+a⊥
Where $a^{\perp }$ means { x ∣ a T x = 0 } \{ x\mid a^Tx =0 \} {x∣aTx=0}. Shown as Fig 2.6.

A hyperplane devide space into 2 halfspaces:
{ x ∣ a T x ≤ b } \{ x\mid a^Tx\leq b\} {x∣aTx≤b}
or
{ x ∣ a T ( x − x 0 ) ≤ 0 } \{ x\mid a^T(x-x_0)\leq 0\} {x∣aT(x−x0​)≤0}

2.2.2 Euclidean balls, ellipsoids

A Euclidean balls in ${\mathbf{R}}^n$ is
B ( x c , r ) = { x ∣ ∣ ∣ x − x c ∣ ∣ 2 ≤ r } = { x ∣ ( x − x c ) T ( x − x c ) ≤ r 2 } = { x c + r u ∣ ∣ ∣ u ∣ ∣ 2 ≤ 1 } B(x_c,r)=\{ x \mid ||x-x_c||_2\leq r\}=\{ x \mid (x-x_c)^T(x-x_c)\leq r^2\}=\{x_c+ru\mid ||u||_2\leq1\} B(xc​,r)={x∣∣∣x−xc​∣∣2​≤r}={x∣(x−xc​)T(x−xc​)≤r2}={xc​+ru∣∣∣u∣∣2​≤1}
where $r>0,||u|{|}_2=(u^{T}u{)}^{\frac{1}{2}}$.

A Ellipsoid is
E = { x ∣ ( x − x c ) T P − 1 ( x − x c ) ≤ 1 } = { x c + A u ∣ ∣ ∣ u ∣ ∣ 2 ≤ 1 } \mathcal{E}=\{ x|(x-x_c)^TP^{-1}(x-x_c)\leq 1 \}=\{ x_c+Au\mid ||u||_2\leq1 \} E={x∣(x−xc​)TP−1(x−xc​)≤1}={xc​+Au∣∣∣u∣∣2​≤1}
where $P=P^T,P\succ 0,A$ square and nonsingular.

2.2.3 Norm ball, Norm cone

A norm ball is { x ∣ ∣ ∣ x − x c ∣ ∣ ≤ r } \{ x\mid ||x-x_c|| \leq r \} {x∣∣∣x−xc​∣∣≤r}, where $||\cdot ||$ is any norm.

A norm cone is { ( x , t ) ∣ ∣ ∣ x ∣ ∣ ≤ t } ⊆ R n + 1 \{ (x,t) \mid ||x|| \leq t \} \subseteq\mathbf{R}^{n+1} {(x,t)∣∣∣x∣∣≤t}⊆Rn+1, where $||\cdot ||$ is any norm.

2.2.4 Polyhedra

A polyhedron is thus the intersection of a finite number of halfspaces and hyperplanes:
P = { a j T x ≤ b j , j = 1 , ⋯   , m ; c i T x = d i , i = 1 , ⋯   , p } \mathcal{P}=\{ a^T_jx\leq b_j,j=1,\cdots,m;c^T_ix =d_i, i=1,\cdots,p \} P={ajT​x≤bj​,j=1,⋯,m;ciT​x=di​,i=1,⋯,p}
It can be represented as compact form:
P = { x ∣ A T x ⪯ b ; C T x = d } \mathcal{P}=\{ x\mid A^Tx\preceq b;C^Tx=d \} P={x∣ATx⪯b;CTx=d}
where $A=[a_1^T,\cdots ,a_m^T],C=[c_1^T,\cdots ,c_p^T]$.

Simplexes are special polyhedron:
C = c o n v   { v 0 , ⋯   , v k } = { ∑ i = 0 k θ i v i ∣ θ ⪰ 0 ; 1 θ = 1 } C={\mathbf{conv} \space }\{ v_0,\cdots,v_k \}=\{ \sum_{i=0}^k\theta_iv_i \mid \theta \succeq0;\mathbf{1}\theta=1 \} C=conv {v0​,⋯,vk​}={i=0∑k​θi​vi​∣θ⪰0;1θ=1}
where $v_i-x_0$ are linearly independent.

A polyhedron can be expressed in the form of convex hull add conic hull:
{ ∑ i k θ i x i ∣ ∑ i m θ i = 1 ; θ i > 0 } \{ \sum_i^{k}\theta_ix_i\mid\sum_i^m\theta_i=1;\theta_i>0 \} {i∑k​θi​xi​∣i∑m​θi​=1;θi​>0}
where $m<k$.

2.2.5 positive semidefinite cone

S n = { X ∈ R n × n ∣ X = X T } S + n = { X ∈ S n ∣ X ⪰ 0 } S + + n = { X ∈ S + n ∣ X ≻ 0 } \begin{aligned} \mathbf{S}^n &= \{ X\in \mathbf{R}^{n\times n} \mid X=X^T \} \\ \mathbf{S}^n_+ &= \{ X\in \mathbf{S}^{n} \mid X\succeq0 \} \\ \mathbf{S}^n_{++} &= \{ X\in \mathbf{S}^{n}_+ \mid X\succ0\} \end{aligned} SnS+n​S++n​​={X∈Rn×n∣X=XT}={X∈Sn∣X⪰0}={X∈S+n​∣X≻0}​

${\mathbf{S}}_{+}^n$ is a convex cone.

2.3 Operations that preserve convexity

2.3.1 Intersection

If $S_1$ and $S_2$ are convex, then $S_1\cap S_2$ is convex. It holds for infinite intersections:
$$S_{\alpha }\in S\Rightarrow \bigcap S_{\alpha }\in S$$

2.3.2 Affine

Let the function $f:{\mathbf{R}}^n\to {\mathbf{R}}^m$ is affine, $S$ is convex set, then { f ( x ) ∣ x ∈ S } \{ f(x)\mid x\in S \} {f(x)∣x∈S} is convex.

2.3.3 Linear fractional, perspective functions

Define perspective function $P:{\mathbf{R}}^{n+1}\to {\mathbf{R}}^n$, with $\mathbf{d}\mathbf{o}\mathbf{m}P={\mathbf{R}}^n\times {\mathbf{R}}_{++},P(x,t)=\frac{x}{t}$.

A linear-fractional function is formed by composing the perspective function with an affine function. Suppose $g(x)=\left[\begin{array}{c}A\\ c^T\end{array}\right]x+\left[\begin{array}{c}b\\ d\end{array}\right]$, where $A\in {\mathbf{R}}^{m\times n},c\in {\mathbf{R}}^n,b\in {\mathbf{R}}^m,d\in \mathbf{R}$. Then $f$ is $P\circ f$:
f = A x + b c T x + d ,   d o m f = { x ∣ c T x + d > 0 } f=\frac{Ax+b}{c^Tx+d}, \space \mathbf{dom}f = \{ x| c^Tx+d>0\} f=cTx+dAx+b​, domf={x∣cTx+d>0}
it can be represented as $f(x)={\mathcal{P}}^{-1}(x)(Q\mathcal{P}(x))$, where P ( x ) = { t ( x , 1 ) ∣ t > 0 } \mathcal{P}(x)=\{ t(x,1) \mid t>0 \} P(x)={t(x,1)∣t>0}.

2.4 Generalized inequalities

2.4.1 Proper cones, generalized inequalities

A proper cone is a cone convex, close, solid and pointed.

Generalized inequalities $y{⪯}_{K}x$ means that $x-y\in K$. When $K={\mathbf{R}}_{+}^n$, we drop symbol $K$. So as $K={\mathbf{S}}_{+}^n$.

2.4.2 Minimum, minimal elements

$x\in S$ is the minimum element if for $y\in S$, then $x{⪯}_{K}y$.

$x\in S$ is the minimal element if $y\in S,x{⪯}_{K}y$ only if $y=x$.

2.5 Separating and supporting hyperplanes

2.5.1 Separating hyperplane theorem

Separating hyperplane theorem: Suppose $C$ and $D$ are nonempty convex sets and $C\cap D=\varnothing$. Then there exist $a\ne 0$ and $b$ such that. $a^{T}x\le b$ for all $x\in C$ and $a^{T}x\ge b$ for all $x\in D$.

The strict seperation is ${\theta }^{T}x<b$ for all $x\in C$ and $a^{T}x>b$ for all $x\in D$.

2.5.2 Supporting hyperplane

Suppose $C\subseteq R_n$, and $x0$ is a point in its boundary $\mathbf{b}\mathbf{d}C$,
$$x_0\in \mathbf{b}\mathbf{d}C=\mathbf{c}\mathbf{l}C\backslash \mathbf{i}\mathbf{n}\mathbf{t}C$$
If. $a\ne 0$ satisfies $a^{T}x\le a^T{x}_0$ for all $x\in C$,then the hyperplane { x ∣ a T x = a T x 0 } \{x|a^Tx=a^Tx_0\} {x∣aTx=aTx0​} is called a supporting hyperplane to $C$ at the point $x_0$.

2.6 Dual cones, generalized inequalities

2.6.1 Dual cones

If $K$ is a cone, then K ∗ = { y ∣ x T y ≥ 0 , x ∈ K } K^*=\{ y\mid x^Ty\geq0,x\in K \} K∗={y∣xTy≥0,x∈K} is the dual cone of $K$, it is always convex.

2.6.2 Dual generalized inequalities

Dual generalized inequalities is the one of $K^{\ast }$.