minf0(x)
s.t.fi(x)≤bi,i=1...m
Here
x∈Rn,fi:Rn→R,i=0...m
are convex function.
I.e.
fi(θx+(1−θ)y)≤θf(x)+(1−θ)f(y)
for all
θ∈[0,1]
and all
x,y∈domfi
where
domfi⊆Rn
is a convex set
where
C⊆Rn
is convex if and only if
θx+(1−θ)y∈C
for all
x,y∈C
, for all
θ∈[0,1]
- Fact: Set of
x
s
s.t.fi(x)≤b is a convex set. Proof: Take x,y,s.t.fi(x)≤bi,fi(y)≤bi and θ∈[0,1] ,
Then fi(θx+(1−θ)y)≤θfi(x)+(1−θ)fi(y)≤θbi+(1−θ)bi=biFact: The function f:R→R,f(x)=x2 is convex.
Proof: (θx+(1−θ)y)2=θ2+x2+2θ(1−θ)xy+(1−θ)2y2
=[θ+θ2−θ]x2+2θ(1−θ)xy+[(1−θ)+(1−θ)2−(1−θ)]y2
=θx2+(1−θ)y2+θ(θ−1)x2+2θ(1−θ)xy+(1−θ)(−θ)y2
=θf(x)+(1−θ)f(y)+θ(θ−1)(x−y)2
≥θf(x)+(1−θ)f(y)All linear function is convex function.
Fact: Let A∈Rm×n,b∈Rm,f:Rm be convex,
Then g:Rn defined by g(x)=f(Ax+b) is convex.Proof: Let x,y∈domg,θ∈[0,1]
Then A(θx+(1−θ)y)+b=θ(Ax+b)+(1−θ)(Ay+b)∈domf
Take x,y∈domg,θ∈[0,1]
Then g(θx+(1−θ)y)=f(θ(Ax+b)+(1−θ)(Ay+b))≤θf(Ax+b)+(1−θ)f(Ay+b)=θg(x)+(1−θ)g(y)Fact: The sum of two convex functions is convex.
- Note: Proof starts by nothing that if
f
,
g are convex functions, then dom(f+g)=domf∩domg is convex, because intersection of convex sets is convex. - Example: The function f:Rn→R defined by f(x)=∥Ax−b∥22 is convex.
- Proof: ∥Ax−b∥22=(aT1x−b1)2+...+(aTmx−bm)2 is a sum of convex functions.
Example: min∥Ax−b∥22 is a convex optimization problem. This kind of problem is known as a least-square problem.
Definition: Norms: A function f:Rn→R is a norm if and only if
f(x)≥0 for all x∈Rn non-negetivity
f(x)=0⟺x=0 definiteness
f(t⋅x)=∣t∣⋅∥x∥ for all t∈R,x∈Rn homogeneity
f(x+y)≤f(x)+f(y),∀x,y∈Rn triangle inequality- Example: For
p≥1
,
∥x∥p=(∣x1∣p+...+∣xn∣p)1p
is a norm
∥x∥∞=maxi=1...n∣xi∣
Geometric Picture: The set
B1=x∈Rn:∥x∥≤1
is bounded, convex, centrally symmetric at 0, nonempty interior.
(If
C⊆Rn
the interior
C0
of
C
is the set
- Definition: We say that sequence
(Zi)∞i=1
converges to
z
under the norm
∥⋅∥ if and only if for every ε>0 there exists N∈N,s.t.∥zi−z∥<ε for all i≥N Fact: In Rn , convergence happens in one norm if and only if it happens in any other norm.
Let ∥⋅∥ be a norm in Rn ,
Then ∃c1,c2>0 , such that c1∥x∥1<∥x∥<c2∥x∥1 for all x∈RnLemma 1: ∥x−y∥≥∥x∥−∥y∥ for any norm Triangle inequality
- Proof: ∥x∥=∥y+(x−y)∥≤∥y∥+∥x−y∥
- Lemma 2: The norm
∥⋅∥
is continous with respect to
∥⋅∥1
.
I.e., for any x∈Rn , for any ε>0 , there exists δ>0
s.t.∥x−y∥1<δ⟹∣∥x∥−∥y∥∣<ε Proof: ∣∥x∥−∥y∥∣≤∣∥x−y∥∣
and ∥x−y∥=∥∑ni=1ei→(xi−yi)∥
≤∑ni=1∥ei→(xi−yi)∥=∑ni=1∣xi−yi∣∥ei→∥
≤maxi(∥ei→∥)⋅∑ni=1∣xi−yi∣=maxi(∥ei→∥)⋅∥x−y∥1
where e1→,...,en→ is the standard basis
so if δ=εmaxi(∥ei→∥)
then ∣x−y∣1<δ⟹∣∥x∥−∥y∥∣<ϵ
∥⋅∥ attains its minimum and maximum on the set {x:∥x∥1=1} which is closed and bounded.
Let xmin be such that ∥xmin∥≤∥x∥ for all x∈{x:∥x∥1≤1} ,
let xmax be such that ∥xmax∥≥∥x∥ for all x∈{x:∥x∥1≤1}
Let c1=∥xmin∥≠0,c2=∥xmax∥ then for arbitrary x,x≠0 ,
∥x∥=∥x∥x∥1⋅∥x∥1∥=∥x∥1⋅∥x∥x∥1∥≤∥x∥1⋅∥xmax∥=c2⋅∥x∥1
Similarity, ∥x∥≥c1∥x∥1Definition: A matrix A∈Rn×n is positive semidefinite, if xTAx≥0,∀x∈R , and A is symmetric.
- Fact: Let
A symmetric, A=PTDP where P is orthogonal,D diagonal,
Then A is positive semidefinite iffD≥0 . - Proof: Say
D=diag(λ1,...,λn),λi<0
Let x=PTei→ , Then xTAx=(PTei→)TPTDPPTei→=ei→TDei→=λi<0
If D≥0 , then xTAx=(xTPT)D(Px)≥0