∥y∥∗=sup{xTy:∥x∥≤1}⟹xTy≤∥x∥⋅∥y∥∗
(because
xT∥x∥y≤∥y∥∗
)
Want inequality of type:
xTy≤f(x)+"f∗(y)"
for “general”
f
(Fenchel’s Inequality)
- Definition: For
f:Rn→R , the conjugate f∗ of f is defined byf∗(y)=supx(xTy−f(x))
with domf∗= set of y ’s for whichsup is <∞ .Example:
-
f(x)=aTx+b(x∈Rn)
f∗(y)=supxxTy−aTx−b={∞−bif y≠aif y=a -
f(x)=−logx(x>0)
(xy+logx)′=y+1x=0⟹x=−1y
f∗(y)=supx>0xTy+logx={∞−log(−y)−1if y≥0if y<0 -
f(x)=ex(x∈R)
(xy−ex)′=y−ex=0⟹x=logy
f∗(y)=supxxTy−ex={∞ylogy−yif y<0if y≥0 -
f(x)=xlogx(x≥0)
(xy−xlogx)′=y−logx−1=0⟹x=ey−1
f∗(y)=supx≥0xTy−xlogx=yey−1−(y−1)ey−1=ey−1 -
f(x)=12xTQx
with
Q∈Sn++
f∗(y)=supxxTy−12xTQx=yTQ−1y−12yTQ−1y=12yTQ−1y
( infxxTAx+xTb⟹bestx=−12A−1b )
So x=Q−1y
⟹xTy≤12xTQx+12yTQ−1y , for all Q≻0 -
f(x)=log(∑ni=1exi)
f∗(y)=supxxTy−log(∑ni=1exi)
(xy−log(∑ni=1exi))′=y−exi∑ni=1exi=0
⟹yi=exi∑ni=1exi,y⪰0,1Ty=1
assume for simplicity, y≻0
put xi=log(yi) , then ∑exi=1Ty=1 and optimality conditions hold
then f∗(y)=∑ni=1yilog(yi)−log(1Ty)=∑ni=1yilog(yi) -
f(x)=∥x∥
f∗(y)=supxxTy−∥x∥={0∞if ∥y∥∗≤1if ∥y∥∗>1
xTy−∥x∥≤∥x∥⋅∥y∥∗−∥x∥=∥x∥(∥y∥∗−1)≤0 if ∥y∥∗−1≤0 -
f(x)=12∥x∥2
f∗(y)=supxxTy−12∥x∥2=12∥y∥2∗
xTy−12∥x∥2≤∥x∥⋅∥y∥∗−12∥x∥2≤12∥y∥2∗ ( ∥x∥=∥y∥∗ )
⟹xTy≤12∥x∥2+12∥y∥2∗
-
f(x)=aTx+b(x∈Rn)
Proof of general hyperplane seperation:
Let C⊆Rn be a convex set, H⊆R be the affine subspace of smallest dimention containing C , we writeCε={x:Bε(x)⋂H⊆C}
then Cε⊆"relint(C)"=⋃ε>0Cε . (relint: relative interior)
( C⊆relint(C)¯¯¯¯¯¯¯¯¯¯¯¯¯ , C is a subset of closure ofrelint(C) )
Let C,D be disjoint convex sets. Then for every ε>0 the sets Aε=Cε¯¯¯¯⋂B1ε(0) , D¯¯¯ are closed disjoint convex sets with Cε¯¯¯¯⋂B1ε(0) bounded, and dist(Aε,D¯¯¯)≥ε>0 .
So ∃Aε∈Rn , aε≠0 , bε∈R s.t. (aε,bε) define a seperating hyperplane for Aε,D¯¯¯ .
aTεx≤bε∀x∈Aε , aTεx≥bε∀x∈D¯¯¯
WLOG ∥aε∥=1
The sequence (a⃗ 1n)∞n=1 is a sequence of unit vectors and so has a convergent subsequence, say WLOG convergent to a0∈Rn .
can assume sequence b1n is bonded (or else one of the sets C,D is empty)
and so also convergent to some value b0∈R .
Want to show (a0,b0) is SH for C,D , i.e., that
aT0x≤b0∀x∈C,aT0x≥b0∀x∈D
(Assume C is not a point, proof like above; then assume D is not a point, switchC,D .
If C,D are points, obious true.)Log-convexity and log-concavity
- Definition: f:Rn→R>0 is log-convex (log-concave) if log(f) is convex (concave).
- Convexity:
log(f(θx+(1−θ)y))≤θlog(f(x))+(1−θ)log(f(y))=log(f(x)θf(y)1−θ)
⟺f(θx+(1−θ)y)≤f(x)θf(y)1−θ
- Remark 2: log-convex ⟹ convex, f(x)=elogf(x) , (composition function, QED)
concave ⟹ log-concave