## Life's Joy & Comfortable

http://lijiancheng0614.github.io/

# [笔记] Convex Optimization 2015.11.18

Let {fa:aA}$\{ f_a : a \in \mathcal{A} \}$ be a collection of convex functinos from Rn$\mathbb{R}^n$ to R$\mathbb{R}$, with same domain, then f(x)=supaAfa(x)$f(x) = \sup_{a \in \mathcal{A}} f_a(x)$ is a convex function.

• Proof1: Take x,ydomf$x, y \in dom \, f$, θ[0,1]$\theta \in [0, 1]$,
f(θx+(1θ)y)==supafa(θx+(1θ)y)supa[θfa(x)+(1θ)fa(y)]θsupafa(x)+(1θ)supafa(y)θf(x)+(1θ)f(y)

(The proposition is true for domf=aAdomfa$dom \, f = \bigcap _{a \in \mathcal{A}} dom \, f_a$, but false for domf=aAdomfa$dom \, f = \bigcup _{a \in \mathcal{A}} dom \, f_a$.)

• Proof2:

epi(f)==={(x,t):xdomf,t>fa(x)aA}{(x,t):(x,t)epi(fa)aA}aAepi(fa)

• Example: Let x[i]$x_{[i]}$ denote the i$i$-th largest component of x=(x1,,xn)Rn$x = (x_1, \cdots, x_n) \in \mathbb{R}^n$,
then maxsumr(x)=x[1]+x[2]++x[r]$max-sum_r(x) = x_{[1]} + x_{[2]} + \cdots + x_{[r]}$ is convex.

• Proof: maxsumr(x)xi1+xi2++xir$max-sum_r(x) \ge x_{i_1} + x_{i_2} + \cdots + x_{i_r}$
for any {i1,,ir}{1,,n}$\{ i_1, \cdots, i_r \} \subseteq \{ 1, \cdots, n \}$ with ijik$i_j \neq i_k$ for jk$j \neq k$
so it is convex.

• Example: Let CRn$C \subseteq \mathbb{R}^n$, define Sc(x)=sup{yTx:yC}$S_c(x) = \sup \{ y^T x : y \in C \}$, then Sc$S_c$ is convex.

• Example: Let f:SnR$f : S^n \to \mathbb{R}$, f(X)$f(X)$ is the largest eigenvalue of X$X$.
Claim: f$f$ is convex.

• Proof: First claim that f(X)=sup{yTXy:y2=1}$f(X) = \sup \{ y^T X y : \lVert y \rVert _2 = 1 \}$
• Proof of claim:

supy2=1yTXy====supy2=1yTPDPTysupv2=1vTDvsupv2=1λiv2isupv2=1max(λi)i=1nv2imax(λi)

• Example: Let f:Rm×nR$f : \mathbb{R}^{m \times n} \to \mathbb{R}$ be defined by f(X)=X2$f(X) = \lVert X \rVert _2$ where X2=supy2=1Xy2$\lVert X \rVert _2 = \sup _{\lVert y \rVert _2 = 1} \lVert X y \rVert _2$ is the spectural norm of XRm×n$X \in \mathbb{R}^{m \times n}$.

• Claim: f(X)=supu,v{uTXv:u2=1,v2=1}$f(X) = \sup _{u, v} \{ u^T X v : \lVert u \rVert _2 = 1, \lVert v \rVert _2 = 1 \}$
because Xv2=sup{uTXv:uRn,u2=1}$\lVert Xv \rVert _2 = \sup \{ u^T X v : u \in \mathbb{R}^n, \, \lVert u \rVert _2 = 1 \}$
more generally: Xa,b=sup{Xvb:va=1}=sup{uTXv:va=1,ub=1}$\lVert X \rVert _{a, b} = \sup \{ \lVert Xv \rVert _b : \lVert v \rVert _a = 1 \} = \sup \{ u^T X v : \lVert v \rVert _a = 1, \lVert u \rVert _{b*} = 1 \}$
(Xvb=Xvb=supu{uTXv:ub=1}$\lVert Xv \rVert _b = \lVert Xv \rVert _{b**} = \sup _u \{ u^T X v : \lVert u \rVert _{b*} = 1 \}$)

• Composition: Have f(x)=h(g(x))$f(x) = h(g(x))$, xRn$x \in \mathbb{R}^n$, g(x)R$g(x) \in \mathbb{R}$, when is f$f$ convex?

• Would-be-proof: Take x,ydomf$x, y \in dom \, f$, θ[0,1]$\theta \in [0, 1]$ then
f(θx+(1θ)y)==h(g(θx+(1θ)y))h(θg(x)+(1θ)g(y))θh(g(x))+(1θ)h(g(y))θf(x)+(1θ)f(y)use domf is convexuse g is convex/concave, h is nondecreasing/nonincreasinguse h is convex
- - 1 2 3 4
Condition g convex concave convex concave
- h nondecreasing nonincreasing nonincreasing nondecreasing
- h convex convex concave concave
Result f convex convex concave concave

- Example: g(x)=x21,h(x)=x3/2,domh=R+$g(x) = x^2 - 1, h(x) = x^{3 / 2}, dom \, h = \mathbb{R}_+$
then dom(hg)=(,1][1,+]$dom \, (h \circ g) = (-\infty, -1] \bigcup [1, +\infty]$ is not convex.

• Example 3.13:
If g$g$ is convex then eg(x)$e^{g(x)}$ is convex. 1
If g$g$ is concave and positive then log(g(x))$\log(g(x))$ is concave. 4
If g$g$ is concave and positive then 1g(x)$\frac{1}{g(x)}$ is convex. 2
If g$g$ is convex and nonnegative and p1$p \ge 1$, g(x)p$g(x)^p$ is convex. 1
g:RnRm$g : \mathbb{R}^n \to \mathbb{R}^m$, say g$g$ is K-convex, where K$K$ is a cone in Rm$\mathbb{R}^m$,
if domg$dom \, g$ is convex and g(θx+(1θ)y)Kθg(x)+(1θ)g(t)$g(\theta x + (1 - \theta)y) \preceq _K \theta g(x) + (1 - \theta) g(t)$
xKyh(x)h(y)$x \succeq _K y \implies h(x) \ge h(y)$ K-nondecreasing.

• Example 3.14:
h(z)=log(ki=1ezi)$h(z) = \log \left(\sum _{i = 1}^k e^{z_i} \right)$, so log(ki=1egi(x))$\log \left( \sum _{i = 1}^k e^{g_i(x)} \right)$ will be convex if g1,,gk$g_1, \cdots, g_k$ are convex.

• Minimization: Let f:Rn×Rm$f : \mathbb{R}^n \times \mathbb{R}^m$ be a convex function, then
g(x)=infy:(x,y)domff(x,y)$g(x) = \inf _{y : (x, y) \in dom \, f} f(x, y)$ is convex.

• Proof: Let x1,x2domg$x_1, x_2 \in dom \, g$, θ[0,1]$\theta \in [0, 1]$.
then for any ε>0$\varepsilon \gt 0$, y1,y2$\exists y_1, y_2$, s.t.
g(x1)f(x1,y1)ε$g(x_1) \ge f(x_1, y_1) - \varepsilon$, g(x2)f(x2,y2)ε$g(x_2) \ge f(x_2, y_2) - \varepsilon$and

g(θx1+(1θ)x2)=infyf(θx1+(1θ)x2,y)f(θx1+(1θ)x2,θy1+(1θ)y2)θf(x1,y1)+(1θ)f(x2,y2)θg(x1)+(1θ)g(x2)+ε

• Example: Let CRn$C \subseteq \mathbb{R}^n$ be a convex set, then
g(x)=infyCxy$g(x) = \inf _{y \in C} \lVert x - y \rVert$ is a convex function.

• Proof: Use f(x,y)=xydomf=Rn×C$f(x, y) = \lVert x - y \rVert \; dom \, f = \mathbb{R}^n \times C$,
f(θx1+(1θ)x2,θy+(1θ)y2)===θx1+(1θ)x2θy1(1θ)y2θ(x1y1)+(1θ)(x2y2)θx1y1+(1θ)x2y2f(x1,y1)+f(x2,y2)

Consider function g(w)=infxmi=1wi(aTixbi)2$g(w) = \inf _x \sum _{i = 1}^m w_i (a_i^T x - b_i)^2$, “weighted least square”
concave function of w$w$
Let g(w)=infx(Axb)TW(Axb)=infx(xTATWAx2bTWAx+bTWb)$g(w) = \inf _x (Ax - b)^T W (Ax - b) = \inf _x (x^T A^T W A x - 2b^T W A x + b^T W b)$
assume ATWA0$A^T W A \succ 0$, then optimal x=(ATWA)1ATWb$x = (A^T W A)^{-1} A^T W b$
(minx(xTAx+2bTx),=2Ax+2b=0bestx=A1b$\min _x(x^T A x + 2b^T x), \nabla = 2Ax + 2b = 0 \implies \text{best} \, x = -A^{-1} b$)

optimal value====bTWA(ATWA)1ATWb2bTWA(ATWA)1ATWb+bTWbbTWA(ATWA)1ATWb+bTWbi=1mb2iwi(i=1mwibiaTi)(i=1mwiaiaTi)1(i=1mwibiai)i=1mb2iwii,jwiwjbibjaTi(i=1mwiaiaTi)1aj

(ATWA=ai(wiaTi)=wiaiaTi$A^T W A = \sum a_i (w_i a_i^T) = \sum w_i a_i a_i^T$)

11-09 649

11-30 123

01-31 9684

04-18 5174

#### 凸优化convex optimization 中文电子版

2017年01月23日 47.21MB 下载

#### Convex Optimization_Boyd_英文版_凸优化_王会宁译_中文版

2016年03月10日 59.27MB 下载

#### Introductory lectures on convex optimization

2013年04月23日 8.93MB 下载

#### Convex_Optimization.pdf

2017年09月13日 5.62MB 下载

#### Convex Optimization - Algorithms and Complexity

2018年02月26日 1.11MB 下载

#### DIMITRI BERTSEKAS_Convex Optimization Theory_solutions

2018年02月03日 4.4MB 下载