Introduction to Convex Optimization Basic Concepts 详细

往期文章链接目录

Optimization problem

All optimization problems can be written as:

Optimization Categories

1. convex v.s. non-convex
   Deep Neural Network is non-convex

2. continuous v.s.discrete
   Most are continuous variable; tree structure is discrete

3. constrained v.s. non-constrained
   We add prior to make it a constrained problem

4. smooth v.s.non-smooth
   Most are smooth optimization

Different initialization brings different optimum (if not convex)

Idea: Give up global optimal and find a good local optimal.

• Purpose of pre-training: Find a good initialization to start training, and then find a better local optimal.

• Relaxation: Convert to a convex optimization problem.

• Brute force: If a problem is small, we can use brute force.

Affine sets

A set $C\subseteq {\mathbf{R}}^n$ is affine if the line through any two distinct points in $C$ lies in $C$, i.e., if for any $x1$, $x2\in C$ and $\theta \in \mathbf{R}$, we have $$\theta x_1+(1-\theta )x_2\in C.$$

Note: The line passing throught $x_1$ and $x_2$: $y=\theta x_1+(1-\theta )x_2$.

Affine combination

We refer to a point of the form ${\theta }_1{x}_1+{\theta }_2{x}_2+...+{\theta }_k{x}_k$, where ${\theta }_1+{\theta }_2+...+{\theta }_k=1$ as an affine combination of the points $x_1,x_2,...,x_k$. An affine set contains every affine combination of its points.

Affine hull

The set of all affine combinations of points in some set $C\subseteq {\mathbf{R}}^n$ is called the affine hull of $C$, and denoted $\mathbf{a}\mathbf{f}\mathbf{f}C$:

a f f   C = { θ 1 x 1 + θ 2 x 2 + . . . + θ k x k   ∣ x 1 , x 2 , . . . , x k ∈ C , θ 1 + θ 2 + . . . + θ k = 1 } . \mathbf{aff}\, C =\{\theta_1 x_1 + \theta_2 x_2 + ... + \theta_k x_k \, | x_1, x_2, ..., x_k \in C, \theta_1 + \theta_2 + ... + \theta_k = 1\}. affC={ θ1​x1​+θ2​x2​+...+θk​xk​∣x1​,x2​,...,xk​∈C,θ1​+θ2​+...+θk​=1}.

The affine hull is the smallest affine set that contains $C$, in the following sense: if
$S$ is any affine set with $C\subseteq S$, then $\mathrm{aff}C\subseteq S$.

Affine dimension: We define the affine dimension of a set $C$ as the dimension of its affine hull.

Convex Sets

A set $C$ is convex if the line segment between any two points in C