凸优化理论学习笔记

凸集的定义（Convex Set）

A set ( S ) in a vector space (like${\mathbb{R}}^n$), the space of n-dimensional real vectors) is called convex if, for any two points within the set, every point on the straight line segment that connects these two points also lies within the set. Mathematically, this can be expressed as:
for any $x,y\in S$ and any $0\le \lambda \le 1$,the point $z=\lambda x+(1-\lambda )y$ must also be in $S$.

凸约束（Convex Constraints）

• Convex Constraint: A constraint is convex if it defines a convex set. For inequality constraints $g(x)\le 0$, $g$ must be a convex function. For equality constraints $h(x)=0$, $h$ must be affine (i.e., $h(x)=Ax+b$ where $A$ is a matrix and $b$ is a vector).

凸函数（Convex Functions）

Determining if a function is convex is essential in various fields like optimization, economics, and machine learning. Convex functions have important properties that make optimization problems easier to solve because a local minimum of a convex function is also a global minimum. Here are several methods to determine if a function is convex:

1. Definition

A function $f:{\mathbb{R}}^n\to \mathbb{R}$ is convex on a convex set $D$ if for all $x,y\in D$ and for all $\lambda$ such that $0\le \lambda \le 1$, the following inequality holds:
$$f(\lambda x+(1-\lambda )y)\le \lambda f(x)+(1-\lambda )f(y)$$
This definition states that the line segment connecting any two points on the graph of the function lies above or on the graph itself.

2. First Derivative Test (for functions of one variable)

A function $f$ of one variable is convex on an interval if its first derivative $f^{\prime }$ is monotonically non-decreasing on that interval. This means as $x$ increases, $f^{\prime }(x)$ does not decrease.

3. Second Derivative Test

A function $f$ is convex on an interval if its second derivative $f^{\prime \prime }(x)$ is non-negative for all $x$ in that interval. This is a straightforward test because it only involves checking the sign of the second derivative:
$$f^{\prime \prime }(x)\ge 0\text{for all }x\in D$$

4. Hessian Matrix (for multivariable functions)

For functions of several variables, $f:{\mathbb{R}}^n\to \mathbb{R}$, the function is convex on a convex set if its Hessian matrix $H_f(x)$ is positive semidefinite（半定） for all $x$ in the domain of $f$. The Hessian matrix of $f$ at $x$ is given by:
$$H_f(x)=\left[\begin{array}{cccc}\frac{{\partial }^{2}f}{\partial x_1^2}& \frac{{\partial }^{2}f}{\partial x_1\partial x_2}& \cdots & \frac{{\partial }^{2}f}{\partial x_1\partial x_n}\\ \frac{{\partial }^{2}f}{\partial x_2\partial x_1}& \frac{{\partial }^{2}f}{\partial x_2^2}& \cdots & \frac{{\partial }^{2}f}{\partial x_2\partial x_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial x_n\partial x_1}& \frac{{\partial }^{2}f}{\partial x_n\partial x_2}& \cdots & \frac{{\partial }^{2}f}{\partial x_n^2}\end{array}\right]$$
A matrix is positive semidefinite if all its eigenvalues（特征值） are non-negative.

凸近似（Convex Approximation）

Convex approximation involves using a convex function to approximate a non-convex function.

1.Linearization (First-Order Taylor Expansion)

The simplest form of convex approximation is to perform a first-order Taylor expansion around a point $x_0$:
$f(x)\approx f(x_0)+f^{\prime }(x_0)(x-x_0)$
Here, $f^{\prime }(x_0)$ is the derivative of the function at $x_0$. This linear approximation is usually only valid within a very small neighborhood of $x_0$.

2. Second-Order Taylor Expansion

If the second-order derivative (Hessian matrix) is semi-definite at some condition, a second-order Taylor expansion can be used:
$f(x)\approx f(x_0)+f^{\prime }(x_0)(x-x_0)+\frac{1}{2}(x-x_0{)}^{T}H(x_0)(x-x_0)$
Where $H(x_0)$ is the Hessian matrix at $x_0$. If $H(x_0)$ is semi-definite, this approximation is convex.

3. Substitution and Relaxation

Sometimes, convex approximations can be created by substituting non-convex expressions or relaxing constraints. For instance, the absolute value function can be approximated by the square of the absolute value.