Convex Optimization 读书笔记 (5)

Chapter 6: Approximation and fitting

6.1 Norm approximation

6.1.1 Basic norm approximation problem

The simplest norm approximation problem is an unconstrained problem of the form
$$\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}||Ax-b||$$where $A\in {\mathbf{R}}^{m\times n},b\in {\mathbf{R}}^m,x\in {\mathbf{R}}^n,||\cdot ||$ is a norm on ${\mathbf{R}}^m$. A solution of the norm approximation problem is sometimes called an approximate solution of $Ax\approx b$, in the norm $\Vert \cdot \Vert$. The vector
$$r=Ax-b$$is called the residual for the problem; its components are sometimes called the individual residuals associated with $x$.

6.1.2 Penalty function approximation

In $l_p$-norm approximation, for $1\le p<\infty$, the objective is
$$(|r_1{|}^p+\cdots +|r_m{|}^p{)}^{\frac{1}{p}}$$As in least-squares problems, we can consider the equivalent problem with objective
$$|r_1{|}^p+\cdots +|r_m{|}^p$$The penalty function approximation problem has the form
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& \varphi (r_1)+\cdots +\varphi (r_m)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& r=Ax-b\end{array}$$where $\varphi :\mathbf{R}\to \mathbf{R}$ is called the (residual) penalty function. We assume that $\varphi$ is convex, so the penalty function approximation problem is a convex optimization problem.

deadzone-linear

$$\varphi (u)=\{\begin{array}{rlc}& 0& |u|\le a\\ & |u|-a& |u|>a\end{array}$$

log barrier

$$\varphi (u)=\{\begin{array}{rlc}& -a\log (1-(\frac{u}{a}{)}^2)& |u|\le a\\ & \infty & |u|>a\end{array}$$

The log barrier penalty puts weight very much like the $l_2$-norm penalty for small residuals, but puts very strong weight on residuals larger than around 0.8, and infinite weight on residuals larger than 1.

6.1.3 Approximation with constraints

It is possible to add constraints to the basic norm approximation problem. When these constraints are convex, the resulting problem is convex.

6.2 Least-norm problems

The basic least-norm problem has the form
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& ||x||\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Ax=b\end{array}$$where $A\in {\mathbf{R}}^{m\times n},b\in {\mathbf{R}}^m,x\in {\mathbf{R}}^n,||\cdot ||$ is a norm on ${\mathbf{R}}^m$.

6.3 Regularized approximation

6.3.1 Bi-criterion formulation

In the basic form of regularized approximation, the goal is to find a vector $x$ that is small (if possible), and also makes the residual $Ax-b$ small. This is naturally described as a (convex) vector optimization problem with two objectives, $\Vert Ax-b\Vert$ and $\Vert x\Vert$:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}(\mathrm{w}.\mathrm{r}.\mathrm{t}.{\mathbf{R}}_{+}^2)& (||Ax-b||,||x||)\end{array}$$
The two norms can be different: the first, used to measure the size of the residual, is on ${\mathbf{R}}^m$; the second, used to measure the size of $x$, is on ${\mathbf{R}}^n$.

6.3.2 Regularization

Regularization is a common scalarization method used to solve the bi-criterion problem. One form of regularization is to minimize the weighted sum of the objectives:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& ||Ax-b||+\gamma ||x||\end{array}$$where $\gamma >0$ is a problem parameter. As $\gamma$ varies over $(0,\infty )$, the solution of traces out the optimal trade-off curve.

Another common method of regularization, especially when the Euclidean norm is used, is to minimize the weighted sum of squared norms:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& ||Ax-b|{|}^2+\delta ||x|{|}^2\end{array}$$for a variety of values of $\delta >0$.

6.3.3 Reconstruction, smoothing, and de-noising

Quadratic smoothing

The simplest reconstruction method uses the quadratic smoothing function
$${\varphi }_{\mathrm{q}\mathrm{u}\mathrm{a}\mathrm{d}}(x)=\sum _{i=1}^{n-1}(x_{i-1}-x_i{)}^2$$
We can obtain the optimal trade-off between $\Vert \hat{x}-x_{\mathrm{c}\mathrm{o}\mathrm{r}}{\Vert }^2$ and $\Vert D\hat{x}{\Vert }^2$ by minimizing
$$||\hat{x}-x_{\mathrm{c}\mathrm{o}\mathrm{r}}|{|}_2^2+\delta ||D\hat{x}|{|}_2^2$$
where $D\in {\mathbf{R}}^{(n-1)\times n}$ is the bidiagonal matrix.

But any rapid variations in the original signal will, obviously, be attenuated or removed by quadratic smoothing.

Total variation reconstruction

The method is based on the smoothing function
$${\varphi }_{\mathrm{t}\mathrm{v}}(x)=\sum _{i=1}^{n-1}|x_{i-1}-x_i|$$

6.4 Robust approximation

6.4.1 Stochastic robust approximation

We assume that $A$ is a random variable taking values in ${\mathbf{R}}^{m\times n}$ with mean $\bar{A}$ so we can describe $A$ as with mean $\bar{A}$ as
$$A=\bar{A}+U$$where $U$ is a random matrix with zero mean. Then the problem is
$$\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}\mathbb{E}[||Ax-b|{|}_2^2]$$
We can express the objective as
$$\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}||\bar{A}x-b|{|}_2^2+||P^{\frac{1}{2}}x|{|}_2^2$$where $P=\mathbb{E}[U^{T}U]$. The solution is
$$x=({\bar{A}}^T\bar{A}+P{)}^{-1}{\bar{A}}^{T}b$$

6.4.2 Worst-case robust approximation

The (worst-case) robust approximation problem is to minimize the worst-case error:
m i n i m i z e      e w c ( x ) = sup ⁡ { ∣ ∣ A x − b ∣ ∣ ∣ A ∈ A } {\rm minimize} \ \ \ \ e_{\rm wc}(x)=\sup\{ ||Ax-b|| \mid A\in \mathcal{A} \} minimize    ewc​(x)=sup{∣∣Ax−b∣∣∣A∈A}where $\mathcal{A}$ is a set of all possible value. The robust approximation problem is always a convex optimization problem, but its tractability depends on the norm used and the description of the uncertainty set $\mathcal{A}$.

6.5 Function fitting and interpolation

6.5.1 Function families

We consider a family of functions $f_1,\cdots ,f_n:{\mathbf{R}}^k\to \mathbf{R}$, with common domain $\mathbf{d}\mathbf{o}\mathbf{m}{f}_i=D$. With each $x\in {\mathbf{R}}^n$ we associate the function $ f : \mathbf{R}^k → \mathbf{R} $ given by
$$f(u)=x_1{f}_1(u)+\cdots +x_n{f}_n(x)$$
The family { f 1 , ⋯   , f n } \{f_1, \cdots , f_n\} {f1​,⋯,fn​} is sometimes called the set of basis functions (for the fitting problem) even when the functions are not independent. The vector $x\in {\mathbf{R}}^n$, which parametrizes the subspace of functions, is our optimization variable, and is sometimes called the coefficient vector.

6.5.2 Constraints

Interpolation and inequalities

Interpolation conditions
$$f(v_j)=z_j,j=1,...m$$which require the function $f$ to have the values $z_j\in \mathbf{R}$ at specified points $v_j\in D$, form a set of linear equalities in $x$.

Derivative constrains

The gradient
$$\nabla f(v)=\sum _{i=1}^n{x}_i\nabla f_i(v)$$is a linear function of $x$, so interpolation conditions on the derivative of f at $v$ reduce to linear equality constraints on $x$:
$$||\nabla f(v)||=||\sum _{i=1}^n{x}_i\nabla f_i(v)||\le M$$
is a convex constrain on $x$.

Integral constrains

Any linear functional $\mathcal{L}$ on the subspace of functions can be expressed as a linear function of $x$:
$$\mathcal{L}=c^{T}x$$where $c_i={\int }_D\varphi (u)f_i(i)du$

6.5.3 Fitting and interpolation problems

6.5.4 Sparse descriptions and basis pursuit

In basis pursuit, there is a very large number of basis functions, and the goal is to find a good fit of the given data as a linear combination of a small number of the basis functions.

Thus we seek a function $f\in \mathcal{F}$ that fits the data well,
$$f(u_i)\approx y_i,i=1,...,m$$with a sparse coefficient vector $x$, i.e., $\mathbf{c}\mathbf{a}\mathbf{r}\mathbf{d}(x)$ small:
$$f=\sum _{i\in \mathcal{B}}{x}_i{f}_i$$

6.5.5 Interpolation with convex functions

If and only if there exist $g_1,...,g_m$ such that
$$y_j\ge y_i+g_i^T(u_j-u_i),i,j=1,...,m$$then there exsit a convex function $f:{\mathbf{R}}^k\to \mathbf{R}$ with $\mathbf{d}\mathbf{o}\mathbf{m}f={\mathbf{R}}^k$, that satisfies
$$f(u_i)=y_i,i=1,...,m$$
For example, $f(z)={\max }_{i=1,...,m}(y_i+g_i^T(z-u_i)).$