Convex Optimization 读书笔记 (3)

4 Convex optimization problems

4.1 Optimization problems

4.1.1 Basic terminology

We use the notation
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$
to describe the problem of finding an $x$ that minimizes $f_0(x)$ among all $x$ that satisfy the conditions $f_i(x)\le 0,i=1,...,m$, and $h_i(x)=0,i=1,...,p$. We call $x\in {\mathbf{R}}^n$ the optimization variable and the function $f_0:{\mathbf{R}}^n\to \mathbf{R}$ the objective function or cost function. The inequalities $f_i(x)\le 0$ are called inequality constraints, and the corresponding functions $f_i:{\mathbf{R}}^n\to \mathbf{R}$ are called the inequality constraint functions. The equations $h_i(x)=0$ are called the equality constraints, and the functions $h_i:{\mathbf{R}}^n\to \mathbf{R}$ are the equality constraint functions.

Optimal and locally optimal points

We say $x^{\star }$ is an optimal point, or solves the problem above, if $x^{\star }$ is feasible and $f_0(x^{\ast })=p^{\ast }$.

The set of all optimal points is the optimal set, denoted
X o p t = { x ∣ f i ( x ) ≤ 0 , i = 1 , . . . , m , h i ( x ) = 0 , i = 1 , . . . , p , f 0 ( x ) = p ⋆ } X_{opt} =\{x|f_i(x)≤0, i=1,...,m, h_i(x)=0, i=1,...,p, f_0(x)=p^⋆\} Xopt​={x∣fi​(x)≤0,i=1,...,m,hi​(x)=0,i=1,...,p,f0​(x)=p⋆}
A feasible point $x$ with $f_0(x)\le p^{\star }+ϵ$ (where $ϵ>0$) is called $ϵ$-suboptimal, and the set of all $ϵ$-suboptimal points is called the $ϵ$-suboptimal set for the problem.

We say a feasible point x is locally optimal if there is an $R>0$ such that
f 0 ( x ) = inf ⁡ { f 0 ( z ) ∣ f i ( z ) ≤ 0 , i = 1 , . . . , m , h i ( z ) = 0 , i = 1 , . . . , p , ∥ z − x ∥ 2 ≤ R } f_0(x) = \inf\{f_0(z) \mid f_i(z) ≤ 0, i = 1,...,m,h_i(z)=0, i=1,...,p, ∥z−x∥_2 ≤R\} f0​(x)=inf{f0​(z)∣fi​(z)≤0,i=1,...,m,hi​(z)=0,i=1,...,p,∥z−x∥2​≤R}
or, in other words, $x$ solves the optimization problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(z)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(z)\le 0,i=1,...m\\ & h_i(z)=0,i=1,...p\\ & ||z-x|{|}_2\le R\end{array}$$

Feasibility problems

We call this the feasibility problem, and will sometimes write it as
$$\begin{array}{rl}\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{d}& x\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$

4.1.2 Expressing problems in standard form

A standrad form of an optimization problem is
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$

Maximization problems

We can solve the maximization problem $f$ as minimize $-f$.

4.1.3 Equivalent problems

We call two problems equivalent if from a solution of one, a solution of the other is readily found, and vice versa.

Change of variables

Suppose $\varphi :{\mathbf{R}}^n\to {\mathbf{R}}^n$ is one-to-one, with image covering the problem domain $D$. We define functions $f_i$ and $h_i$ as
$$\widetilde{f_i}(z)=f_i(\varphi (z)),i=0,...,m,{\tilde{h}}_i(z)=h_i(\varphi (z)),i=1,...p$$
The problem is equivalent
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& {\tilde{f}}_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& {\tilde{f}}_i(x)\le 0,i=1,...m\\ & {\tilde{h}}_i(x)=0,i=1,...p\end{array}$$
We say that the problems are related by the change of variable or substitution of variable $x=\varphi (z)$.

Transformation of objective and constraint functions

Suppose that ${\psi }_0:\mathbf{R}\to \mathbf{R}$ is monotone increasing, ${\psi }_1,...,{\psi }_m:\mathbf{R}\to \mathbf{R}$ satisfy ${\psi }_i(u)\le 0$ if and only if $u\le 0$, and ${\psi }_{m+1},...,{\psi }_{m+p}:\mathbf{R}\to \mathbf{R}$ satisfy ${\psi }_i(u)=0$ if and only if $u=0$. We define functions ${\tilde{f}}_i$ and ${\tilde{h}}_i$ as the compositions
$${\tilde{f}}_i(x)={\psi }_i(f_i(x)),i=0,...,m,{\tilde{h}}_i(x)={\psi }_{m+i}(h_i(x)),i=1,...,p$$
The problem is equivalent
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& {\tilde{f}}_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& {\tilde{f}}_i(x)\le 0,i=1,...m\\ & {\tilde{h}}_i(x)=0,i=1,...p\end{array}$$

Slack variables

One simple transformation is based on the observation that $f_i(x)\le 0$ if and only if there is an $s_i\ge 0$ that satisfies $f_i(x)+s_i=0$. The problem is equivalent:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& s_i\ge 0\\ & f_i(x)+s_i=0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$
Where $x\in {\mathbf{R}}^n,s\in {\mathbf{R}}^m$.

Eliminating equality constraints

Suppose the functionk $\varphi :{\mathbf{R}}^k\to {\mathbf{R}}^n$ is such that $x$ satisfies if and only if there is some $z\in \mathbf{R}$ such that $x=\varphi (z)$. The optimization problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& {\tilde{f}}_0(x)=f_0(\varphi (x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& {\tilde{f}}_i(x)=f_i(\varphi (x)\le 0,i=1,...m\end{array}$$

Eliminating linear equality constraints

Substituting $x=Fz+x_0$ into the original problem yields the problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(Fz+x_0)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(Fz+x_0)\le 0,i=1,...m\end{array}$$
Where $x$ is attained by eqauations.

Introducing equality constraints

We can also introduce equality constraints and new variables into a problem.

Optimizing over some variables

We always have
$$\underset{x,y}{\inf }f(x,y)=\underset{x}{\inf }\underset{y}{\inf }f(x,y)$$
In other words, we can always minimize a function by first minimizing over some of the variables, and then minimizing over the remaining ones.

Epigraph problem form

The epigraph form of the standard problem is the problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& t\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_0(x)-t\le 0\\ & f_i(x)\le 0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$
We can easily see that it is equivalent to the original problem: $(x,t)$ is optimal if and only if $x$ is optimal and $t=f_0(x)$.

Implicit and explicit constraints

As an extreme example, the standard form problem can be expressed as the unconstrained problem
$$\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}F(z)$$
where we define the function F as f0, but with domain restricted to the feasible set:
d o m   F = { x ∈ d o m   f 0 ∣ f i ( x ) ≤ 0 , i = 1 , . . . , m ,      h i ( x ) = 0 , i = 1 , . . . , p } \mathbf{dom} \ F =\{x∈\mathbf{dom}\ f_0 \mid f_i(x)≤0, i=1,...,m, \ \ \ \ h_i(x)=0, i=1,...,p\} dom F={x∈dom f0​∣fi​(x)≤0,i=1,...,m,    hi​(x)=0,i=1,...,p}

4.1.4 Parameter and oracle problem descriptions

4.2 Convex optimization

4.2.1 Convex optimization problems in standard form

A convex optimization problem is one of the form
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 0,i=1,...m\\ & a_i^{T}x=b_i,i=1,...p\end{array}$$
where $f_0,...,f_m$ are convex functions.

If $f_0$ is quasiconvex instead of convex, we say the problem is a (standard form) quasiconvex optimization problem.

Concave maximization problems

If $f_0$ is concave, the optimization problem is concave. This concave maximization problem is readily solved by minimizing the convex objective function $-f_0$.

Abstract form convex optimization problem

4.2.2 Local and global optima

A fundamental property of convex optimization problems is that any locally optimal point is also (globally) optimal.

4.2.3 An optimality criterion for differentiable f0

Suppose that the objective $f_0$ in a convex optimization problem is differentiable, then $x$ is optimal if and only if $x\in X$ and
$$\nabla f_0(x{)}^T(y-x)\ge 0\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{l}\mathrm{l}y\in X$$

4.2.4 Equivalent convex problems

Eliminating equality constraints

Introducing equality constraints

Slack variables

Introducing slack variables for linear inequalities preserves convexity of a problem.

Epigraph problem form

Minimizing over some variables

4.2.5 Quasiconvex optimization

Solving a quasiconvex optimization problem can be reduced to solving a sequence of convex optimization problems.

Locally optimal solutions and optimality conditions

The most important difference between convex and quasiconvex optimization is that a quasiconvex optimization problem can have locally optimal solutions that are not (globally) optimal.

It follows from the first-order condition for quasiconvexity that $x$ is optimal if
∇ f 0 ( x ) T ( y − x ) > 0   o r   a l l   y ∈ X   \ { x } ∇f_0(x)^T (y − x) > 0 \ {\rm or \ all} \ y ∈ X \ \backslash \{x \} ∇f0​(x)T(y−x)>0 or all y∈X \{x}

Quasiconvex optimization via convex feasibility problems

Let $\phi t:{\mathbf{R}}^n\to \mathbf{R},t\in \mathbf{R}$, be a family of convex functions that satisfy
$${\varphi }_0(x)\le t⟺{\varphi }_t(x)\le 0,$$
and also, for each $x,{\varphi }_s(x)\le {\varphi }_t(x)\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{r}s\ge t$.

Let $p^{\star }$ denote the optimal value of the quasiconvex optimization problem, define convex feasibility function
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& x\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& {\varphi }_t(x)\le 0\\ & f_i(x)\le 0,i=1,...m\\ & Ax=b\end{array}$$
This is repeated until the width of the interval is small enough:

4.3 Linear optimization problems

When the objective and constraint functions are all affine, the problem is called a linear program (LP). A general linear program has the form
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& c^{T}x+d\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Gx⪯h\\ & Ax=b\end{array}$$
where $G\in {\mathbf{R}}^{m\times n}\mathrm{a}\mathrm{n}\mathrm{d}A\in {\mathbf{R}}^{p\times n}$.

Standard and inequality form linear programs

In a standard form LP the only inequalities are componentwise nonnegativity constraints $x⪰0$:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& c^{T}x\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Ax=b\\ & x⪰0\end{array}$$
If LP has no equality constraints, it is a inequality form LP:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& c^{T}x\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Ax⪯b\end{array}$$
Converting LPs to standard form

It is sometimes useful to transform a general LP to one in standard form. The first step is to introduce slack variables $s_i$ for the inequalities, which results in
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& c^{T}x+d\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Gx+s=h\\ & Ax=b\\ & s⪰0\end{array}$$
The second step is to express the variable $x$ as the difference of two nonnegative variables $x^{+}$ and $x^{-}$, i.e., $x=x^{+}-x^{-},x^{+},x^{-}⪰0$. This yields the problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& c^T(x^{+}-x^{-})+d\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& G(x^{+}-x^{-})+s=h\\ & A(x^{+}-x^{-})=b\\ & x^{+},x^{-},s⪰0\end{array}$$

4.3.1 Examples

4.3.2 Linear-fractional programming

The problem of minimizing a ratio of affine functions over a polyhedron is called a linear-fractional program:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Gx⪯h\\ & Ax=b\end{array}$$
Where f 0 ( x ) = c T x + d e T x + f , d o m   f 0 = { x ∣ e T x + f > 0 } f_0(x)=\frac{c^Tx+d}{e^Tx+f}, \mathbf{dom} \ f_0=\{ x\mid e^Tx+f>0 \} f0​(x)=eTx+fcTx+d​,dom f0​={x∣eTx+f>0}.

Transforming to a linear program

the linear-fractional program can be transformed to an equivalent linear program:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& c^{T}y+dz\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Gy-hz⪯0\\ & Ay-bz=0\\ & e^{T}y+fz=1\\ & z⪰0\end{array}$$
Where $y=\frac{x}{e^{T}x+f},z=\frac{1}{e^{T}x+f}$

Generalized linear-fractional programming

A generalization of the linear-fractional program is the generalized linear-fractional program in which
f 0 ( x ) = max ⁡ i = 1 , . . . , r c i T x + d i e i T x + f i ,    d o m   f 0 = { x ∣ e i T x + f i > 0 , i = 1 , . . . , r } f_0(x)=\max_{i=1,...,r}\frac{c^T_ix+d_i}{e^T_ix+f_i}, \ \ \mathbf{dom}\ f_0=\{ x \mid e_i^Tx+f_i>0,i=1,...,r \} f0​(x)=i=1,...,rmax​eiT​x+fi​ciT​x+di​​,  dom f0​={x∣eiT​x+fi​>0,i=1,...,r}

4.4 Quadratic optimization problems

The convex optimization problem is called a quadratic program (QP) if the objective function is (convex) quadratic, and the constraint functions are affine:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& \frac{1}{2}{x}^{T}Px+q^{T}x+r\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Gx⪯h\\ & Ax=b\end{array}$$
where $P\in {\mathbf{S}}_{+}^n,P\in {\mathbf{R}}^{m\times n},A\in {\mathbf{R}}^{p\times n}$.

4.4.1 Examples

4.4.2 Second-order cone programming

A problem that is closely related to quadratic programming is the second-order cone program (SOCP):
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f^{T}x\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& ||A_{i}x+b_i|{|}_2\le c_i^{T}x+d_i,i=1,...,m\\ & Fx=g\end{array}$$
The constrain of the form
$$\Vert Ax+b{\Vert }_2\le c^{T}x+d$$
a second-order cone constraint.

4.5 Geometric programming

4.5.1 Monomials and posynomials

A function $f:{\mathbf{R}}^n\to \mathbf{R}$ with $\mathbf{d}\mathbf{o}\mathbf{m}f={\mathbf{R}}_{++}^n$ is a monomial function, or simply, a monomial that
$$f(x)=c{x}_1^{a_1}\cdots x_n^{a_n}$$
where $c>0 \ \mbox{and} \ a_i \in \mathbf{R} $.

A sum of monomials
$$f(x)=\sum _k{c}_k{x}_1^{a_{1}k}\cdots x_n^{a_{n}k}$$
where $c_k>0$, is called a posynomial function (with $K$ terms), or simply, a posynomial.

4.5.2 Geometric programming

An optimization problem of the form
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 1,i=1,...m\\ & h_i(x)=1,i=1,...p\end{array}$$
where $f_0,...,f_m$ are posynomials and $h_1,...,h_p$ are monomials, is called a geometric program (GP).

4.5.3 Geometric program in convex form

Define $y_i=\log x_i$, and take logarithm
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& \log \sum _k{e}^{a_{0k}^{T}y+b_{0}k}\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& \log \sum _k{e}^{a_{ik}^{T}y+b_{i}k}\le 1,i=1,...m\\ & g_i^{T}y+h_i=0,i=1,...p\end{array}$$

4.5.4 Examples

4.6 Generalized inequality constraints

The standard form of optimization problem can be extended to vector valued:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x){⪯}_{K_i}0,i=1,...m\\ & Ax=b\end{array}$$
where $f_0:{\mathbf{R}}^n\to \mathbf{R},K_i\in {\mathbf{R}}^{k_i}$ are proper cones and $f_i:{\mathbf{R}}^n\to {\mathbf{R}}^{k_i}$ are $K_i$-convex. We refer to this problem as a (standard form) convex optimization problem with generalized inequality constraints.

4.6.1 Conic form problems

Among the simplest convex optimization problems with generalized inequalities are the conic form problems (or cone programs)
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& c^{T}x\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Fx+g{⪯}_{K}0,i=1,...m\\ & Ax=b\end{array}$$

4.6.2 Semidefinite programming

When $K$ is ${\mathbf{S}}_{+}^k$, the cone of positive semidefinite $k\times k$ matrices, the associated conic form problem is called a semidefinite program (SDP), and has the form
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& c^{T}x\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& x_1{F}_1+\cdots +x_n{F}_n+G{⪯}_{}0,i=1,...m\\ & Ax=b\end{array}$$
where $G,F_1,…,F_n\in \mathbf{S}^k,A\in \mathbf{R}^{p\times n} $.

4.7 Vector optimization

4.7.1 General and convex vector optimization problems

We denote a general vector optimization problem as
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}(\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}K)& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$

4.7.2 Optimal points and values

Consider the set of objective values of feasible points,
O = { f 0 ( x ) ∣ ∃ x ∈ D , f i ( x ) ≤ 0 , i = 1 , . . . , m ; h i ( x ) = 0 , i = 1 , . . . p } ⊆ R q \mathcal{O}=\{ f_0(x) \mid \exist x\in \mathcal{D},f_i(x)\leq0,i=1,...,m;h_i(x)=0,i=1,...p \}\subseteq\mathbf{R}^{q} O={f0​(x)∣∃x∈D,fi​(x)≤0,i=1,...,m;hi​(x)=0,i=1,...p}⊆Rq
which is called the set of achievable objective values. If this set has a minimum element, i.e., there is a feasible $x$ such that $f_0(x){⪯}_K{f}_0(y)$ for all feasible $y$, then we say $x$ is optimal for the problem, and refer to $f_0(x)$ as the optimal value of the problem.

4.7.3 Pareto optimal points and values

Thus, a point $x$ is Pareto optimal if it is feasible and, for any feasible $y,f_0(y){⪯}_K{f}_0(x)$ implies $f_0(y)=f_0(x)$.

4.7.4 Scalarization

Scalarization is a standard technique for finding Pareto optimal (or optimal) points for a vector optimization problem. Choose any $\lambda {\succ }_{K^{\ast }}0$, i.e., any vector that is positive in the dual generalized inequality. Now consider the scalar optimization problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& {\lambda }^T{f}_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$

4.7.5 Multicriterion optimization

In other words, $x^{\star }$ is simultaneously optimal for each of the scalar problems
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& F_j(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$
it is called a multicriterion or multi-objective optimization problem.