Linear Programming Learning Notes (5) Duality Theory

Linear Programming Learning Notes (5) Duality Theory

All the resources come from Linear Programming: Foundations and Extensions by Professor Robert J. Vanderbei.
Explore the link below for further information:
LP Book Resources

Part 1

Basic Theory: The Simplex Method and Duality

Chapter 5 Duality Theory

Associated with every linear program is another called its dual. The dual of this dual program is the original linear program.
Hence, linear programs come in primal/dual pairs. It turns out that every feasible solution for one of these two linear programs gives a bound on the optimal objective function value for the other. Thes ideas form a subject called duality theory.

Movitation: Finding Upper Bounds

Let’s begin with an example:
maximize        $4x_1 + x_2 + 3x_3$
s.t.                   $x_1 + 4x_2 \leq 1$
                        $3x_1 - x_2 + x_3 \leq 3$
                                 $x_1,x_2,x_3 \geq 0$
Every feasible solution provides a lower bound on the optimal objective function value $\zeta^{*}$. Let’s multiply the two constraints by nonnegative numbers, $y_1$ and $y_2$, respectively, to get:
$y_1(x_1+4x_2)\leq y_1$
$+y_2(3x_1 - x_2 + x_3)\leq 3y_2$
$\overline{(y_1+3y_2) x_1 + (4y_1-y_2)x_2 + (y_2)x_3 \leq y_1 + 3y_2}$
If we stipulate that each of the coefficients of the $x_i$’s be at least as large as the corresponding coefficient in the objective funcion,
$y_1 + 3y_2 \geq 4$
$4y_1 - y_2 \geq1$
$y_2 \geq3$
so that :
$\zeta \leq y_1 + 3y_2$
Now we have the upper bound, which we should minimize in our effort to obtain the best possible upper bound. Therefore, we are naturally led to the following optimization problem:
minimize        $y_1+3y_2$
s.t.                    $y_1+3y_2\geq4$
                         $4y_1 -y_2 \geq 1$
                              $4y_2 \geq 3$
                                   $y_1,y_2 \geq 0$
This problem is called the dual linear programming problem associated with the given one.

The Dual Problem

Given a linear programming problem in standard form,
maximize        $\sum_{j=1}^n c_jx_j$
s.t.                   $\sum_{j=1}^n a_{ij}x_j\leq b_i, i=1,2,...,m$
                                        $x_j \geq0, j=1,2,...,m$
the associated dual linear program is given by
minimize        $\sum_{i=1}^m b_iy_i$
s.t.                   $\sum_{i=1}^m a_{ij}y_i\geq c_j, j=1,2,...,n$
                                        $y_i \geq0, i=1,2,...,n$
and the standard form of the dual is
-maximize        $\sum_{i=1}^m (-b_i)y_i$
s.t.                   $\sum_{i=1}^m (-a_{ij})y_i\leq (-c_j), j=1,2,...,n$
                                        $y_i \geq0, i=1,2,...,n$

The Weak Duality Theorem

If $(x_1,x_2,...,x_n)$ is feasible for the primal and $(y_1my_2,...,y_m)$ is feasible for the dual, then
$\sum _j c_jx_j \leq \sum_i b_iy_i$

The Strong Duality Theorem

If the primal problem has an optimal solusion,
$x^* = (x_1^*, x_2^*,...,x_n^*)$
then the dual also has an optimal solution,
$y^* = (y_1^*, y_2^*,...,y_m^*)$
such that
$\sum _j c_jx_j^* = \sum_i b_iy_i^*$
The main idea is that, as the simplex method solves the primal problem, it also implicitly solves the dual problem, and it does so in a way that the theorem holds.

We should notice here that, the dual dictionary is the negative transpose of the primal dictionary. For example,
$\underline{\zeta = 4x_1+x_2+3x_3}$ …………………(P)
$w_1 = 1-x_1-4x_2$
$w_2 = 3-3x_1+x_2-x_3$
$.$
$\underline{-\zeta = -y_1 - 3y_2}$ ……………………….(D)
$z_1 = -4+y_1+3y_2$
$z_2 = -1+4y_1-y_2$
$z_3 = -3+y_2$
It can be written as:
$\begin{bmatrix}0& 4&1&3\\1&-1&-4&0\\3&-3&1&-1 \end{bmatrix} \\$ $\xrightarrow{neg. -transpo.}$ $\begin{bmatrix} 0&-1&-3 \\ -4&1&3 \\ -1&4&-1\\ -3&0&1 \end{bmatrix}$
Each primal dictionary generated by the simplex method implicited defines a corresponding dual dictionary as follows: first write down the negative transpose and then replace each $x_j$ with a $z_j$ and each $w_i$ with a $y_i$. As long as the primal dictionary is not optimal, the implicitly defined dual dictionary will be infeasible. But once an optimal primal dictionary is found, the corresponding dual dictionary will be feasible.

Complementary Slackness

Sometimes it is necessary to recover an optimal dual solution when only an optimal primal solution is known. The following theorem, known as the Complementary Slackness Theorem, can help in this regard.
THEOREM Suppose that $x=(x_1,x_2,...,x_n)$ is primal feasible and that $y=(y_1,y_2,...,y_m)$ is dual feasible. Let$(w_1,w_2,...,w_m)$ denote the corresponding primal slack variables, and let$(z_1,z_2,...,z_n)$ denote the corresponding dual slack variables. Then x and y are optimal for their respective problems if and only if:
$x_jz_j=0$, for $j=1,2,...,n$
$w_iy_i=0$, for $i=1,2,...,m$
And below is how we recover the dual solution. Suppose that we have a nondegenerate primal basic optimal solution
$x^* = (x_1^*,x_2^*,...,x_n^*)$
and we wish to find a corresponding optimal solution for the dual. Let
$w^* = (w_1^*,w_2^*,...,w_m^*)$
denote the corresponding slack variables, which can be obtained as:
$w_i^* = b_i - \sum_ja_{ij}x_j^*$
The dual constraints are
$\sum_i y_ia_{ij} - z_j = c_j, j=1,2,...,n$
These constraints form n equations in $m+n$ unknowns. Since the primal solution is assumed to be nondegenerate, it follows that the $m$ basic variables will be strictly positive. The complementary slackness theorem then tells us that the corresponding dual variables must vanish. Hence we are then left with just $n$ equations in $n$ unknowns. If there is a unique solution, all the components should be nonnegative, or it would stand in contradiction to the assumed optimality of $x^*$.

The Dual Simplex Method

We use this method when the primal dictionary is not feasible while the dual is feasible. When we just focus on the primal, we choose the entering and leaving variables following the rules below:
1. Select the basic variable whose constant value is the most negative.
2. Pick the entering variable by scanning the row of this basic variable, looking for the largest negated ratio.
For example:
$\underline{\zeta = -x_1-x_2}$
$w_1 = 4+2x_1+x_2$
$w_2 =-8+2x_1-4x_2$
$w_3 = -7+x_1-3x_2$
First we select $w_2$ as the leaving variable, since -8 is the most negative. Then we pick $x_1$ as the entering variable, since $2/(-1) = -2$ is more negative than $(-4)/(-1)=4$.
Finally we can get an optimized dictionary as:
$\underline{\zeta = -7-w_3-4x_2}$
$w_1 = 18+2w_3+7x_2$
$x_1 = 7+w_3+3x_2$
$w_2 = 6+2w_3+2x_2$

A Dual-Based Phase I Algorithm

At least it is more elegant than the Phase I before. Consider the following example:
$\underline{\zeta = -x_1+4x_2}$
$w_1 = 4+2x_1+x_2$
$w_2 =-8+2x_1-4x_2$
$w_3 = -7+x_1-3x_2$
Now, neither the primal and the dual is feasible. Let us temporarily change the primal objective function to
$\eta = -x_1-x_2$
Then the corresponding initial dual dictionary is feasible. In fact, it is the same dictionary as that above. The optimal is
$\underline{\zeta = -7-w_3-4x_2}$
$w_1 = 18+2w_3+7x_2$
$x_1 = 7+w_3+3x_2$
$w_2 = 6+2w_3+2x_2$
Now we simply reinstate the intended objective function and continue with Phase II.
$\zeta = -x_1 +4x_2 =-(4+w_3+3x_2)+4x_2=-7-w_3+x_2$
Hence, the starting dictionary for Phase II is
$\underline{\zeta = -7-w_3+x_2}$
$w_1 = 18+2w_3+7x_2$
$x_1 = 7+w_3+3x_2$
$w_2 = 6+2w_3+2x_2$

Rules of Forming the Dual

Primal	Dual
Equality constraint	Free variable
Inequality constraint	Nonnegative variable
Free variable	Equality constraint
Nonegative variable	Inequality constraint

Lagrangian Duality

$\pi(x_1,...,x_n,y_1,...,y_m) = \sum_j c_jx_j - \sum_i\sum_j y_ia_{ij}x_j + \sum_i y_ib_i$
$\max_{x\geq0} \min_{y\geq0} \pi(x,y) = \min_{y\geq0} \max_{x\geq 0} \pi(x,y)$
The max-min is the primal, and the min-max is the dual.