9.3 Linear Programming---Simplex Method (线性规划---单纯形法)

本文为《Linear algebra and its applications》的读书笔记

The first example is simple, but it suggests how a problem of linear programming could involve hundreds, if not thousands, of variables and equations.

EXAMPLE 1
A retail sales company has two warehouses and four stores. A particular model of outdoor hot tub is sold at all four stores, and each store has placed an order with company headquarters for a certain number of these hot tubs. Headquarters determines that the warehouses have enough hot tubs and can ship them immediately. The distances from the warehouses to the stores vary, and the cost of transporting a hot tub from a warehouse to a store depends on the distance. The problem is to decide on a shipping schedule that minimizes the total cost of shipping.

Let $x_{ij}$ be the number of units (hot tubs) to ship from warehouse $i$ to store $j$.

Let $a_1$ and $a_2$ be the numbers of units available at warehouses 1 and 2, and let $r_1,...,r_4$ be the numbers of units requested by the various stores. Then the $x_{ij}$ must satisfy the equations

and $x_{ij}\ge 0$ for $i=1,2$ and $j=1,...,4$. If the cost of shipping one unit from warehouse $i$ to store $j$ is $c_{ij}$ , then the problem is to minimize the function

subject to the four equalities and ten inequalities listed above.

Simplex Method

The simplex method , discussed below, can easily handle problems the size of Example 1. To introduce the method, however, this section focuses mainly on the canonical linear programming problem, in which the objective function must be maximized. Here is an outline of the steps in the simplex method.

1. Select an extreme point $\mathbf{x}$ of the feasible set $\mathcal{F}$.
2. Consider all the edges of $\mathcal{F}$ that join at $\mathbf{x}$. If the objective function $f$ cannot be increased by moving along any of these edges, then $\mathbf{x}$ is an optimal solution.
3. If $f$ can be increased by moving along one or more of the edges, then follow the path that gives the largest increase and move to the extreme point of $\mathcal{F}$ at the opposite end.
4. Repeat the process, beginning at step 2.

Since the value of $f$ increases at each step, the path will not go through the same extreme point twice. Since there are only a finite number of extreme points, this process will end at an optimal solution (if there is one) in a finite number of steps. If the problem is unbounded, then eventually the path will reach an unbounded edge at step 3 along which $f$ increases without bound.

这是单纯形法的主要思想。具体做法会在下面阐述

---

The next examples concern canonical linear programming problems in which each of the entries in the $m$-tuple $\mathbf{b}$ is positive:

Here $\mathbf{c}$ and $\mathbf{x}$ are in ${\mathbb{R}}^n$, $A$ is an $m\times n$ matrix, and $\mathbf{b}$ is in ${\mathbb{R}}^m$.

---

The simplex method begins by changing each constraint inequality into an equality. This is done by adding one new variable (slack variable) to each inequality.

slack variable: 松弛变量

EXAMPLE 2
Change the inequality

into the equality

by adding the slack variable $x_3$.

If $A$ is $m\times n$, the addition of $m$ slack variables in $A\mathbf{x}\le \mathbf{b}$ produces a linear system with $m$ equations and $n+m$ variables.

• A solution to this system is called a basic solution if no more than $m$ of the variables are nonzero.
• A solution to the system is called feasible if each variable is nonnegative.

Thus, in a basic feasible solution, each variable must be nonnegative and at most $m$ of them can be positive. Geometrically, these basic feasible solutions correspond to the extreme points of the feasible set.

EXAMPLE 3
Find a basic feasible solution for the system

SOLUTION
Add $slack$ $variables$ to obtain a system of three equations:

A basic solution of (1) has at most three nonzero values for the variables. The following simple solution is called the basic feasible solution associated with (1):

This solution corresponds to the extreme point $0$ in the feasible set (in ${\mathbb{R}}^3$).

It is customary to refer to the nonzero variables $x_4,x_5$, and $x_6$ in system (1) as basic variables because each has a coefficient of 1 and occurs in only one equation. The basic variables are said to be “in” the solution of (1). The variables $x_1,x_2$, and $x_3$ are said to be “out” of the solution. In a linear programming problem, this particular solution would probably not be optimal since only the slack variables are nonzero.

---

A standard procedure in the simplex method is to change the role a variable plays in a solution.
(通过这个方法在不同的极端点处求得目标函数的最优解)

For example, although $x_2$ is out of the solution in (1), it can be introduced “into” a solution by using elementary row operations. The goal is to pivot on the $x_2$ entry in the third equation of (1) to create a new system in which $x_2$ appears only in the third equation.

To “pivot” on a particular term here means to transform its coefficient into a 1 and then use it to eliminate corresponding terms in all the other equations, not just the equations below it, as was done in Section 1.2.

First, divide the third equation in (1) by the coefficient of $x_2$ to obtain a new third equation:

Second, to equations 1 and 2 of (1) add multiples of this new equation that will eliminate $x_2$ from those equations. This produces the system

The basic solution associated with this new system is

The variable $x_2$ has come into the solution, and the variable $x_6$ has gone out. Unfortunately, this basic solution is not feasible since $x_4<0$. This lack of feasibility was caused by an improper choice of a pivot equation. The next paragraph shows how to avoid this problem.

---

In general, consider the system

and suppose the next step is to bring the variable $x_k$ into the solution by using equation $p$ to pivot on entry $a_{pk}{x}_k$ . The basic solution corresponding to the resulting system will be feasible if the following two conditions are satisfied:

1. The coefficient $a_{pk}$ of $x_k$ must be positive.
   (When the $p$th equation is divided by $a_{pk}$, the new $b_p$ term must be positive.)
2. The ratio $b_p/a_{pk}$ must be the smallest among all the ratios $b_i/a_{ik}$ for which $a_{ik}>0$.
   (This will guarantee that when the $p$th equation is used to eliminate the $x_k$ term from the $i$ th equation, the resulting $b_i$ term will be positive.)

EXAMPLE 4
Determine which row to use as a pivot in order to bring $x_2$ into the solution in Example 3.
SOLUTION
Compute the ratios $b_i/a_{i2}$:

Since the first ratio is the smallest, pivot on the $x_2$ term in the first equation. This produces the system

Now the basic feasible solution is

A matrix format greatly simplifies calculations of this type. For instance, system (1) in Example 3 is represented by the augmented matrix

The circled 3 in the $x_2$ column indicates that this entry will be used as a pivot to bring $x_2$ into the solution. Complete row reduction in column 2 produces the new matrix that corresponds to the new system in Example 4:

As in Example 4, the new basic feasible solution is

读到现在大概能理解为什么说一个基本可行解就对应一个极端点了：
如果引入的松弛变量为 0，则表明某个不等式取得了相等关系，也就是说解在该等式表示的超平面上；而如果 $x_i=0$ ($x_1$ 非松弛变量)，则表明解在 $x_i=0$ 这个超平面上。
所以当取得基本可行解时，至少有 $n$ 个变量为 0，表明解(该解在 $n$ 维空间内) 至少在 $n$ 个不同的表示凸包边界的超平面的交点上，该交点一定为顶点，也就一定为极端点

---

The preceding discussion has prepared the way for a full demonstration of the simplex method, based on the constraints in Example 3. At each step, the objective function in Example 5 will drive the choice of which variable to bring into the solution of the system.

EXAMPLE 5
Maximize $25x_1+33x_2+18x_3$

SOLUTION
First, add slack variables. Then change the objective function $25x_1+33x_2+18x_3$ into an $equation$ by introducing a new variable $M$ given by $M=25x_1+33x_2+18x_3$. Now the goal is to maximize the variable $M$ , where $M$ satisfies the equation

The original problem is now restated as follows:

find a solution for which $x_j\ge 0(j=1,...,6)$ and for which $M$ is as large as possible.

The augmented matrix for this new system is called the initial simplex tableau (初始单纯形表). It is written with two ruled lines in the matrix:

The horizontal line above the bottom row isolates the equation corresponding to the objective function. This last row will play a special role in what follows. (The bottom row is used only to decide which variable to bring into the solution. Pivot positions are never chosen from the bottom row.)

The column headings for the slack variables are in color, to remind us at the end of the calculations that only the original variables are part of the final solution of the problem.

Look in rows 1 to 3 of the tableau above to find the basic feasible solution. The $columns$ $of$ $the$ $3\times 3$ $identity$ $matrix$ in these three rows $identify$ the basic variables—namely, $x_4,x_5$, and $x_6$. The basic solution is

This solution is not optimal, however, since only the slack variables are nonzero. However, the bottom row implies that

The value of $M$ will rise when any of the variables $x_1,x_2$, or $x_3$ rises. Since the coefficient of $x_2$ is the largest of the three coefficients, bringing $x_2$ into the solution will cause the greatest increase in $M$.

To bring $x_2$ into the solution, follow the pivoting procedure outlined earlier. The pivot should be the entry 3 that is circled in the first row.

The result of the pivot operation is

Now the columns of the $3\times 3$ identity matrix are in columns 2, 5, and 6 of the tableau. So the basic feasible solution is

Thus $M$ has increased from 0 to 660. To see if $M$ can be increased further, look at the bottom row of the tableau and solve the equation for $M$ :

The value of $M$ will increase only if $x_1$ increases (from 0). So $x_1$ needs to come into the solution:

After pivoting, the resulting tableau is

The corresponding basic feasible solution is

The bottom row shows that

The negative coefficients of the variables here show that $M$ can be no larger than $\frac{4854}{7}$, so the solution is optimal. The maximum occurs when $x_1=\frac{78}{7},x_2=\frac{88}{7},$ and $x_3=0$.

The fact that the slack variables $x_4$ and $x_5$ are zero means that the first two inequalities listed at the beginning of this example are both equalities at the optimal values of $x_1,x_2$, and $x_3$.

---

The goal of step 3 is to produce the greatest increase possible in the value of $M$ . This happens when only one variable $x_k$ satisfies the conditions. Suppose, however, that the most negative entry in the bottom row appears in both columns $j$ and $k$ . Step 3 says that either $x_j$ or $x_k$ should be brought into the solution, and that is correct. Occasionally, a few computations can be avoided by first using step 4 to compute the “smallest ratio” for both columns $j$ and $k$ , and then choosing the column for which this “smallest ratio” is larger. This situation will arise in Section 9.4.

Two things can go wrong in the simplex algorithm.

• At step 4, there might be a negative entry in the bottom row of the $x_k$ column, but no positive entry $a_{ik}$ above it. In this case, it will not be possible to find a pivot to bring $x_k$ into the solution. This corresponds to the case where the objective function is unbounded and no optimal solution exists.

• The second potential problem also occurs at step 4. The smallest ratio $b_i/a_{ik}$ may occur in more than one row. When this happens, the next tableau will have at least one basic variable equal to zero, and in subsequent tableaus the value of $M$ may remain constant. Theoretically it is possible for an infinite sequence of pivots to occur and fail to lead to an optimal solution. Such a phenomenon is called cycling. Fortunately, cycling occurs only rarely in practical applications. In most cases, one may arbitrarily choose either row with a minimum ratio as the pivot.

Minimization Problems

So far, each canonical maximizing problem involved a vector $\mathbf{b}$ whose coordinates were positive. But what happens when some of the coordinates of $\mathbf{b}$ are zero or negative? And what about a minimizing problem?

---

If some of the coordinates of $\mathbf{b}$ are zero, then it is possible for cycling to occur and the simplex method to fail to terminate at an optimal solution.
(Fortunately, cycling does not generally happen in practical applications.)

---

The case when one of the coordinates of $\mathbf{b}$ is negative can occur in practice and requires some special consideration. The difficulty is that all the $b_i$ terms must be nonnegative in order for the slack variables to provide an initial basic feasible solution.

One way to change a negative $b_i$ term into a positive term would be to multiply the inequality by $-1$ (before introducing slack variables). But this would change the direction of the inequality. Thus a negative $b_i$ term causes the same problem as a reversed inequality. The following example discusses this case.

EXAMPLE 7

SOLUTION
The minimum of $f(x_1,x_2)$ over a set is the same as the maximum of $-f(x_1,x_2)$ over the same set. However, in order to use the simplex algorithm, the canonical description of the feasible set must use $\le$ signs. So the first inequality above must be rewritten. Thus the original problem is equivalent to the following:

To solve this, let $M=-x_1-2x_2$ and add slack variables to the inequalities. This creates the linear system

To find a nonnegative solution to this system for which $M$ is a maximum, construct the initial simplex tableau:

The corresponding basic solution is

However, since $x_3$ is negative, this basic solution is not feasible.

In order to replace a negative $b_i$ entry by a positive number, find another negative entry in the same row. (If all the other entries in the row are nonnegative, then the problem has no feasible solution.) This negative entry is in the column corresponding to the variable that should now come into the solution.

In this example, the first two columns both have negative entries, so either $x_1$ or $x_2$ should be brought into the solution. For example, to bring $x_2$ into the solution, select as a pivot the entry $a_{i2}$ in column 2 for which the ratio $b_i/a_{i2}$ is the smallest non-negative number. In this case, only the ratio $(-14)/(-1)$ is nonnegative, so the −1 in the first row must be the pivot. After the pivot operations on column 2, the resulting tableau is

Now each entry in the augmented column (except the bottom entry) is positive, and the simplex method can begin. (Sometimes it may be necessary to pivot more than once in order to make each of these terms nonnegative.) The next tableau turns out to be optimal:

The maximum feasible value of $-x_1-2x_2$ is $-20$, when $x_1=8$ and $x_2=6$. So the minimum value of $x_1+2x_2$ is $20$.

The “Simplex” in the Simplex Algorithm

The simplex algorithm focuses on the columns of $A$ instead of the rows. Suppose that $A$ is $m\times n$ and denote the columns by ${\mathbf{a}}_1,...,{\mathbf{a}}_m$. The addition of $m$ slack variables creates an $m$ by $n+m$ system of equations of the form

where $x_1,...,x_{n+m}$ are nonnegative and { e 1 , . . . , e m } \{\boldsymbol e_1, . . . , \boldsymbol e_m\} {e1​,...,em​} is the standard basis for ${\mathbb{R}}^m$.

The initial basic feasible solution is obtained when $x_1,...,x_n$ are zero and $b_1{\mathbf{e}}_1+\cdot \cdot \cdot +b_m{\mathbf{e}}_m=\mathbf{b}$. If $s=b_1+\cdot \cdot \cdot +b_m$, then the equation

If ${\mathbf{v}}_1,...,{\mathbf{v}}_m$ are linearly independent vectors in ${\mathbb{R}}^m$, then the convex hull of the set { 0 , v 1 , . . . , v m } \{\boldsymbol 0, \boldsymbol v_1 , . . . , \boldsymbol v_m\} {0,v1​,...,vm​} is an $m$-dimensional simplex, $S$. A typical vector in $S$ has the form $c_{0}0+c_1{\mathbf{v}}_1+\cdot \cdot \cdot +c_m{\mathbf{v}}_m$, where the weights are nonnegative and sum to one.

shows that $\mathbf{b}$ is in what is called the $simplex$ generated by $0,s{\mathbf{e}}_1,...,s{\mathbf{e}}_m$. For simplicity, we say that “$\mathbf{b}$ is in an $m$-dimensional simplex determined by ${\mathbf{e}}_1,...,{\mathbf{e}}_m$.” This is the first simplex in the simplex algorithm.

In general, if ${\mathbf{v}}_1,...,{\mathbf{v}}_m$ is any basis of ${\mathbb{R}}^m$, selected from the columns of the matrix $P=\left[\begin{array}{cccccc}{\mathbf{a}}_1& \cdot \cdot \cdot & {\mathbf{a}}_n& {\mathbf{e}}_1& \cdot \cdot \cdot & {\mathbf{e}}_m\end{array}\right]$, and if $\mathbf{b}$ is a linear combination of these vectors with nonnegative weights, then $\mathbf{b}$ is in an $m$-dimensional simplex determined by ${\mathbf{v}}_1,...,{\mathbf{v}}_m$.

A $basic$ feasible solution of the linear programming problem corresponds to a particular basis from the columns of $P$. The simplex algorithm changes this basis and hence the corresponding simplex that contains $\mathbf{b}$, one column at a time. The various ratios computed during the algorithm drive the choice of columns. Since row operations do not change the linear dependence relations among the columns, each basic feasible solution tells how to build $\mathbf{b}$ from the corresponding columns of $P$.