LI_AX_and_the_column_space_of_A

Lecture 1: AX and the column space of A

How to look at matrix times a vector
$AX$:

$$\left[\begin{array}{ccc}2& 1& 3\\ 3& 1& 4\\ 5& 7& 12\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]$$

1. standard way: dot product of row x,
   get a component at a time——a low level way
2. vector wise——right way
   $$\left[\begin{array}{ccc}2& 1& 3\\ 3& 1& 4\\ 5& 7& 12\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]=x_1\left[\begin{array}{c}2\\ 3\\ 5\end{array}\right]+x_2\left[\begin{array}{c}1\\ 1\\ 7\end{array}\right]+x_3\left[\begin{array}{c}3\\ 4\\ 12\end{array}\right]$$
   think of a matrix as a whole thing,
   not just a bunch of m times n numbers, but a thing —— a matrix multiplies a vector to give another vector. $AX$ is a combination of the columns of $A$.

Column Space:

A more general situation: take a matrix $A$, take all $Xs$, and imagine all the outputs, what’s that look like? It’s the column space of $A$, denoted as $C(A)$, which is a space depend on $A$.

$As$ to $A$ = $\left[\begin{array}{ccc}1& 3& 8\\ 1& 3& 8\\ 1& 3& 8\end{array}\right],$ $C(A)=x_1\left[\begin{array}{c}1\\ 1\\ 1\end{array}\right]+x_2\left[\begin{array}{c}3\\ 3\\ 3\end{array}\right]+x_3\left[\begin{array}{c}8\\ 8\\ 8\end{array}\right]\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{X}$ is a line, and we can see rank(A) = 1.

The Rank of a matrix A is the dimension of its column space.

Simply put, the columns of A (a set of vectors) form a space, which is the column space of A. The columns are the bases of that space.

The first A’s rank is two, because the third column is a combination (multiply and add) of the others. The independent columns would be bases for the column space.

Rank-1 matrixes are the building blocks of linear algebra, which would be elaborated more detailed after. A special way to write those rank one matrixes is a column times a row:
$$\left[\begin{array}{ccc}1& 3& 8\\ 1& 3& 8\\ 1& 3& 8\end{array}\right]=\left[\begin{array}{c}1\\ 1\\ 1\end{array}\right]\left[\begin{array}{ccc}1& 3& 8\end{array}\right]$$

Independent Columns:

Find a basis for the column space of $A$ and Factor $A$ into $C$ times $R$

Basis of a space:

• a set of independent columns (No column is a combination of others)
• their combinations have to fill the space

To find a basis of $A$, the goal is to create a matrix $C$ whose columns directly from $A$, but not include any column that is a combination of previous columns. A natural construction of $C$:

• If column 1 of $A$ is not all zero, put it into the matrix $C$.
• If column 2 of $A$ is not a multiple of column 1 put it into $C$.
• If column 3 of $A$ is not a combination of column 1 and 2, put if into $C$. $Continue$

The final $C$ will be a “basis” for the column space of $A$.

Example: If $A=\left[\begin{array}{ccc}1& 3& 8\\ 1& 2& 6\\ 0& 1& 2\end{array}\right]$, then $C=\left[\begin{array}{cc}1& 3\\ 1& 2\\ 0& 1\end{array}\right]$.The column 3 of $A$ is the combination of 1 and 2, so it is dropped. It’s obvious that the number of $C$’s columns is the rank of $A$, also the rank of $C$. It counts the independent columns, which is mentioned above.

Again:

The column rank = The number of independent columns.
The rank of a matrix is the dimension of its column space.

The matrix $C$ connects to $A$ by a third matrix $R$: $A=CR$. It’s an important ** factorization ** of $A$.

$$A=\left[\begin{array}{ccc}1& 3& 8\\ 1& 2& 6\\ 0& 1& 2\end{array}\right]=\left[\begin{array}{cc}1& 3\\ 1& 2\\ 0& 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 2\\ 0& 1& 2\end{array}\right]=CR$$
We can see it in the right way of matrix times vector in the beginning:

• column $i$ of $A$ is $C$ times column $i$ of $R$, take column 1 for example: $\left[\begin{array}{c}1\\ 1\\ 0\end{array}\right]=1\ast \left[\begin{array}{c}1\\ 1\\ 0\end{array}\right]+0\ast \left[\begin{array}{c}3\\ 2\\ 1\end{array}\right]$

First great theorem: The column rank = The row rank

Next, We are going to prove a Great Theorem in Linear Algebra:

The number of independent columns equals the number of independent rows.
The column rank = The row rank

Obviously, $R$’s rank is equal to $C$’s rank, which is $A$’s column rank. Next, we need to prove $R$ is a basis for $A$’s row space:

• check1: check the vectors are independent
• check2: check their combinations produce all three of $A$’s rows

Matrix multiplication in another way:
taking combinatioins of the rows of $R$, and we can get $A$ from the rows of $A$ if row i of $C$ times $R$, take row 1 for example: $\left[\begin{array}{ccc}1& 3& 8\end{array}\right]=1\ast \left[\begin{array}{ccc}1& 0& 2\end{array}\right]+3\ast \left[\begin{array}{ccc}0& 1& 2\end{array}\right]$.
—— The wonderful thing about matrix multiplication is that you can do it a lot of ways, it comes out the same every way, but each way tells you some different things.

So $R$ is a basis of the row space of $A$. The column space and row space of $A$ both have dimension 2, with 2 basis vectors——columns of $C$ and rows of $R$.Proof down.

In conclusion, the proof is exactly to look at the multiplication $CR$ in two ways. First, look at it as combinations of columns of $C$ to give $A$’s columns;Second, look as combinations of rows of $R$ to produce $A$’s rows. So the factorization $A=CR$ is the key idea.

$R=\left[\begin{array}{ccc}1& 0& 2\\ 0& 1& 2\end{array}\right]$ is a famous matrix in linear algebra, called the row reduced-echelon form of $A$, with an identity there and other columns.

A big factorization for data science is “SVD” of A, when the first factor $C$ has $r$ orthogonal columns and the second factor $R$ has $r$ orthogonal rows. ——CUR, to be introduced in follow-up lectures.

How to deal with a matrix of size $10^5$, it is hard to be put into the fast memory.——sample a matrix
$ABCx$ is also in the $C(A)$,cause it’s A times things.

matrix multiplication:

$A$ : $m$ by $n$ matrix, $B$: $n$ by $p$ matrix.
how to see $A$ times $B$:

• dot products of rows and columns —— low level for beginners
• columns of $A$ times rows of $B$ is a high level way:
  $(m,n)(n,p)=(m,p)=sumof(m,1)(1,p)$ ——Rank 1’s matrixes are building blocks.

$col1$ times $row1$ + $colk$ times $rowk$ + $coln$ times $rown$

more in next lecture.

Conclusion:

1. column space of $A$.

2. a factorization: $A=CR$

3. The theorem: column rank = row rank