Mathematics Basics - Linear Algebra (Matrix Part 1)

Matrix

Intro

A lot of ideas in matrix are related to what we have already learned in vector. Recall that we can express a simultaneous equation as multiplication of a matrix and a vector.

$$\begin{gathered} 2a+3b=12 \\ 5a+b=17 \end{gathered}$$

$$\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)\ast \left(\begin{array}{c}a\\ b\end{array}\right)=\left(\begin{array}{c}12\\ 17\end{array}\right)$$

In this chapter, we are going to look under the hood to understand what exactly happens during this multiplication operation. It’s intriguing to see how powerful matrices are in terms of transforming vectors. This concept is also at the heart of linear algebra.

We will then dive into the solution for simultaneous equations by calculating matrix inverse and determinant. Understanding matrix inverse then unlocks the way we convert a vector space in general. As compared to what we have learned previously on changing basis, now we are able to convert a vector to any space without being restricted to a space of orthogonal basis vectors.

We are also going to look at two interesting extensions of matrix. One is called orthogonal matrix. It is the most convenient kind of matrix when we need to perform matrix inverse operation. We will construct such a matrix by the Gram-Schmidt process. The other extension is Eigen vectors and Eigen values. Eigen vectors have some special properties during matrix transformation. Knowing the Eigen vectors for a matrix will also make repeated multiplication of this matrix by itself much simpler.

Let’s get started!

Matrix as a Transformation

We start by showing a matrix $\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)$ multiplied by the basis vector $\left(\begin{array}{c}1\\ 0\end{array}\right)$ and $\left(\begin{array}{c}0\\ 1\end{array}\right)$.

$$\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)\ast \left(\begin{array}{c}1\\ 0\end{array}\right)=\left(\begin{array}{c}2\\ 5\end{array}\right)$$

$$\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)\ast \left(\begin{array}{c}0\\ 1\end{array}\right)=\left(\begin{array}{c}3\\ 1\end{array}\right)$$

To read this result, we can see matrix $\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)$ transforms vector $\left(\begin{array}{c}1\\ 0\end{array}\right)$ into a new vector $\left(\begin{array}{c}2\\ 5\end{array}\right)$ and transforms vector $\left(\begin{array}{c}0\\ 1\end{array}\right)$ into a new vector $\left(\begin{array}{c}3\\ 1\end{array}\right)$. Graphically, we get below transformation from $e_1$ to $e_1^{\prime }$ and from $e_2$ to $e_2^{\prime }$.

We learned from previous chapter on vector that $e_1$ and $e_2$ describe a basis vector space. After being multiplied by a matrix $\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)$, the original vector space changes to a new one described by $e_1^{\prime }$ and $e_2^{\prime }$. If we call $e_1$ and $e_2$ our input vectors and $e_1^{\prime }$ and $e_2^{\prime }$ our output vectors, what matrix multiplication does therefore is equivalent to a vector space transformation.

This part is tricky, but very critical for us to form a right intuition of matrix multiplication. Let’s do a matrix multiplication in a step-by-step manner to illustrate this idea of vector space transformation.

$$\begin{array}{rl}\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)\ast \left(\begin{array}{c}3\\ 2\end{array}\right)& =\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)\ast \left(3\ast \left(\begin{array}{c}1\\ 0\end{array}\right)+2\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)\ast (3\ast e_1+2\ast e_2)\\ & =3\ast \left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)\ast e_1+2\ast \left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)\ast e_2\\ & =3\ast \left(\begin{array}{c}2\\ 5\end{array}\right)+2\ast \left(\begin{array}{c}3\\ 1\end{array}\right)\\ & =3\ast e_1^{\prime }+2\ast e_2^{\prime }\end{array}$$

Note here we make use of the associative and distributive properties of matrix multiplication.

The vector $\left(\begin{array}{c}3\\ 2\end{array}\right)$ is originally expressed in basis vectors $e_1$ and $e_2$. After being multiplied by matrix $\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)$, we have projected it onto the new basis vectors $e_1^{\prime }$ and $e_2^{\prime }$, i.e., $3e_1^{\prime }+2e_2^{\prime }$. So we can think of matrix multiplication as the vector sum of the transformed basis vectors. Matrix $\left(\begin{array}{cc}2& 3\\ 5& 1\end{array}\right)$ tells us where the original basis vectors $e_1$ and $e_2$ should go after the transformation. In addition, we can see that columns of the transformation matrix are just the new basis vectors, $e_1^{\prime }=\left(\begin{array}{c}2\\ 5\end{array}\right)$, $e_2^{\prime }=\left(\begin{array}{c}3\\ 1\end{array}\right)$.

If we treat matrix as a way to transform vector space, how many ways can we transform it? Next, we will discuss some possible matrix transformation in a 2-dimensional space.

Identity Matrix

$\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)$ is an identity matrix. It keeps the vector space unchanged. That means a vector is not altered at all when it is multiplied by an identity matrix. We can look at an example of vector $\left(\begin{array}{c}3\\ 2\end{array}\right)$.

$$\begin{array}{rl}\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)\ast \left(\begin{array}{c}3\\ 2\end{array}\right)& =\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)\ast \left(3\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)\ast \left(2\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =3\ast \left(\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+2\ast \left(\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =3\ast \left(\begin{array}{c}1\\ 0\end{array}\right)+2\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\\ & =\left(\begin{array}{c}3\\ 2\end{array}\right)\end{array}$$

Scaling Matrix

Scaling matrix has this form $\left(\begin{array}{cc}a& 0\\ 0& b\end{array}\right)$. It alters the area of basis vector space by a factor of $a\ast b$. Let’s look at an example below.

$$\begin{array}{rl}\left(\begin{array}{cc}5& 0\\ 0& 7\end{array}\right)\ast \left(\begin{array}{c}x\\ y\end{array}\right)& =\left(\begin{array}{cc}5& 0\\ 0& 7\end{array}\right)\ast \left(x\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+\left(\begin{array}{cc}5& 0\\ 0& 7\end{array}\right)\ast \left(y\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\left(\begin{array}{cc}5& 0\\ 0& 7\end{array}\right)\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+y\ast \left(\left(\begin{array}{cc}5& 0\\ 0& 7\end{array}\right)\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\begin{array}{c}5\\ 0\end{array}\right)+y\ast \left(\begin{array}{c}0\\ 7\end{array}\right)\end{array}$$

The basis vectors for $\left(\begin{array}{c}x\\ y\end{array}\right)$ have shifted from $\left(\begin{array}{c}1\\ 0\end{array}\right)$ and $\left(\begin{array}{c}0\\ 1\end{array}\right)$ to $\left(\begin{array}{c}5\\ 0\end{array}\right)$ and $\left(\begin{array}{c}0\\ 7\end{array}\right)$. This stretches the vector space from a 1X1 square to a 5X7 rectangle as shown in the graph below.

Similarly, we can also have a scaling matrix with fractional values. This squashes the original vector space into a smaller one.

Flipping Matrix

We can use matrix $\left(\begin{array}{cc}-1& 0\\ 0& 1\end{array}\right)$ to flip the horizontal basis vector $\left(\begin{array}{c}1\\ 0\end{array}\right)$ to the other side. Here is an illustration.

$$\begin{array}{rl}\left(\begin{array}{cc}-1& 0\\ 0& 1\end{array}\right)\ast \left(\begin{array}{c}x\\ y\end{array}\right)& =\left(\begin{array}{cc}-1& 0\\ 0& 1\end{array}\right)\ast \left(x\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+\left(\begin{array}{cc}-1& 0\\ 0& 1\end{array}\right)\ast \left(y\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\left(\begin{array}{cc}-1& 0\\ 0& 1\end{array}\right)\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+y\ast \left(\left(\begin{array}{cc}-1& 0\\ 0& 1\end{array}\right)\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\begin{array}{c}-1\\ 0\end{array}\right)+y\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\end{array}$$

Graphically, this flips the basis vectors along the vertical axis.

Similarly, we can use matrix $\left(\begin{array}{cc}1& 0\\ 0& -1\end{array}\right)$ to flip the basis vectors along the horizontal axis.

$$\begin{array}{rl}\left(\begin{array}{cc}1& 0\\ 0& -1\end{array}\right)\ast \left(\begin{array}{c}x\\ y\end{array}\right)& =\left(\begin{array}{cc}1& 0\\ 0& -1\end{array}\right)\ast \left(x\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+\left(\begin{array}{cc}1& 0\\ 0& -1\end{array}\right)\ast \left(y\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\left(\begin{array}{cc}1& 0\\ 0& -1\end{array}\right)\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+y\ast \left(\left(\begin{array}{cc}1& 0\\ 0& -1\end{array}\right)\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\begin{array}{c}1\\ 0\end{array}\right)+y\ast \left(\begin{array}{c}0\\ -1\end{array}\right)\end{array}$$

Inverse Matrix

When we flip the basis vectors in both vertical and horizontal direction, we get a new vector space that has inverted the values in both basis vectors. This inverse matrix is $\left(\begin{array}{cc}-1& 0\\ 0& -1\end{array}\right)$.

$$\begin{array}{rl}\left(\begin{array}{cc}-1& 0\\ 0& -1\end{array}\right)\ast \left(\begin{array}{c}x\\ y\end{array}\right)& =\left(\begin{array}{cc}-1& 0\\ 0& -1\end{array}\right)\ast \left(x\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+\left(\begin{array}{cc}-1& 0\\ 0& -1\end{array}\right)\ast \left(y\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\left(\begin{array}{cc}-1& 0\\ 0& -1\end{array}\right)\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+y\ast \left(\left(\begin{array}{cc}-1& 0\\ 0& -1\end{array}\right)\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\begin{array}{c}-1\\ 0\end{array}\right)+y\ast \left(\begin{array}{c}0\\ -1\end{array}\right)\end{array}$$

Diagonal Mirroring Matrix

We can also do a mirroring of vector space along the 45° line with matrix $\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)$.

$$\begin{array}{rl}\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)\ast \left(\begin{array}{c}x\\ y\end{array}\right)& =\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)\ast \left(x\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)\ast \left(y\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+y\ast \left(\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\begin{array}{c}0\\ 1\end{array}\right)+y\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\end{array}$$

The basis vector $\left(\begin{array}{c}1\\ 0\end{array}\right)$ is shifted to $\left(\begin{array}{c}0\\ 1\end{array}\right)$ and basis vector $\left(\begin{array}{c}0\\ 1\end{array}\right)$ is shifted to $\left(\begin{array}{c}1\\ 0\end{array}\right)$ as shown in the graph.

If we combine the inverse matrix and diagonal mirroring matrix, we would get a mirroring of vector space along the -45° line. This matrix is $\left(\begin{array}{cc}0& -1\\ -1& 0\end{array}\right)$.

$$\begin{array}{rl}\left(\begin{array}{cc}0& -1\\ -1& 0\end{array}\right)\ast \left(\begin{array}{c}x\\ y\end{array}\right)& =\left(\begin{array}{cc}0& -1\\ -1& 0\end{array}\right)\ast \left(x\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)\ast \left(y\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\left(\begin{array}{cc}0& -1\\ -1& 0\end{array}\right)\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+y\ast \left(\left(\begin{array}{cc}0& -1\\ -1& 0\end{array}\right)\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\begin{array}{c}0\\ -1\end{array}\right)+y\ast \left(\begin{array}{c}-1\\ 0\end{array}\right)\end{array}$$

Sheering Matrix

Our basis vectors can also be transformed in more interesting ways like sheering when the vector space is no longer square or rectangle. Let’s see an example of sheering transformation matrix $\left(\begin{array}{cc}1& 1\\ 0& 1\end{array}\right)$.

$$\begin{array}{rl}\left(\begin{array}{cc}1& 1\\ 0& 1\end{array}\right)\ast \left(\begin{array}{c}x\\ y\end{array}\right)& =\left(\begin{array}{cc}1& 1\\ 0& 1\end{array}\right)\ast \left(x\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+\left(\begin{array}{cc}1& 1\\ 0& 1\end{array}\right)\ast \left(y\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\left(\begin{array}{cc}1& 1\\ 0& 1\end{array}\right)\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+y\ast \left(\left(\begin{array}{cc}1& 1\\ 0& 1\end{array}\right)\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\begin{array}{c}1\\ 0\end{array}\right)+y\ast \left(\begin{array}{c}1\\ 1\end{array}\right)\end{array}$$

The new vector space after transformation is a parallelogram. Sheering transformation is not limited to just one basis vector. We can do a sheering transform for both $e_1$ and $e_2$ at the same time.

Rotation Matrix

Rotation is another common type of transformation. We can rotate the basis vectors by 90° anti-clockwise using the transformation matrix $\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)$.

$$\begin{array}{rl}\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)\ast \left(\begin{array}{c}x\\ y\end{array}\right)& =\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)\ast \left(x\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)\ast \left(y\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)\ast \left(\begin{array}{c}1\\ 0\end{array}\right)\right)+y\ast \left(\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)\ast \left(\begin{array}{c}0\\ 1\end{array}\right)\right)\\ & =x\ast \left(\begin{array}{c}0\\ 1\end{array}\right)+y\ast \left(\begin{array}{c}-1\\ 0\end{array}\right)\end{array}$$

A more general case to rotate the basis vectors by any angle θ anti-clockwise is given by $\left(\begin{array}{cc}\cos \theta & \sin \theta \\ -\sin \theta & \cos \theta \end{array}\right)$.

Composite Transformation Matrix

It is also possible to apply transformation on the basis vectors more than once. For example, we can first do a 90° anti-clockwise rotation using $\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)$, followed by a flipping along the vertical axis using $\left(\begin{array}{cc}-1& 0\\ 0& 1\end{array}\right)$. The composite transformation is just the matrix product of the two basic transformation matrix.
$$\left(\begin{array}{cc}-1& 0\\ 0& 1\end{array}\right)\ast \left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)=\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)$$

We can add on more matrices to the expression for more transformation steps. Note the order of operation in matrix multiplication matters. This is because matrix multiplications are not commutative, i.e. $\left(\begin{array}{cc}-1& 0\\ 0& 1\end{array}\right)\ast \left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)\ne \left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)\ast \left(\begin{array}{cc}-1& 0\\ 0& 1\end{array}\right)$. The later transformation matrix should always precede the earlier transformation matrix in multiplication expression.

Matrix transformation is a very important step in a lot of machine learning applications, especially image-related tasks. One good example is face recognition problem whereby you need to center and align the subject faces in the image captured.

---

(Inspired by Mathematics for Machine Learning lecture series from Imperial College London)