Chapter 1 Introduction to Vectors (Introduction to Linear Algebar written by Dr. Gilber Strang)

今天开始重新学习线性代数，从数据分析、机器学习的角度，争取今后可以看懂专业论文。其实，大学时光是学过的，但加上遗忘和本身是从市政工程的角度学的，目前只记得一些基本的操作，更深的就忘记了。

Introduction to Linear Algebar (Fifth edition) written by Dr. Gilber Strang.

The really impressive feature of linear algebra is how smoothly is takes that step into n-dimensional space. Your mental picture stays completely correct, even if drawing a ten-dimensional vector is impossible.
第二句话提供了想象高维空间操作的方法，即从二维、三维向量操作的空间想象，来窥视高维空间的操作。简单的，应该可以取几个高维向量的前2~3个tuples的组成的向量，想象加法操作，来进一步读懂高维的相关操作。这样就不会被挡在高维空间无法想象这个障碍前面了，这是一个线性代数的重要思想方法。

1.1 Vectors and Linear Combinations

1. The combinations cu + dv + ew fill three-dimensional space. But if the w happens to be cu + dv, that third vector w is in the plane of the first two. The combinations of u, v , w will not go outside that uv plane. We do not get the full three-dimensional space.
2. ${\mathrm{I}\mathrm{R}}^n$ is the set of all n-tuples of real numbers.
   这解释了$\mathrm{I}\mathrm{R}$代表的是Real numbers.
3. In applying mathematics, many problems have two parts:
   (1) Modelling part Express the problem by a set of equations.
   (2) Computational part Solve those equations by a fast and accurate algorithm.
   Find $n$ numbers $c_1,...,c_n$ so that $c_1{\mathbf{v}}_1+...+c_n{\mathbf{v}}_n=\mathbf{b}$.
   The elimination method in Chapter 2 succeeds far beyond n = 1000. For n greater than 1 billion, see Gaussian elimination method in Chapter 11.
4. When climbing to Chapter 6, remember to use the new method of calculating eigenvectors which is from three physicists. The method can be found at Neutrinos Lead to Unexpected Discovery in Basic Math.
   学习感悟：慢慢来，弄明白，当小说，深修行。

1.2 Lengths and Dot Products

Dot Product or Inner Product can be used to describe the weight balance and the total income from trading.
In Microsoft Excel, the function sumproduct can be used to calculate dot product.

Definition:
The length of $\Vert \mathbf{v}\Vert$ of a vector is the square root of $\mathbf{v}\cdot \mathbf{v}:$
length = $\Vert \mathbf{v}\Vert =\sqrt{\mathbf{v}\cdot \mathbf{v}}=(v_1^2+v_2^2+\cdots +v_n^2{)}^{1/2}$

Definition:
A unit vector $\mathbf{u}$ is a vector whose length equals one. Then $\mathbf{u}\cdot \mathbf{u}=1$
According to the Q&A of Quora, the unit vector can be used for the fast calculation among vectors by converting vectors to the line combinations of perpendicular unit vectors. The transformation can separate the scalar calculation and the directionality.

Unit Vector:
$u=\frac{v}{\Vert v\Vert }$ is a unit vector in the same direction as $v$.

Right angles:
The dot product is $v\cdot w=0$ when $v$ is perpendicular to $w$.

Unit vectors $u$ and $U$ at angle $\theta$ have $u\cdot U=cos\theta .$ Certainly $|u\cdot U|\le 1.$

---

The dot product reveals the exact angle $\theta$.

$$cos\theta =\frac{\mathbf{v}\cdot \mathbf{w}}{\Vert \mathbf{v}\Vert \Vert \mathbf{w}\Vert }$$

Schwarz inequality($|v\cdot w|\le \Vert v\Vert \Vert w\Vert$), more correctly named as Cauchy-Schwarz-Buniakowsky inequality which was found in France and Germany and Russis, is the most important inequality in mathematics.

Triangle Inequality: $\Vert v+w\Vert \le \Vert v\Vert +\Vert w\Vert$

geometric mean $\le$ arithmetic mean:
$\sqrt{xy}\le \frac{x+y}{2}$
$\sqrt[3]{xyz}\le \frac{x+y+z}{3}$(How to prove it???)

---

Problem 1:
Pick any numbers that add to $x+y+z=0$. Find the angle between your vector $v=(x,y,z)$ and the vector $w=(z,x,y)$. Challenge question: Explain why $\frac{v\cdot w}{\Vert v\Vert \Vert w\Vert }$ is always $-\frac{1}{2}$.
Step 1: $(x+y+z{)}^2=x^2+y^2+z^2+2xy+2xz+2yz=0\Rightarrow x^2+y^2+z^2=-2(xy+xz+yz)$
Step 2: $$\frac{v\cdot w}{\Vert v\Vert \Vert w\Vert }=\frac{xz+xy+yz}{\sqrt{x^2+y^2+z^2}\sqrt{x^2+y^2+z^2}}=\frac{xz+xy+yz}{x^2+y^2+z^2}=\frac{xz+xy+yz}{-2(xy+xz+yz)}=-\frac{1}{2}$$

Problem 2:
Using $v=randn(3,1)$ in MATLAB, create a random unit vector $u=\frac{v}{\Vert v\Vert }$. Using $V=randn(3,30)$ create 30 more random unit vectors $U_j$. What is the average size of the dot products $|u\cdot U_j|$? In calculus, the average is ${\int }_0^{\pi }|cos\theta |d_{\theta }/\pi =2/\pi$.
In the OCTAVE, I did not correctly answer this question. Because I have not use the MATLAB or OCTAVE for long time, so I tried to answer this question using numpy. The following code is:

>>> import numpy as np
>>> v = np.random.random((1,3))
>>> u = v/np.linalg.norm(v,2)
>>> V = np.random.random((30,3))
>>> D = np.apply_along_axis(np.linalg.norm, 1, V, 2).reshape(30,1)
>>> U = V/D
>>> np.average(np.abs(U.dot(u[0])))
0.8702124234966323
>>> 2/np.pi
0.6366197723675814

I do not know whether my answer is right…

1.3 Matrices

A crucial change in viewpoint about the combination of the vectors and the matrix times vector(Combination of columns).
$$x_1\left[\begin{array}{c}1\\ -1\\ 0\end{array}\right]+x_2\left[\begin{array}{c}0\\ 1\\ -1\end{array}\right]+x_3\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]=\left[\begin{array}{c}{x}_1\\ x_2-x_1\\ x_3-x_2\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ -1& 1& 0\\ 0& -1& 1\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]$$
$$x_1\mathbf{u}+x_2\mathbf{v}+x_3\mathbf{w}=A\mathbf{x}$$

$A\mathbf{x}$ is also dot products with rows:
$$A\mathbf{x}=\left[\begin{array}{ccc}1& 0& 0\\ -1& 1& 0\\ 0& -1& 1\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]=\left[\begin{array}{c}(1,0,0)\cdot (x_1,x_2,x_3)\\ (-1,1,0)\cdot (x_1,x_2,x_3)\\ (0,-1,1)\cdot (x_1,x_2,x_3)\end{array}\right]$$
The difference matrix can find all the differences among the number in vector $\mathbf{x}$.
$$\left[\begin{array}{ccc}1& 0& 0\\ -1& 1& 0\\ 0& -1& 1\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]=\left[\begin{array}{c}{x}_1\\ x_2-x_1\\ x_3-x_2\end{array}\right]$$

$$\left[\begin{array}{cccc}1& 0& 0& 0\\ -1& 1& 0& 0\\ 0& -1& 1& 0\\ 0& 0& -1& 1\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\\ x_4\end{array}\right]=\left[\begin{array}{c}{x}_1\\ x_2-x_1\\ x_3-x_2\\ x_4-x_3\end{array}\right]$$

$Cx=b:$
$$\left[\begin{array}{c}{x}_1-x_3\\ x_2-x_1\\ x_3-x_2\end{array}\right]=\left[\begin{array}{c}1\\ 3\\ 5\end{array}\right]$$
Left sides add to 0
Right sides add to 9
No solution $x_1$, $x_2$, $x_3$

Independence and Dependence

Independent columns $\Rightarrow$ Invertable matrix
Dependent columns $\Rightarrow$ Singular matrix

$u=\left[\begin{array}{c}1\\ -1\\ 0\end{array}\right]$ $v=\left[\begin{array}{c}0\\ 1\\ -1\end{array}\right]$ $w=\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]$ $w^{\ast }=\left[\begin{array}{c}-1\\ 0\\ 1\end{array}\right]$

$A=[u,v,w]$ $C=[u,v,w^{\ast }]$

For matrix $A$, $w$ is not in the plane of $u$ and $v$. The sum of elements in $w$ and $v$ is all equal to 0, but the sum of elements in $w$ is equal to 1.
For matrix $C$, $w^{\ast }$ is in the plane of $u$ and $v$. The sum of elements in $w$ ,$v$ and $w^{\ast }$ is all equal to 0

$u,v,w$ are independent . No combination except $0u+0v+0w=0$ gives $b=0.$
$u,v,w^{\ast }$ are dependent . Other combinations like $u+v+w^{\ast }$ give $b=0.$

Independent columns: $Ax=0$ has one solution. $A$ is an invertible matrix.
Dependent columns: $Cx=0$ has many solutions. $C$ is a singular matrix.