MIT | 数据分析、信号处理和机器学习中的矩阵方法 笔记系列: Lecture 8 Norms of Vectors and Matrices

本系列为MIT Gilbert Strang教授的"数据分析、信号处理和机器学习中的矩阵方法"的学习笔记。

• Gilbert Strang & Sarah Hansen | Sprint 2018
• 18.065: Matrix Methods in Data Analysis, Signal Processing, and Machine Learning
• 视频网址: https://ocw.mit.edu/courses/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/
• 关注 下面的公众号，回复“ 矩阵方法 ”，即可获取 本系列完整的pdf笔记文件~

---

内容在CSDN、知乎和微信公众号同步更新

• CSDN博客
• 知乎
• 微信公众号
• qq邮箱: 981591477

• Markdown源文件暂未开源，如有需要可联系邮箱
• 笔记难免存在问题，欢迎联系邮箱指正

---

Lecture 0: Course Introduction

Lecture 1 The Column Space of $A$ Contains All Vectors $Ax$

Lecture 2 Multiplying and Factoring Matrices

Lecture 3 Orthonormal Columns in $Q$ Give $Q^{\prime }Q=I$

Lecture 4 Eigenvalues and Eigenvectors

Lecture 5 Positive Definite and Semidefinite Matrices

Lecture 6 Singular Value Decomposition (SVD)

Lecture 7 Eckart-Young: The Closest Rank $k$ Matrix to $A$

Lecture 8 Norms of Vectors and Matrices

Lecture 9 Four Ways to Solve Least Squares Problems

Lecture 10 Survey of Difficulties with $Ax=b$

Lecture 11 Minimizing ||x|| Subject to $Ax=b$

Lecture 12 Computing Eigenvalues and Singular Values

Lecture 13 Randomized Matrix Multiplication

Lecture 14 Low Rank Changes in $A$ and Its Inverse

Lecture 15 Matrices $A(t)$ Depending on $t$, Derivative = $dA/dt$

Lecture 16 Derivatives of Inverse and Singular Values

Lecture 17 Rapidly Decreasing Singular Values

Lecture 18 Counting Parameters in SVD, LU, QR, Saddle Points

Lecture 19 Saddle Points Continued, Maxmin Principle

Lecture 20 Definitions and Inequalities

Lecture 21 Minimizing a Function Step by Step

Lecture 22 Gradient Descent: Downhill to a Minimum

Lecture 23 Accelerating Gradient Descent (Use Momentum)

Lecture 24 Linear Programming and Two-Person Games

Lecture 25 Stochastic Gradient Descent

Lecture 26 Structure of Neural Nets for Deep Learning

Lecture 27 Backpropagation: Find Partial Derivatives

Lecture 28 Computing in Class [No video available]

Lecture 29 Computing in Class (cont.) [No video available]

Lecture 30 Completing a Rank-One Matrix, Circulants!

Lecture 31 Eigenvectors of Circulant Matrices: Fourier Matrix

Lecture 32 ImageNet is a Convolutional Neural Network (CNN), The Convolution Rule

Lecture 33 Neural Nets and the Learning Function

Lecture 34 Distance Matrices, Procrustes Problem

Lecture 35 Finding Clusters in Graphs

Lecture 36 Alan Edelman and Julia Language

---

---

Lecure 8: Norms of Vectors and Matrices

• 一个与本课程无关的问题(现象)：Probability Matching

  • Biased Coin: 但是参与者不知道 (无先验)

    ▪ 75% likely to produce heads

    ▪ 25% likely to produce tails

  • payoff 1 guess right or 1 guess wrong

  • Optimal strategy: 刚开始会根据之前的经验猜测都是 50%, 但尝试过程中会慢慢优化 $\Rightarrow$ 最终，to guess heads all the time

  • 但人类实际上会：guess heads three quarters of the time and tails one quarter of the time

  • 该问题说明了: there is good math questions everywhere, 与本节课无关

• 下面开始本节课的内容: Norms

8.1 Vectors norms $\Vert v{\Vert }_p$

• $\Vert v{\Vert }_p=(|v_1{|}^p+|v_2{|}^p+...+|v_n{|}^p{)}^{1/p}$

• Norm is a way to measure

• the size of a vector / matrix / tensor

• p = 2

  • $\sqrt{v_1^2+v_2^2+...+v_n^2}$

• p = 1

  • $|v_1|+|v_2|+...+|v_n|$
  • some things really work best in the $L_1$ norm

• p = $\infty$

  • $max|v_i|$
  • as increasing p, whichever one is biggest is going to take over

• p = 0

  • $\Vert v{\Vert }_0$ is the number of non-zero components

  • important in question of sparsity

    🚩 you might want to min $\Vert v{\Vert }_0$ $\Rightarrow$ to get sparse vectors $\Rightarrow$ 以加快运算速度

  • Note: $\Vert v{\Vert }_0$ is not a norm! $\Rightarrow$ $2\Vert v{\Vert }_0\ne 2\Vert v{\Vert }_0$ $\Rightarrow$ violates violates 违反 违背 the rule for a norm

• The Geometry of a norm

  • plots the norm in 2D space ${\mathbb{R}}^2$

    ✅ 如下图

  • 哪些范数适合优化？True norm: convex unit ball $\Vert v\Vert \le 1$

    🚩 $L_1$ 优化的时候 NOT Convex!

    🚩 $L_2$: Convex, 使优化求解稳定快速

• p = S

  • S: means a positive definite symmetric matrix

  • $\Vert v{\Vert }_S=\sqrt{v^{T}Sv}$

  • $\sqrt{\cdot }$的意义：使$\Vert 2v{\Vert }_S=2\times$ $\Vert v{\Vert }_S$ $\Rightarrow$ grow linearly

  • What is the shape of $\Vert v{\Vert }_S=\sqrt{v^{T}Sv}=1$? (S: symmetric positive definite)

    ✅ let $S=\left[\begin{array}{cc}2& 0\\ 0& 3\end{array}\right]$

    ✅ $\Rightarrow$ $v^{T}Sv=2v_1^2+3v_2^2=1$

    🚩 S范数的意义：get a new picture: a new norm that is kind of adjustable $\Rightarrow$ a weighted norm $\Rightarrow$ pixk some numbers sort of appropriate to the particular problem

    🚩 the shape is an ellipse

8.2 Optimal Problem

• A simple example:

  • $min$ $\Vert x{\Vert }_1$ or $\Vert x{\Vert }_2$

    🚩 Ax = b (e.g., $a_1{x}_1+a_2{x}_2=b$)

  • L1: a famous name basis pursuit 基底追踪?

  • L2: Ridge regression

  • 下图，可见$L_1$ 倾向于稀疏结果 (可从geometry 角度解释);

8.3 Matrix Norms

• $\Vert A{\Vert }_2$ = ${\sigma }_1$

  • the largest singular
  • How to connect it to 2 norms of vectors?
  • Matrix norm from vector norm = max blowup
  • $||A|{|}_2=ma{x}_{\forall x}\frac{\Vert Ax{\Vert }_2}{||x|{|}_2}$

• How to prove : $||A|{|}_2=ma{x}_{\forall x}\frac{\Vert Ax{\Vert }_2}{||x|{|}_2}$ $={\sigma }_1$ ?

  • when x is the singular vector $v_1$ (eigenvectors of $A^{T}A$)
  • $\Vert A{v}_1\Vert /\Vert v_1\Vert =\Vert A{v}_1\Vert /1=\Vert A{v}_1\Vert =\Vert {\sigma }_1{u}_1\Vert ={\sigma }_1$

• $\Vert A{\Vert }_F=\sqrt{addall|a_{ij}{|}^2}=\sqrt{{\sigma }_1^2+...+{\sigma }_r^2}$

  • Frobenius norm
  • In SVD: $A=U\Sigma V^T$ $\Rightarrow$ Orthogonal matrix $U,V$ does not change any of these particular norms $\Rightarrow$ $\Vert \Sigma {\Vert }_F=\Vert A{\Vert }_F$

• $\Vert A{\Vert }_N$

  • Nuclear norm: ${\sigma }_1+{\sigma }_2+...+{\sigma }_r$

  • 在Deep Learning中的应用：

    🚩 conjecture 推测 – in a typical deep learning problem, there are many more weights than samples, and so there are a lot of possible minima – many different weights give the same minimum loss $\Rightarrow$ too many parameters (但也不见得是坏事，even part of the sucess)

    🚩 in a model situation, 梯度下降法 picks out the weights that minimize the nuclear norm $\Rightarrow$ 🚩 a norm of a lot of weights

  • 在压缩感知中也有应用

This lecture: about the norms

Next lecture: least squares problems

本节小结

• 向量范数: $L_1$, $L_2$ norm等
• 向量范数 geometry，范数的意义，几种范数用于优化时的特点
• 矩阵范数