MIT | 数据分析、信号处理和机器学习中的矩阵方法 笔记系列 Lecture 0: Course Introduction

本系列为MIT Gilbert Strang教授的"数据分析、信号处理和机器学习中的矩阵方法"的学习笔记。

• Gilbert Strang & Sarah Hansen | Sprint 2018
• 18.065: Matrix Methods in Data Analysis, Signal Processing, and Machine Learning
• 视频网址
• Markdown源文件暂未开源，如有需要可联系 981591477@qq.com
• 笔记难免存在问题，欢迎联系上述邮箱指正
• 关注下面的公众号，回复“ 矩阵方法 ”，即可获取 本系列完整的pdf笔记文件~

---

Lecture 0: Course Introduction
Lecture 1 The Column Space of AAA Contains All Vectors AxAxAx
Lecture 2 Multiplying and Factoring Matrices
Lecture 3 Orthonormal Columns in QQQ Give Q′Q=IQ’Q=IQ′Q=I
Lecture 4 Eigenvalues and Eigenvectors
Lecture 5 Positive Definite and Semidefinite Matrices
Lecture 6 Singular Value Decomposition (SVD)
Lecture 7 Eckart-Young: The Closest Rank kkk Matrix to AAA
Lecture 8 Norms of Vectors and Matrices
Lecture 9 Four Ways to Solve Least Squares Problems
Lecture 10 Survey of Difficulties with Ax=bAx=bAx=b
Lecture 11 Minimizing ||x|| Subject to Ax=bAx=bAx=b
Lecture 12 Computing Eigenvalues and Singular Values
Lecture 13 Randomized Matrix Multiplication
Lecture 14 Low Rank Changes in AAA and Its Inverse
Lecture 15 Matrices A(t)A(t)A(t) Depending on ttt, Derivative = dA/dtdA/dtdA/dt
Lecture 16 Derivatives of Inverse and Singular Values
Lecture 17 Rapidly Decreasing Singular Values
Lecture 18 Counting Parameters in SVD, LU, QR, Saddle Points
Lecture 19 Saddle Points Continued, Maxmin Principle
Lecture 20 Definitions and Inequalities
Lecture 21 Minimizing a Function Step by Step
Lecture 22 Gradient Descent: Downhill to a Minimum
Lecture 23 Accelerating Gradient Descent (Use Momentum)
Lecture 24 Linear Programming and Two-Person Games
Lecture 25 Stochastic Gradient Descent
Lecture 26 Structure of Neural Nets for Deep Learning
Lecture 27 Backpropagation: Find Partial Derivatives
Lecture 28 Computing in Class [No video available]
Lecture 29 Computing in Class (cont.) [No video available]
Lecture 30 Completing a Rank-One Matrix, Circulants!
Lecture 31 Eigenvectors of Circulant Matrices: Fourier Matrix
Lecture 32 ImageNet is a Convolutional Neural Network (CNN), The Convolution Rule
Lecture 33 Neural Nets and the Learning Function
Lecture 34 Distance Matrices, Procrustes Problem
Lecture 35 Finding Clusters in Graphs
Lecture 36 Alan Edelman and Julia Language

---

0 Course Introduction of 18.065 by Professor Strang

主要内容 (Topics)

• There are really 2 essential topics and 2 supplementary (supporting subjects):
  • T1: Linear Algebra
  • S1: Probability and Statistics
  • S2: Optimization
  • T2: Deep Learning

Focus on 矩阵

• Topic 1: Linear Algebra
  • 1 some of the best matrices
    • Symmetric Matrices
    • Orthogonal Matrices
    • and their relations
    • 他们是stars of linear algebra
  • 2 将其他矩阵转换为上述两种矩阵
    • Key step: matrix factorization (矩阵分解)
      • e.g., factor into symmetric times orthogonal matrix
      • e.g., orthogonal times diagonal times orthogonal matrix
        • SVD: 奇异值分解 Singular Value Decomposition (SVD)

矩阵在其他Topics中的角色

• Topic 2: Deep Learning
  • Deep Learning: Creat Function
    • 例如，识别、检测、重建、生成等不同tasks
    • so, learning a function
      • take the data as input and output the result
  • What is the function like?
    • involves matrix multiplication
      • just a linear function
    • Deep learning的function 显然不止是线性函数
      • 两个matrix中间可能会有非线性函数
        • e.g., f(x) = x when x is positive, else f(x) = 0
        • (激活函数)
  • 但Deep Learning 找到矩阵一定是非常重要的！
• Supporting Subjects 1: Optimization
  • We have to find the entries in those matrices that go into the learning function
    • a crucial step
  • A problem minimizing the error with all those matrix entries as variables
  • $\Rightarrow$ Multivariable calculus, like 1000000
    • unthinkable in a basic calculus course, 但深度学习中的确是这样
    • 但也能得到amazing results
• Supporting Subjects: Probalility and Statistics
  • when you are multiplying a whole sequence of matrices in deep learning
  • very possible for the numbers to be 爆炸(特别大) 或 消失(特别小)
    • both are bad for learning function
  • 因此，需要 Keep the mean and variance at the right spot!
    • to keep those numbers, those matrices in a good range
  • 本节课不会讲解Statistics, 但是会用到 Statistics

An interview with Gilbert Strang on the course

adventure 冒险

• This was my adventure into the subject of deep learning

pedagogically 教学法上地

• What did that feel like to try something new pedagogically?
  • (指课程变得student-led and project-based)
    • 事实上主导和基于项目就是人类学习的过程
    • a very effective way to learn
    • 还能学到how to presenting
    • 老师的工作是帮助学生学习或和学生一起学习 – grade不重要