【学习笔记】吴恩达机器学习 | 第四章 | 多变量线性回归

简要声明

---

1. 课程学习相关网址
   1. Bilibili
   2. 网易云课堂
   3. 学习讲义
2. 由于课程学习内容为英文，文本会采用英文进行内容记录，采用中文进行简要解释。
3. 本学习笔记单纯是为了能对学到的内容有更深入的理解，如果有错误的地方，恳请包容和指正。
4. 非常感谢Andrew Ng吴恩达教授的无私奉献！！！

专有名词

---

feature	特征	Multivariate linear regression	多元线性回归
Feature Scaling	特征缩放	Example automatic convergence test	自动收敛测试
Polynomial regression	多项式回归	Normal equation	正规方程

Multiple features

---

Training set of housing prices

1. Notation
   • n = number of features →训练样本的数量
   • x⁽ⁱ⁾ = input features of i^th training example →第i个训练样本的输入特征值
   • x_j⁽ⁱ⁾ = value of feature j in i^th training example →第i个训练样本中第j个特征值

Hypothesis

$$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots +{\theta }_n{x}_n$$

1. For convenience of notation, define x_0=1 →定义x_0^{(i)}=1
   .

$$h_{\theta }(x)={\theta }_0{x}_0+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots +{\theta }_n{x}_n={\theta }^{T}x$$

1. Multivariate linear regression →多元线性回归

Gradient descent for multiple variables

---

Formulation

• Hypotheis:

$$h_{\theta }(x)={\theta }^{T}x={\theta }_0{x}_0+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots +{\theta }_n{x}_n(x_0=1)$$

• Parameters:

$${\theta }_0,{\theta }_1,\dots ,{\theta }_{n}or\theta (n+1)-dimensionalvector$$

• Cost function:

$$J({\theta }_0,{\theta }_1,\dots ,{\theta }_n)=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2$$

• Gradient descent:

Repeat {

$${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J({\theta }_0,{\theta }_1,\dots ,{\theta }_n)$$

} (simultaneously update for every j = 0, … , n)

Gradient Descent ( n ≥ 1 )

Repeat {

$${\theta }_j:={\theta }_j-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_j^{(i)}$$

} (simultaneously update θ_j for every j = 0, … , n)

---

$${\theta }_0:={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_0^{(i)}$$

$${\theta }_1:={\theta }_1-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_1^{(i)}$$

$${\theta }_2:={\theta }_2-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_2^{(i)}$$

1. 由于x⁽ⁱ⁾_0=1，所以不需要区分n=0和n=1。

Feature Scaling

1. Make sure features are on a similar scale →确保特征都处在一个相近的范围
2. 特征缩放可以使梯度更快收敛
3. Get every feature into approximately a -1 ≤ x_i ≤ 1 range →归一化
4. 不用过于担心特征是否在相同的范围或者区间内 →足够近似-1或1即可
5. Mean normalization 均值归一化 用 x-μ 代替 x 使特征值具有为0的平均值

$$x=\frac{x-\mu }{s}$$

1. μ为特征x的平均值，s为特征x的最大值减最小值或者标准差

Learning rate

1. “Debugging”: How to make sure gradient descent is working correctly →确保梯度下降是正常工作的
2. 曲线的横坐标代表梯度下降算法的迭代次数，纵坐标代表最小化代价函数的值
3. 如果梯度下降算法正常工作的话，每一步迭代之后J(θ)都应该下降，判断算法是否收敛
4. Example automatic convergence test自动收敛测试：如果代价函数J(θ)一步迭代后的下降小于一个很小的值ε，则判断函数已经收敛

1. For sufficiently small α, J(θ) should decrease on every iteration →只要学习率α足够小，那么每次迭代后代价函数J(θ)都会下降
2. But if α is too small, gradient descent can be slow to converge →如果学习率α太小，梯度下降算法会收敛很慢
3. If α is too small: slow convergence →如果学习率α太小，收敛速度慢
   If α is too large: may not decrease on every iteration; may not converge →如果学习率α太大，代价函数可能不会再每次迭代中都下降，甚至可能不收敛
4. To choose α, try …,0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, … →按3的倍数取值

Features and polynomial regression

---

Polynomial regression

$$\begin{gathered} h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_3 \\ ={\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2+{\theta }_3{x}^3 \end{gathered}$$

1. 三个特征范围很大，如果使用梯度下降法，特征缩放非常重要，才能将值的范围变得具有可比性

$$\begin{gathered} h_{\theta }(x)={\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2 \\ {h}_{\theta }(x)={\theta }_0+{\theta }_{1}x+{\theta }_2\sqrt{x} \end{gathered}$$

Normal equation

---

Normal equation

Normal equation: Method to solve for θ analytically →提供一个求θ的解析解法

1. 最小化函数→对函数求导并且将导数置零→得到最小值时的θ
2. 最小化代价函数J的θ值→对逐个θ求J的偏导数并且全部置零

$$\theta =(X^{T}X{)}^{-1}{X}^{T}y$$

1. design matrix 设计矩阵 X 每一行为一个训练样本的转置
2. (X^TX)^-1是X^TX矩阵的逆
3. 如果使用正规方程法，就不需要特征缩放

Gradient Descent vs. Normal Equation

1. 梯度下降法的缺点是需要选择学习速率α，通常需要运行多次
2. 梯度下降法的缺点是需要多次的迭代，计算可能会变慢
3. 正规方程不需要选择学习速率α也不需要迭代
4. 梯度下降法在特征变量很多的情况下能运行得相当好
5. 正规方程需要计算X转置乘以X的逆，是一个n×n的矩阵，如果数量n很大的话计算会很慢
6. 如果n很大（n≥10000）可选择梯度下降法，如果n比较小（n=100 or 1000）可选择正规方程法
7. 正规方程不适用于更复杂的学习算法

Normal equation and non-­‐invertibility

$$\theta =(X^{T}X{)}^{-1}{X}^{T}y$$

1. What if X^TX is non-­‐invertible? (singular/ degenerate) →如果X^TX不可逆怎么办？
2. Redundant features (linearly dependent) →包含了多余的特征（存在线性相关）
3. Too many features ( m ≤ n ) →有太多特征（ m ≤ n ）→ 参数太多样本太少
   • Delete some features →删除一些特征
   • use regularization →使用正则化

吴恩达教授语录

---

• “So, gradient descent is a very useful algorithm to know.”