Linear Ragression with multiple variables
Multiple features
Hypothesis:
Gradient descent for multiple variables
Hypothesis:
Parameters: θ⃗ =[θ0,θ1,θ2,⋯,θn]
Cost function:
Practice I : Feature Scaling
Feature Scaling
E.g. x1=size(0−2000feet2)==>x1=size(feet2)2000
x2=numberofbedrooms(1−5)==>x2=number_of_bedrooms5
Mean normalization
E.g. x1=size(feet2)−10002000
Practice II : Learning rate
J(θ) should decrease after every iteration.
Gradient descent not working, Use smaller α .
Summary
- if α is too small : slow convergence.
- if α is too large : J(θ) may not decrease on every iteration; may not converge.
Features and polynomial regression
Feature scaling !!
Normal equation
Normal equation: Method to solve for θ analytically.
Intuition:
θ∈R J(θ0,θ1,θ2,⋯,θn)=12m∑mi=1(hθ(x(i))−y(i))2
∂∂θjJ(θ)=⋯=0 (for every j)
Solve for θ0,θ1,θ2,⋯,θn
So: (not use Features scaling!)
θ⃗ =(θ0,θ1,⋯,θn) X=(x(1)T→,x(2)T→,x(3)T→,⋯,x(m)T→)
x(i)T→=(xi0,xi1,xi2,⋯,xin) yT→=(y(1),y(2),⋯,y(m))
θ=(XTX)−1XTy
Octave: pinv(x'*x)*x'*y
m
training examples,
Gradient Descent | Normal Equation |
---|---|
Need to choose α | No need to choose α |
Needs many iterations | Dont’ need to iterate |
Work well even when n is large | Need to compute (XTX)−1 |
Slow if n is very large ( O(n3) ) |
Tips
我们有如下定义:
对于一个函数
f:Rm×n−>R
,将
m×n
的矩阵
A
映射为一个实数,定义
引入矩阵的 迹 : tr 。 对于一个 n 阶方阵
容易得出如下性质:
对于一个向量 z , 有
最小化 J(θ) ,就要求解它关于 θ 的导数:
得出正规方程 :
Octave 的
pinv
在
(XTX)
不可逆的情况下依旧可以得出
θ
的解。