【ML】李宏毅机器学习笔记

最新推荐文章于 2024-09-27 16:47:35 发布

jc2020_

最新推荐文章于 2024-09-27 16:47:35 发布

阅读量4.4k

点赞数 1

本文链接：https://blog.csdn.net/u013541048/article/details/81335256

版权

本文是李宏毅机器学习课程的笔记，涵盖了从机器学习定义、模型选择到深度学习、梯度下降法、分类算法、正则化、误差分析等多个主题。深入浅出地讲解了机器学习中的基本概念和技术，包括线性回归、逻辑回归、深度学习模型、梯度下降优化方法及其变种，以及无监督学习和半监督学习的原理。

摘要由CSDN通过智能技术生成

我的github链接 - 课程相关代码：

https://github.com/YidaoXianren/Machine-Learning-course-note

0. Introduction

Machine Learning: define a set of function, goodness of function, pick the best function
Regression输出的是一个标量，Classification输出的是(1)是或否(Binary Classification) (2) Multi-class Classification
选不同的function set其实就是选不同的model，model里面最简单的就是linear model;此外还有很多nonlinear model，如deep learning, SVM, decision tree, kNN...... 以上都是supervised learning - 需要搜集很多training data
Semi-supervised learning(半监督学习) - 有些有有些没有label
Transfer Learning - data not related to the task considered
Unsupervised Learning(非监督学习)
Structured Learning - Beyond Classification (输出的是一个有结构性的object)
Reinforcement Learning - 没有监督知道，只有一个好or坏的评分机制(learning from critics)
蓝色: scenario; 红色: task - 要解的问题; 绿色: method.

li1

1. Regression

output a scalar
Step1: Model: $y = b + wx_{cp}$ w and b are parameters, w: weight, b: bias
Linear Model: $y = b + \sum w_ix_i$
Step2: Goodness of Function - Loss function L: input is a function, output is how bad it is (损失函数)
first version of Loss Function: $L(f) = \sum (\hat y^n - f(x^n_{cp}))^2$ or $L(w,b) = \sum (\hat y^n - (b+wx^n_{cp}))^2$
Step3: Best Function - $w^*b^* = arg \min_{w,b}L(w,b) = arg \min_{w,b}\sum_n (\hat y^n-(b+wx^n_{cp}))^2$
Gradient Descent (梯度下降法) - 只要loss func对与它的参数是可微分的就可以用，不需要一定是线性方程
- Pick an initial value $w^0$ ; - Compute $\frac{dL}{dw}|_{w=w^0}$ ; - $w1 \leftarrow w^0 - \eta \frac{dL}{dw}|_{w=w^0}$ , where $\eta$ is learning rate. Continue this step until finding gradient that is equal to zero.
For two parameters: ; - Pick initial value: $w_0, b_0$ ; - Compute $\frac{\partial L}{\partial w}|_{w=w_0, b=b_0}, \frac{\partial L}{\partial b}|_{w=w_0, b=b_0}$ ; - $w^1 \leftarrow w^0 - \eta \frac{\partial L}{\partial w}|_{w=w^0, b=b^0}$ $b^1 \leftarrow b^0 - \eta \frac{\partial L}{\partial b}|_{w=w^0, b=b^0}$ 。Continue this step until finding gradient that is equal to zero.
以上方法得出来的结果 $\theta^*$ 满足： $\theta^* = arg \min_\theta L(\theta)$
gradient descent缺点：可能会卡在saddle point或者local minima
对于linear regression, 由于它是convex的函数，所以不存在上述缺点。
Liner Regression - Gradient descent formula summary:
- $L(w,b) = \sum^{10}_{n=1}(\hat y^n-(b+wx^n_{cp}))^2$
- $\frac{\partial L}{\partial w} = \sum^{10}_{n=1}2(\hat y^n-(b+wx^n_{cp}))(-x^n_{cp})$
- $\frac{\partial L}{\partial b} = \sum^{10}_{n=1}2(\hat y^n-(b+wx^n_{cp}))(-1)$
复杂的模型在test data上不一定有更好的表现，有可能是overfitting(过拟合)
overfit的解决方法：1. 增加input数据集 2. regularization
Regularization (正则化)
- $y = b + \sum w_ix_i$ $L = \sum_n(\hat y^n-(b + \sum w_ix_i))^2 + \lambda (w_i)^2$
- 不但要选择一个loss小的function，还要选择一个平滑的function(正则化使函数更平滑, 因为w比较小) - smoother function is more likely to be correct
- $\lambda$ 大，找出来的function就比较smooth。反之，找出来的则不太smooth. 在 $\lambda$ 由小到大变化的过程中，函数不止要考虑loss最小化还要考虑weights最小化，所以对training error最小化的考虑就会相对(于没有正则化的时候)减小，因此training error会随着 $\lambda$ 增大而增大。test error则先减小再增大。

2. Error

Bias: $m = \frac{1}{N}\sum_nx^n$ ; Variance: $s^2 = \frac{1}{N}\sum_n(x^n-m)^2$ ; $E[s^2] = \frac{N-1}{N}\sigma^2 \neq \sigma^2$ ; want low bias & low variance
when using low degree(simple) models, variance is small, while complicate model leads to large variance. 简单的模型受采样数据的影响较小
Bias: If we average all the $f^*$ , it is close to $\hat f$ . $f^*$ 是每次训练的最佳函数(model)解(注:每次训练包含多个数据样本-sample data)，而 $\hat f$ 是真实的函数(model)。
simple models have larger bias & smaller variance, while complicate models have smaller bias & larger variance.
如果error来自于variance很大，说明现在的模型是overfitting;如果error来自bias很大，说明现在的模型是underfitting
如果模型没法fit training data，说明此时bias很大；如果模型很fit training data, 但是很不fit test data，说明此时variance很大
For large bias: add more feature, make a more complicate model
For large variance: get more data, or regularization (所有曲线都会变得比较平滑)
Cross Validation: Training Set, Validation Set, Testing Set (Public, Private)
N-fold Cross Validation - 交叉验证：可以先分成training set和validation set, train的用来训练model, validation的用来挑选model。选定model之后再用整个data set (training set+validation set)来重新train这个model的参数

3. Gradient Descent

$\theta^* = arg \min_\theta L(\theta)$ L: loss function, $\theta$ : parameters
假设 $\theta$ 有两个变量 ${\theta_1, \theta_2}$ , 则：
$\theta^0 = \left[ \begin{matrix} \theta^0_1 \\ \theta^0_2 \end{matrix} \right]$ ;