深度学习复习提纲

Introduction

• 分类/回归 $⟺$ 函数 $f_i:{\mathbb{R}}^d\to \mathbb{R}$
• 类别, 概率, one hot embedding, softmax + cross entropy.
• 损失
  • mean square error
  • mean absolute error
  • cross entropy (for probability ∈ [0,1])
• 梯度下降
  • 梯度为零, 局部最优vs鞍点; 梯度极其接近于零.
  • $\nabla L({\theta }_t)=(\frac{\partial L}{\partial \theta }{|}_{\theta ={\theta }_t})$
  • vanilla
    ${\theta }_{t+1}={\theta }_t-\eta \nabla L({\theta }_t)$
    ${\theta }_{t+1}={\theta }_t+G_t$, $G_t=-\eta \nabla L({\theta }_t)$.
  • momentum
    ${\theta }_{t+1}={\theta }_t-\eta \sum _{s=1}^t{\lambda }^{t-s}\nabla L({\theta }_t)$
    ${\theta }_{t+1}={\theta }_t+G_t$, $G_t=\lambda G^{(t-1)}-\eta \nabla L({\theta }_t)$.
  • adam = rmsprop + momentum
• adaptive learning rate
  • ${\theta }_{t+1}={\theta }_t+\frac{1}{H_t}{G}_t$
  • adagrad
    $H_t=\sqrt{\frac{1}{t}\sum _{s=1}^t|\nabla L({\theta }_t){|}^2}$
  • rmsprop
    $H_t=\sqrt{{\alpha }^t|\nabla L({\theta }_1){|}^2+(1-\alpha )\sum _{s=1}^t{\alpha }^{t-s}|\nabla L({\theta }_t){|}^2}$
    ${H_t}^2=\alpha {H_{t-1}}^2+(1-\alpha )|\nabla L({\theta }_t){|}^2$
• learning rate scheduling
  • decay ↘
  • warm up ↗ ↘
• batch
  • epoch = sample / batch
  • large batch, fast training, sharp minima; small batch, noisy gradient, better optimization and generalization.
• 激活函数
  • hard sigmoid
  • sigmoid
  • ReLU
• 神经网络
  • $z=\sigma (b+Wx)$, $\frac{\partial L}{\partial W}=\frac{\partial L}{\partial z}\frac{\partial \sigma }{\partial (b+Wx)}\frac{\partial (b+Wx)}{\partial W}=\frac{\partial L}{\partial z}\sigma (1-\sigma )x^T$, $\frac{\partial L}{\partial b}=\frac{\partial L}{\partial z}\frac{\partial \sigma }{\partial (b+Wx)}\frac{\partial (b+Wx)}{\partial b}=\frac{\partial L}{\partial z}\sigma (1-\sigma )$.
• 深度学习 vs 表示学习
• 深度网络 vs 宽度网络
• pytorch
  • 计算图
  • x.grad
  • torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=‘valid’, dilation=1, groups=1, bias=True, padding_mode=‘zeros’, device=None, dtype=None)
  • torch.nn.LSTM(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0, bidirectional=False, proj_size=0)
• 验证与测试
  • oof(out of fold)与cv(cross validation)
  • train loss与test loss(优化与泛化)
    train loss大(无法优化), bias(模型太简单), 优化技术.
    train loss小test loss大(无法泛化), variance(模型太复杂), 分布变化.

CNN

• receptive field
• parameter sharing
• subsampling / pooling
• feature map
• activation maximization vs saliency map (gradient)
• 图像任务
  • LeNet, AlexNet, VGG, GoogleLeNet, ResNet.
  • LeNet, 卷积+池化, 先卷积再全连接. (LeCun)
  • AlexNet, ReLU解决深度梯度消失, DropOut解决深度过拟合. (Hinton)
  • VGG, 3x3核. (牛津)
  • GoogleLeNet, 多种卷积核. (谷歌)
  • ResNet, 残差连接同时解决深度梯度消失和深度过拟合. (何恺明 孙剑)

祭
孙剑今日辞世.

RNN

• elman $y_t=f(c_{t-1},x_t)$
  jordan $y_t=f(y_{t-1},x_t)$
• bidirectional $\bar{y}=y_{\leftarrow }+y_{\to }$
• LSTM
  $\begin{array}{r}{g}_t=\\ i_t=\\ f_t=\\ c_t\end{array}$