Lecture 4: Backpropagation and Neural Networks

CS231n

Lecture 4: Backpropagation and Neural Networks

Backpropagation

反向传播的理论基础：多元函数微分学中的链式法则
$z = z(x,y),x = x(u,v), y = y(u,v) \Rightarrow \mathrm{d}z = \frac{\partial z}{\partial x}\mathrm{d}x + \frac{\partial z}{\partial y}\mathrm{d}y = \frac{\partial z}{\partial x}(\frac{\partial x}{\partial u}\mathrm{d}u + \frac{\partial x}{\partial v}\mathrm{d}v) + \frac{\partial z}{\partial y}(\frac{\partial y}{\partial u}\mathrm{d}u + \frac{\partial y}{\partial v}\mathrm{d}v) \Rightarrow \frac{\partial z}{\partial u} = \frac{\partial z}{\partial x}\frac{\partial x}{\partial u} + \frac{\partial z}{\partial y}\frac{\partial y}{\partial u}$
于是梯度可以不断向前传递
Q2: what does it look like?
A: Jacobian shape难道不是$100\times4096\times4096$？

Neural Networks

[FC+ReLU]xN + Softmax/SVM
单层感知机来源于神经生物学，但是现在的神经网络并没有仿生人类神经