Lecture 9: CNN Architectures

CS231n

Lecture 9: CNN Architectures

Case Studies

LeNet-5

AlexNet

ZFNet

VGGNet

smaller filters:Stack of three 3x3 conv (stride 1) layers has same effective receptive field as one 7x7 conv layer, but deeper, more non-linearities

GoogLeNet

• “Inception” module: a good local network topology (network within a network)
• No FC layers
• 12x less parameters than AlexNet
• 1x1 conv “bottleneck” layers 降低计算量

ResNet

degregation

Network in Network (NiN)

Identity Mappings in Deep Residual Networks

moves activation to residual mapping pathway, Creates a more direct path for propagating information throughout network

Wide Residual Networks

• residuals are the important factor, not depth
• 50-layer wide ResNet outperforms 152-layer original ResNet
• Increasing width instead of depth more computationally efficient (parallelizable)

ResNeXt

ResNet + Inception?

Deep Networks with Stochastic Depth

• reduce vanishing gradients and training time through short networks
• Randomly drop a subset of layers during each training pass
• Bypass with identity function
• Use full deep network at test time

FractalNet

有点像scattering network?
- key is transitioning effectively from shallow to deep and residual representations are not necessary
- Fractal architecture with both shallow and deep paths to output
- Trained with dropping out sub-paths

DenseNet

• each layer is connected to every other layer
• Alleviates vanishing gradient, strengthens feature propagation, encourages feature reuse

SqueezeNet

类似bottleneck的思想
- Fire modules consisting of a ‘squeeze’ layer with 1x1 filters feeding an ‘expand’ layer with 1x1 and 3x3 filters
- AlexNet level accuracy on ImageNet with 50x fewer parameters
- compress to 0.5Mb