深度神经网络加速和压缩

最新推荐文章于 2023-03-04 22:10:46 发布

XSYYMY

最新推荐文章于 2023-03-04 22:10:46 发布

阅读量3.4k

点赞数 2

文章标签：神经网络总结网络加速与压缩

本文链接：https://blog.csdn.net/XSYYMY/article/details/81904882

版权

本文总结了神经网络加速和压缩的方法，包括低秩分解、剪枝、量化、知识蒸馏和紧凑网络设计。低秩方法如SVD、CP分解等已较少使用，剪枝技术如结构化剪枝、网络瘦身、梯度剪枝得到发展。量化技术通过低比特量化、二值权重网络等降低计算复杂度。知识蒸馏则通过定义知识和优化损失来传递教师网络的知识。紧凑网络设计如MobileNet和ShuffleNet在保持效率的同时减少了计算量。

摘要由CSDN通过智能技术生成

模型加速与压缩方法分类总结

• Low-Rank
• Pruning
• Quantization
• Knowledge Distillation
• Compact Network Design

Low-Rank

Previous low-rank based methods:

• SVD
- Zhang et al., “Accelerating Very Deep Convolutional Networks for Classification and Detection”. IEEE TPAMI 2016.

• CP decomposition
- Lebedev et al., “Speeding-up Convolutional Neural Networks Using Fine-tuned CP- Decomposition”. ICLR 2015.

• Tucker decomposition
- Kim et al., “Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications”. ICLR 2016.

• Tensor Train Decomposition
- Novikov et al., “Tensorizing Neural Networks”. NIPS 2016.

• Block Term Decomposition
- Wang et al., “Accelerating Convolutional Neural Networks for Mobile Applications”. ACMMM 2016.

Recent low-rank based methods:

• Tensor Ring (TR) factorizations
- Wang et al., “Wide Compression: Tensor Ring Nets”. CVPR2018

• Block Term Decomposition For RNN
- Ye et al., “Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition ”. CVPR2018.

Why low-rank is not popular anymore?

• Low-rank approximation is not efficient for those 1x1 convolutions
• 3x3 convolutions in bottleneck structure have less computation complexity
• Depthwise convolution or grouped 1x1 convolution is already quite fast.

Pruning

Recent progress in pruning :

• Structured Pruning
– Yoon et al. “Combined Group and Exclusive Sparsity for Deep Neural Networks”. ICML2017
– Ren et al. “SBNet: Sparse Blocks Network for Fast Inference”. CVPR2018

• Filter Pruning
– Luo et al. “Thinet: A filter level pruning method for deep neural network compression”. ICCV2017
– Liu et al., “Learning efficient convolutional networks through network slimming”. ICCV2017
– He et al. “Channel Pruning for Accelerating Very Deep Neural Networks”. ICCV2017

• Gradient Pruning
– Sun et al. “meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting”. ICML2017

• Fine-grained Pruning in a Bayesian View
– Molchanov et al. “Variational Dropout Sparsifies Deep Neural Networks”. ICML2017