DSD:Dense-Sparse-Dense training for deep neural networks

ICLR 2017会议论文。

摘要：

神经网络因为参数很多，所以很难训练。

Modern deep neural networks have a large number of parameters, making them very hard to train.

所以，分步骤训练参数。

We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance. In the first D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the unimportant connections with small weights and retraining the network given the sparsity constraint. In the final D (re-Dense) step, we increase the model capacity by removing the sparsity constraint, re-initialize the pruned parameters from zero and retrain the whole dense network.

实验结果好。

Experiments show that DSD training can improve the performance for a wide range of CNNs, RNNs and LSTMs on the tasks of image classification, caption generation and speech recognition

 

转载于:https://www.cnblogs.com/mengmengmiaomiao/p/7652779.html