dropout、dropconnect、maxout、batch normalization

This success of CNNs is partly due to the availability of large datasets and high-performance computing systems and partly due to the recent technical advances on learning methods and regularization techniques like dropout [27], dropconnect [31], maxout [5] and batch normalization [8].

[27] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever,
and R. Salakhutdinov. Dropout: A simple way to prevent
neural networks from overfitting. The Journal of
Machine Learning Research, 15(1):1929–1958, 2014.

[31] L.Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus.
Regularization of neural networks using dropconnect.
In Proceedings of the 30th International Conference
on Machine Learning (ICML-13), pages 1058–1066,
2013. 1, 2

[5] I. J. Goodfellow, D. Warde-Farley, M. Mirza,
A. Courville, and Y. Bengio. Maxout networks. arXiv
preprint arXiv:1302.4389, 2013. 1, 2, 5, 6

[8] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate
shift. arXiv preprint arXiv:1502.03167, 2015.
1, 2