持续学习——《Selfless Sequential Learning》——ICLR2019

Abstract

sequential learning=lifelong learning=incremental learning = continual learning, look at the scenario with fixed model capacity, the learning process should account for future tasks to be added and thus leave enough capacity for them. (not selfish)

Introduction

the challenge situation of learning a sequence of tasks, without access to any previous or future task data and restricted to a fixed model capacity. 哺乳动物mammalian brain的大脑学习任务。用神经科学的内容来解释motivation.首先，较少数量的神经元被激活用于表征信息，然后被激活的神经元会减少该神经元周围的神经元的活动（lateral inhibition）。This creates a powerful decorrelated and compact representation with minimum interference between different input patterns in the brain (Yu et al., 2014)
《Reducing overfitting in deep networks by decorrelating representations arxiv2015》 show that when the amount of overfitting in a neural network is reduced, the representation correlation is also reduced
Parameter sparsity or representation sparsity.
要理清几个概念，disentangled representation.解耦的表达更不容易遭遇灾难性遗忘
Sparse and decorrelated representation。Decorrelated representation=disentangled representation
EWC, MAS
后面讲sparsity，the main idea of our regularizer is to penalize neurons that are active at the same time.

Method

Sparse coding through Local Neural Inhibition and Discounting (SLNID). 介绍了一种新的regularizer, which encourages sparsity in the activations for each layer.

Conclusion

sparsity should be imposed at the level of representation rather than at the level of the network parameters.
提出方法的motivation来自于lateral inhibition in the mammalian brain. 具体地，a new regularizer that decorrelates nearby active neurons.
Leaning a new task selflessly by leaving capacity for future tasks, avoid forgetting previous tasks通过考虑神经元的重要性neuron importance（之前的工作相似的insight，parameter importance）

Key points: 这篇文章motivation很好;包装方法的解释（神经科学）可以多学习