ELASTIC: Energy-based latent space alignment with second-order influences of memory selection for continual learning

Abstract

Continual learning models encounter a significant challenge, namely catastrophic forgetting. After training on a new task they forget previously learned knowledge, causing a significant drop in models’ performance on previous tasks. Replay-based methods, the most efficient approach for addressing catastrophic forgetting, select a small number of data (the coreset) from each learned task to store into an episodic memory buffer for rehearsal during subsequent stages. This selection significantly impacts the models’ ability to perform well on new tasks while maintaining performance on previous tasks. In this paper, we propose a two-phase method to address catastrophic forgetting in continual learning. The first phase utilizes the second-order influence function to select an effective coreset from previously learned tasks. Even with this effective selection, there is still a problem that the hidden feature space is unexpectedly transformed across each task, causing the model to forget optimal hidden representations on previously learned tasks. To address this issue, the second phase employs an energy-based latent aligner (ELI) to re-align the hidden feature representations of tasks towards the optimal region. Extensive experiments on three continual learning benchmark datasets, i.e. CIFAR-10, CIFAR-100, and Split miniImageNet, demonstrate that our proposed method outperforms several existing state-of-the-art continual learning models.