Question:
In the imbalanced settings (e.g., "Normal," "Reversed," and "Random" orders), apart from the proposed method, which of the following statements reflects the method that retains superior performance among all baseline methods?
A. La-MAML: Its second-order gradient correction is especially robust under imbalanced data streams.
B. CLS-ER: Its classifier-centric approach yields the highest final accuracy in all three imbalance scenarios.
C. VR-MCL: It outperforms other baselines, partly by reducing high variance in its implicit Hessian estimates.
D. DER++: It enhances replay efficiency by distillation, achieving competitive accuracy even in imbalanced conditions.

Answer: D

Question 3: How effective is VR-MCL in dealing with increasingly non-stationary Settings (i.e., imbalanced CL)? In the imbalanced CL setting, where the number of samples varies across tasks, an imbalance issue arises in the samples stored in M. This introduces a high variance during the calculation process of the hyper-gradient, thereby negatively affecting the impact on the implicit Hessian. To assess the effectiveness of VR-MCL in addressing these challenges, we choose different imbalanced settings and explore the performance of various methods under this setup in Table 4. Specifically, we choose three imbalanced settings. The Normal means the total number of samples per streaming task from high to low, and the Reversed is the opposite. The Random setting, on the other hand, represents a situation where there is no specific pattern in the number of samples per streaming task. From Table 4, we can see that the performance of most models degrades in this challenging setup. However, VR-MCL significantly outperforms other baselines and even surpasses methods such as CBRS (Chrysakis & Moens, 2020) and Coresets (Borsos et al., 2020), which are specifically designed for imbalanced CL. This suggests that the VR-MCL has excellent performance under the challenged imbalanced CL setting. The 95% CI for Table 4, as well as additional experimental results for different datasets and various imbalance ratios, can be found in Appendix D.4.

Table 4: Performance on the imbalanced Seq-CIFAR10 (|M|=1000) with imbalance ratio γ=2. The Reversed means the total number of samples per streaming task from low to high. Full table see Appendix D.6.
γ =2 (Normal)
γ =2 (Reversed)
γ =2 (Random)
Methods
AAA
Acc
AAA
Acc
AAA
Acc
SGD
36.64
16.46
35.57
17.54
34.62
17.33
A-GEM
38.50
17.64
36.72
17.43
37.79
17.85
GEM
41.29
17.90
38.46
18.37
39.63
18.00
ER
52.50
33.10
46.58
28.49
43.78
31.10
DER
46.95
16.82
40.62
18.63
41.44
18.78
DER++
62.02
44.14
58.22
39.21
60.25
42.83
CLS-ER
61.37
47.75
54.92
40.51
56.60
47.48
On-EWC
38.85
16.92
37.41
17.79
37.26
14.35
CBRS
59.07
43.81
57.57
44.20
58.16
44.66
Coresets
61.11
45.37
58.12
45.80
58.56
45.63
La-MAML
36.64
29.17
32.08
31.17
42.52
31.24
VR-MCL
65.06
49.82
61.16
51.36
61.91
50.74

Published as a conference paper at ICLR 2024
META CONTINUAL LEARNING REVISITED: IMPLIC-
ITLY ENHANCING ONLINE HESSIAN APPROXIMATION
VIA VARIANCE REDUCTION
Yichen Wu1,2∗, Long-Kai Huang2†, Renzhen Wang3, Deyu Meng3,4, Ying Wei1,5†
1City University of Hong Kong, 2Tencent AI Lab, 3Xi’an Jiaotong University,
4Pazhou Laboratory (Huangpu), 5Nanyang Technological University
ABSTRACT
Regularization-based methods have so far been among the de facto choices for
continual learning. Recent theoretical studies have revealed that these methods
all boil down to relying on the Hessian matrix approximation of model weights.
However, these methods suffer from suboptimal trade-offs between knowledge
transfer and forgetting due to fixed and unchanging Hessian estimations during
training. Another seemingly parallel strand of Meta-Continual Learning (Meta-
CL) algorithms enforces alignment between gradients of previous tasks and that
of the current task. In this work we revisit Meta-CL and for the first time bridge it
with regularization-based methods. Concretely, Meta-CL implicitly approximates
Hessian in an online manner, which enjoys the benefits of timely adaptation but
meantime suffers from high variance induced by random memory buffer sampling.
We are thus highly motivated to combine the best of both worlds, through the pro-
posal of Variance Reduced Meta-CL (VR-MCL) to achieve both timely and ac-
curate Hessian approximation. Through comprehensive experiments across three
datasets and various settings, we consistently observe that VR-MCL outperforms
other SOTA methods, which further validates the effectiveness of VR-MCL1.
1
INTRODUCTION
Continual learning (CL) is a promising approach to constructing learning systems that can contin-
uously process information streams, adapt to changing environments, and acquire new knowledge
while consolidating and retaining previously learned knowledge (Parisi et al., 2019). In contrast
to traditional supervised learning, which trains on independent and identically distributed (i.i.d)
samples, continual learning involves models trained on non-stationary data distributions, where dif-
ferent classification tasks are presented sequentially. The violation of the i.i.d. assumption in CL can
lead to catastrophic forgetting, causing a significant drop in test performance on previously learned
tasks (French, 1999; McClelland et al., 1995; McCloskey & Cohen, 1989).
In recent years, numerous methods in the field of CL have been proposed with the aim of tackling
the catastrophic forgetting problem. Regularization-based methods (Kirkpatrick et al., 2017; Ritter
et al., 2018; Husz´ar, 2017; Zenke et al., 2017), being an important branch of these methods, aim to
discern and retain the important parameter weights associated with prior tasks so as to sustain their
performance. Specifically, they assess the importance of different parameter weights through the
second-order Hessian of previous tasks computed at the end of each task training. The differences
among these regularization-based methods primarily reside in their distinct approaches to approxi-
mating the Hessian. For example, EWC (Kirkpatrick et al., 2017) and Online-EWC (Husz´ar, 2017)
employ the diagonal Fisher information matrix, whereas Kronecker factored Laplace approximation
(KFLA) (Ritter et al., 2018) utilizes a more accurate Kronecker factorization approximation, con-
sidering off-diagonal elements. Similarly, Intelligent Synapses (IS) (Zenke et al., 2017) adopts an
approximate diagonal matrix, but its elements take into account the importance of a task throughout
the entire training trajectory. Despite these methods striving to improve the Hessian approximation
∗Part of the work was done when the author interned in Tencent AI Lab (wuyichen.am97@gmail.com).
†Corresponding author: Long-Kai Huang (hlongkai@gmail.com) and Ying Wei (ying.wei@ntu.edu.sg)
1Code is available at https://github.com/WuYichen-97/Meta-CL-Revised
1
Published as a conference paper at ICLR 2024
to preserve important weights for previous tasks, their estimated Hessian matrices of previous tasks
remain unchanged and fail to update during subsequent training. As the model parameters gradually
deviate from the point at which the Hessian was initially computed, the unaltered estimated Hessian
progressively loses accuracy due to the increasing truncation error of the Taylor expansion.
Different from regularization-based methods that explicitly approximate the second-order Hessian at
the end of training for each task, as will be detailed in Sec. 3, Meta-Continual Learning (Meta-CL)
methods utilize the second-order Hessian information implicitly through the hypergradient obtained
via bi-level optimization (Riemer et al., 2019; Gupta et al., 2020; Von Oswald et al., 2021). The
computation of hypergradient involves a batch of samples drawn from the memory buffer, which
stores examples from previous tasks, enabling Meta-CL to leverage up-to-date second-order infor-
mation from previous tasks. However, it is noteworthy that the utilization of samples drawn from
the memory buffer may not offer a complete representation of previous tasks, and may even lack
data pertaining to certain tasks. This can result in the presence of erroneous information in the Hes-
sian estimation. As a consequence, there may be considerable variation in the implicitly estimated
Hessian, ultimately leading to a decline in performance for previous tasks.
To leverage the benefits of Meta-CL, specifically the timely updating of second-order Hessian infor-
mation, while mitigating the negative impact of variation, we propose a momentum-based Variance-
Reduced Meta-CL (VR-MCL). Through our theoretical analysis, we demonstrate that the integra-
tion of the momentum-based variance-reduced technique into Meta-CL is equivalent to imposing a
penalty on its online estimated Hessian. This penalty effectively suppresses excessive model up-
dates, thereby better preserving crucial weights for previous tasks and their performance. Our main
contributions are summarized as follows:
◦We introduce a new perspective on Meta-CL by characterizing it as a technique that approximates
the Hessian matrix, thereby establishing a connection between Meta-CL and regularized methods.
◦To address the issue of high variance in Meta-CL, we propose Variance-Reduced Meta-CL (VR-
MCL), which incorporates hypergradient variance reduction technique into Meta-CL.
◦Theoretically, we demonstrate that our proposed momentum-based hypergradient variance reduc-
tion technique is equivalent to the inclusion of an effective penalty term within the implicitly
estimated Hessian. Additionally, we provide the regret bound of VR-MCL to further explain its
superior performance in the context of continual learning.
◦We conduct comprehensive experiments covering three datasets and multiple continual learning
settings, providing empirical evidence for the effectiveness of the proposed algorithm.
2
THE UNIFIED FRAMEWORK OF REGULARIZATION-BASED CL METHODS
In this section, we will first derive the unified iterative updating framework used by regularization-
based algorithms following the analysis of Yin et al. (2020) and then show the impact of Hessian
approximation accuracy on the performance of these algorithms.
Analysis of Regularization-based methods. Continual learning considers a sequence of N tasks
[T 1, T 2, · · · , T N], where the i-th task consists of N i samples, i.e., T i = (Xi,Yi) ={(xi
n, yi
n)}Ni
n=0.
Here, T j represents the current training task, T [1:j] refers to all j tasks the model have seen so
far, and Li(θ) denotes the empirical risk of the parameter θ on T i. The objective of CL is to learn a
model with parameters θ that minimizes the empirical risk of all seen tasks T [1:j], i.e., 1
j (Pj
i=1 Li(θ)).
However, in the context of regularization-based CL, we have no access to the samples of previously
learned tasks T [1:j−1]. The model parameters θ cannot be directly optimized to minimize the cor-
responding empirical risks, i.e., Pj−1
i=1 Li(θ). Therefore, regularized CL methods choose to approx-
imate the empirical risk of previous tasks during training on T j (Yin et al., 2020). Concretely, in
the simplest case where j=2, when training on T 2, the objective of regularization-based methods is
to minimize 1
2(Lprox
1
+L2), where Lprox
1
(θ) is an approximation of L1(θ) derived through a Taylor
expansion at ˆθ1 as Lprox
1
= L1(ˆθ1) + (θ −ˆθ1)⊤∇L1(ˆθ1) + 1
2(θ −ˆθ1)⊤∇2L1(ˆθ1)(θ −ˆθ1). Here, ˆθ1
represents the model parameters at the end training of T 1. For a more general setting where j > 2,
we can approximate the empirical loss of the previous (j −1) tasks by,
Lprox
j−1 (θ) =
j−1
X
i=1
Li(ˆθi)
| {z }
(a)
+ (θ −ˆθi)⊤∇θLi(ˆθi)
|
{z
}
(b)
+ 1
2(θ −ˆθi)⊤Hi(θ −ˆθi)
|
{z
}
(c)
,
(1)
2
Published as a conference paper at ICLR 2024
Table 1: Summary of continual learning methods under the unified iterative update rule with various
instantiations of Hessian approximation. Here F and FD are the full and diagonal Fisher information
matrices, respectively. HD denotes diagonal approximation of the Hessian matrix, and HM and
HVR are the online Hessian estimated on previous tasks and its regularized version, respectively.
Method
Hessian Approximation
Iterative Update Rule
Off-diagonal elements
Online update
Penalty
EWC / On-EWC
✗
✗
✗
θ := θ −α(Pj−1
i=1 Fi
D)−1∇θLj(θ)
KFLA
✓
✗
✗
θ := θ −α(Pj−1
i=1 Fi)−1∇θLj(θ)
IS
✗
✓
✗
θ := θ −α(Pj−1
i=1 Hi
D)−1∇θLj(θ)
La-MAML
✓
✓
✗
θ := θ −α(Hj
M)−1∇θLj(θ)
VR-MCL
✓
✓
✓
θ := θ −α(Hj
VR)−1∇θLj(θ)
where ˆθi and Hi = ∇2
θLi(ˆθi) denote the model parameter at the end of training T i and the Hessian
matrix, respectively. For the concrete analysis of Lprox
j−1 (θ) in Eqn. (1), the first term (a) is not related
to θ and thus can be discarded. Furthermore, since the training process at the end of each task is
usually converged, the gradient ∇Li(ˆθi) is near zero and the term (b) can be ignored in practice.
Therefore, the empirical loss approximation Lprox
j−1 (θ) is almost equivalent to approximating the Hes-
sian matrix in item (c). Based on this approximation and the analysis, we can derive the iterative
update rule of regularized methods in Proposition 1. (See Appendix A.1 for proof.)
Proposition 1. In regularization-based continual learning, if the model parameter θ is searched
within the neighborhood set ∪j−1
i=1N i with N i = {θ : d(θ, ˆθi) < δi}, then the iterative update rule of θ
approximately is,
θ :≈θ −α(H1 + H2 + · · · + Hj−1)−1∇θLj(θ).
(2)
The Effect of Hessian in Regularization-based CL. Let H = Pj−1
i=1 Hi denote the Hessian part in
Eqn. (2). This update rule implies that the Hessian H plays a crucial role in the optimization process.
Specifically, by multiplying the inverse of H with the gradient ∇θLj(θ), the gradient is suppressed
along the large curvature directions, i.e., the directions along the eigenvectors corresponding to large
eigenvalues of H, and amplified along the small curvature directions. In regularization-based CL
algorithms, the large curvature directions of the Hessian align with the important weights storing the
knowledge of the previous tasks, which suppresses the update scale to reduce forgetting, while the
small curvature directions facilitate updating the model to fit the current task.
Regularization-based CL in a Unified Framework. Given the critical role of the Hessian in model
updates, various regularization-based methods primarily focus on enhancing their approximation of
the Hessian. By integrating Proposition 1, we encapsulate existing regularization-based CL methods
within a unified iterative update framework as summarized them in Table 1. Concretely, EWC (Kirk-
patrick et al., 2017) and On-EWC (Husz´ar, 2017) utilize the diagonal Fisher information matrix Fi
D
to approximate the Hi in Eqn. (2). In contrast, Intelligent Synapses (IS) (Zenke et al., 2017) em-
ploys a diagonal matrix Hi
D, with elements signifying the importance of a task throughout the entire
training trajectory. To enhance the approximation of the Hessian, Kronecker factored Laplace ap-
proximation (KFLA) (Ritter et al., 2018) opts to use the Kronecker factorization approximation Fi,
taking into account off-diagonal elements, to approximate Hi.
3
REVISITING META-CL FROM A HESSIAN APPROXIMATION PERSPECTIVE
In this section, we revisit the Meta-CL from a Hessian approximation perspective. Specifically,
while regularization-based methods explicitly approximate the second-order Hessian, Meta-CL uti-
lizes second-order information through the computation of hypergradient. Consequently, it can also
be scrutinized within the iterative update framework presented in Eqn. (2).
Meta-Continual Learning (Meta-CL). As one of the earliest works of Meta-CL, Riemer et al.
(2019) propose a meta-learning formulation for minimizing the empirical risk of all seen tasks T [1:j].
In this approach, the data in T [1:j−1] are partially accessible via a memory buffer M, which stores
3
Published as a conference paper at ICLR 2024
……
Inner Loop
Outer Loop
Inputs
Outer Gradient
Weights
𝒯𝒯(1)
𝑗𝑗
𝒯𝒯(2)
𝑗𝑗
𝒯𝒯(𝐾𝐾)
𝑗𝑗
𝜃𝜃𝑏𝑏+1
𝜃𝜃𝑏𝑏(1)
𝜖𝜖𝑏𝑏∼ℳ∪𝒯𝒯𝑗𝑗
𝜃𝜃𝑏𝑏(𝐾𝐾)
𝜃𝜃𝑏𝑏
g𝜃𝜃𝑏𝑏
𝜖𝜖𝑏𝑏
𝜃𝜃(𝐾𝐾) = 𝑈𝑈𝐾𝐾(𝜃𝜃, 𝒯𝒯𝑗𝑗)
min𝜃𝜃ℒ1:𝑗𝑗(𝜃𝜃(𝐾𝐾))
Figure 1: Iterative update process of Meta-CL for the b-th iteration. The notation T j
(k) means samples
drawn from T j, M refers to the memory buffer and gϵb
θb is the update gradient.
data from previous tasks. Concretely, the optimization problem is formulated as,
minθ L[1:j](θ(K))
s.t. θ(K) = UK(θ; T j),
where UK(θ; T j)=
K inner steps
z
}|
{
U ◦· · ·◦U ◦U(θ(0); T j
(0)),
U(θ(k); T j
(k)) = [θ −β∇θLj
(k)(θ)]

θ=θ(k−1),
(3)
the L[1:j](θ(K)) characterizes the empirical risk of θ(K) on all j tasks T [1:j] which in practice is
estimated on T j and the data in memory buffer M, θ(k) denotes the parameters of the k-th inner
step of learning for task T j, and U(·) means one-step SGD update in the inner loop optimization.
We illustrate the iterative update process for the model parameters θ for the b-th iteration, denoted
by θb, in Fig. 1. In the inner loop, the model is initialized with θb(0) = θb and iteratively updated
via K steps of SGD updates, where T j
(k) is the data randomly sampled from T j in the k-th step.
Following the acquisition of θb(K), in the outer loop, we randomly sample data ϵb from T j and the
memory buffer M. Then we compute the loss L[1:j](θb(K)) and obtain the hypergradient w.r.t. θb as
gϵb
θb := ∂L(θb(K); ϵb)/∂θb. We finally update θb using the hypergradient. It should be noted that dif-
ferent Meta-CL methods typically employ different approximations of the update gradient gϵb
θb. For
instance, MER (Riemer et al., 2019) utilizes a first-order approximation, while La-MAML (Gupta
et al., 2020), following MAML (Finn et al., 2017), directly computes the hypergradient.
Revisiting from the Hessian approximation perspective. To solve the bi-level optimization prob-
lem as shown in Eqn. (3), we need to compute the hypergradient of L[1:j](θ(K)) w.r.t. θ, i.e.,
∂L[1:j](θ(K))
∂θ
= ∂L[1:j](θ(K))
∂θ(K)
∂θ(K)
θ
,
where the Taylor approximation of the first term around θ is,
∂L[1:j](θ(K))
∂θ(K)
= ∇θ(K)L[1:j](θ(K)) ≈∇θ(K)L[1:j](θ)+Hj
M(θ(K)−θ) + (θ(K)−θ)T ⊗T⊗(θ(K)−θ).
Note that Hj
M = ∇2
θ(K)L[1:j](θ) and T denote the Hessian matrix and the third-order symmetric
tensor, respectively, and ⊗represents the Kronecker product. Through the above approximation and
assuming a single inner step in optimizing T j (i.e., K=1) for simplicity, we have Proposition 2.
Proposition 2. For Meta-CL with single inner step adaption, suppose that θ(K) is located in the
ϵ-neighborhood N(θ∗, ϵ) of the optimal model parameter θ∗=argminθ L[1:j](θ(K)), L is µ-smooth,
and β <
p
δ/|∇θLj(θ) −(∇θLj(θ))2| where δ is a small number. Let α = β2, then the iterative
update rule approximately is,
θ :≈θ −α(Hj
M)−1∇θLj(θ).
We defer the proof and analysis with K-step (K > 1) adaption to Appendix A.2. Proposition 2
demonstrates that the use of hypergradient by Meta-CL is equivalent to implicitly employing the
Hessian (Hj
M) and adheres to the unified iterative update rule of regularization-based techniques.
By comparing Proposition 2 and Proposition 1, we can deduce that Meta-CL utilizes the samples
drawn from M to implicitly compute Hessian Hj
M so as to approximate (H1+ H2 + ... + Hj−1)
in Eqn (2). In contrast to regularized methods that estimate and fix each individual Hi, Meta-CL
implicitly computes one Hessian Hj
M enabling it to have the up-to-date second-order information.
Despite the advantages of Meta-CL in utilizing up-to-date Hessian information via M, it can also
introduce erroneous information. For example, the samples drawn from M may not adequately
4
Published as a conference paper at ICLR 2024
represent previous tasks or may lack instances from specific tasks. This inadequacy may lead to the
estimated Hessian mistakenly having a small curvature direction along important weights of these
previous tasks, which in turn fails to suppress updates along this direction. Consequently, this could
result in the loss of previous knowledge from those tasks and lead to forgetting.
4
VARIANCE REDUCTION ON META-CL
To tackle the issue of high variance in Meta-CL caused by the random sampling strategy in the mem-
ory buffer, we propose Variance-Reduced Meta-CL (VR-MCL) that effectively reduces the variance
in the hypergradient of Meta-CL. Our theoretical analysis demonstrates that this variance reduction
in the hypergradient is equivalent to the addition of regularization to the implicitly estimated Hes-
sian matrix, resulting in a decrease in the variance of the Hessian matrix and an improvement in its
estimation accuracy. Moreover, we also provide a theoretical guarantee of the regret bound.
4.1
VARIANCE-REDUCED META-CL (VR-MCL)
VR-MCL aims to reduce the variance of its implicitly estimated Hessian by diminishing the variance
of its hypergradient. While there are various variance reduction approaches, such as SAG (Gower
et al., 2020), SAGA (Defazio et al., 2014), and SVRG (Johnson & Zhang, 2013) for convex opti-
mization as well as the non-convex variants (Allen-Zhu & Hazan, 2016; Nguyen et al., 2017; Fang
et al., 2018), directly applying these methods to online continual learning is challenging due to the
requirement of computing the full-batch gradient for all samples in a task. However, in the online
continual learning setting, samples from a task are received in mini-batches, and the full sample set
remains inaccessible, which poses the challenge to the implementation of these techniques.
……
……
𝒯𝒯(1)
𝑗𝑗
Inner Loop
Outer Loop
Inputs
Outer Gradient
Weights
𝒯𝒯(2)
𝑗𝑗
𝒯𝒯(𝐾𝐾)
𝑗𝑗
𝒯𝒯(1)
𝑗𝑗
𝒯𝒯(2)
𝑗𝑗
𝒯𝒯(𝐾𝐾)
𝑗𝑗
𝜖𝜖𝑏𝑏
𝜃𝜃𝑏𝑏−1(1)
𝜃𝜃𝑏𝑏
𝜃𝜃𝑏𝑏+1
𝜃𝜃𝑏𝑏−1
𝜃𝜃𝑏𝑏−1(𝐾𝐾)
𝜃𝜃𝑏𝑏(1)
𝜃𝜃𝑏𝑏(𝐾𝐾)
ොg𝜃𝜃𝑏𝑏
𝜖𝜖𝑏𝑏= g𝜃𝜃
𝜖𝜖𝑏𝑏+ 𝑟𝑟(ොg𝜃𝜃𝑏𝑏−1
𝜖𝜖𝑏𝑏−1 −g𝜃𝜃𝑏𝑏−1
𝜖𝜖𝑏𝑏
)
Figure 2: The iterative update process of VR-
MCL for the b-th iteration, where ˆgϵb
θb is the
final update gradient to obtain the θb+1.
On this account, we propose a momentum-based
variance-reduced Meta-Continual Learning (Meta-
CL) method, inspired by previous works Cutkosky
& Orabona (2019); Khanduri et al. (2021). Instead
of computing the full-batch gradient which is im-
practical for online CL, our method, VR-MCL, re-
tains the parameters from the previous iterative step.
The b-th iteration of VR-MCL is illustrated in Fig. 2,
where ˆg
ϵb−1
θb−1 represents the momentum component
and gϵb
θb−1 and gϵb
θb denote the hypergradients on pre-
vious θb−1 and θb, respectively, computed using data
ϵb. In VR-MCL, the final update gradient with re-
duced variance is shown in Eqn. (4), where r rep-
resents the momentum ratio. For clarity, we present
the pseudo-codes of the algorithm in Appendix E.
ˆgϵb
θb = gϵb
θb + r(ˆgϵb−1
θb−1 −gϵb
θb−1).
(4)
To comprehend why Eqn. (4) delivers a smaller gradient variance, we initially examine ∆b := ˆgϵb
θb −
Gθb. This term measures the error incurred when we use ˆgϵb
θb as the gradient instead of the true,
but unknown full-batch gradient direction Gθb. Through demonstrating that the term E[∥∆b∥2]
decreases over time, we can show the effectiveness of the VR-MCL in variance reduction Cutkosky
& Orabona (2019). Substituting Eqn. (4) into ∆b, we can obtain the following expression,
∆b = r∆b−1 + (1 −r)(gϵb
θb −Gθb) + r(gϵb
θb −gϵb
θb−1 −(Gθb −Gθb−1)).
(5)
The second term (1−r)(gϵb
θb −Gθb) can be modulated by adjusting the ratio of r, and the third term
gϵb
θb−gϵb
θb−1−(Gθb−Gθb−1) is expected in the order of O(∥θb −θb−1∥) = O(α∥ˆgϵb−1
θb−1∥) given the µ-
smooth loss function L. Thus, we have ∥∆b∥= r∥∆b−1∥+S where S is a number that influenced by
r and the learning rate α. As ∥∆b∥keeps decreasing until it reaches S/(1−r), choosing appropriate
values of r and α that render S/(1−r) as small as possible ensures variance reduction as we expect.
Remark. As previously analyzed, most variance reduction methods, such as SAG (Gower et al.,
2020), SVRG (Johnson & Zhang, 2013) and their nonconvex version, are not applicable in online
CL settings. Therefore, we design the momentum-based variance reduction technique specifically
for Meta-CL. We also discuss other potentially viable variance reduction methods in Sec. 5.
5
Published as a conference paper at ICLR 2024
4.2
THEORETICAL ANALYSIS OF VR-MCL.
In this subsection, we will demonstrate that the reduction of hypergradient variance by VR-MCL is
equivalent to imposing a penalty on its implicitly estimated Hessian. As a result, it effectively miti-
gates the adverse effects of significant Hessian variation, as detailed in Proposition 3. Additionally,
we also provide the regret bound of VR-MCL in Theorem 1 to further explain its effectiveness.
Proposition 3. Assume that the batch size for inner step adaptation is sufficiently large and the b-th
step trains on T j. Let Hj
Mb = ∇2
θb(K)L(θb; ϵb) = [h1
b, h2
b,· · ·, hD
b ] denote the Hessian at θb calculated
on ϵb, where hd
b is the d-th column vector of Hj
Mb. Similarly, bHj
Mb−1 = [ˆh1
b−1,· · ·, ˆhD
b−1] denotes the
Hessian for the momentum term at the (b−1)-th iteration. If (PD
d=1 λdhd
b)(PD
d=1 λdˆhd
b−1) ≥0 holds
for any λd ̸=0, d={1, 2,· · ·, D}, then we have the following iterative update rule for VR-MCL,
θ :≈θ −αˆgϵb
θb = θ −α(Hj
VR)−1∇θLj(θ),
where (Hj
VR)−1 =(Hj
Mb)−1+r(( bHj
Mb−1)−1−(H
j
Mb−1)−1) with H
j
Mb−1 denoting the Hessian at θb−1 calcu-
lated on ϵb. The eigenvalues of (Hj
VR)−1 are approximately given by v−1
b + r((ˆvb−1)−1−(vb−1)−1),
where vb, ˆvb−1, and vb−1 denote the eigenvalues of Hj
Mb, bHj
Mb−1, and H
j
Mb−1, respectively.
Please kindly refer to Appendix A.3 for the detailed proof. Proposition 3 develops the relationship
between the Hessian Hj
VR used in the update rule of VR-MCL and that used in the update rule of
Meta-CL (i.e., Hj
Mb). Similar to the analysis of the effect of Hessian in Secs. 2 and 3, we reach the
following conclusions.
◦For those wrongly estimated low-curvature directions by Meta-CL, i.e., ˆvb−1≫vb, vb−1 (some
potential crucial weights are erroneously viewed as unimportant), we have v−1
b
+ r((ˆvb−1)−1 −
(vb−1)−1)≈r(ˆvb−1)−1+(1−r)v−1
b ≤v−1
b . This signifies that VR-MCL can prevent excessive updates
triggered by the wrongly estimated low-curvature direction of the Hessian, thereby retaining the
critical weights for previous tasks and improving the accuracy of its iterative updating formula.
◦For the high-curvature directions with large eigenvalues, i.e., ˆvb−1 ≈vb ≈vb−1, we have v−1
b
+
r((ˆvb−1)−1 −(vb−1)−1) ≈v−1
b . This suggests that VR-MCL can effectively ensure cautious
updates along these high-curvature directions of Meta-CL.
To sum up, Meta-CL is susceptible to inaccurate estimations of Hj
Mb due to the random samples
drawn from M. In contrast, the proposed VR-MCL incorporates a regularization term on Hj
Mb,
which effectively limits the model from making large updates and improves the precision of the
iterative update. As a result, VR-MCL can effectively preserve the important parameters for previous
tasks, thereby maintaining their performance.
Theorem 1 (Regret Bound of VR-MCL). The regret in online continual learning follows CRj =
˜F(θ) −F(θ∗), where ˜F(θ) = Pj
i=1 Li(ˆθj), F(θ∗) = minθ
Pj
i=1 Li(θ). Assuming F with φ-
Lipschitz hessian is µ-strongly convex, G-Lipschitz, η-smooth and grounded on the four assumptions
in Appendix A.4, σ and M are two large constants in Assumption 3 and 4 respectively, and T denotes
the training iteration. Then with probability at least 1 −δ for any δ ∈(0, 1), we can get,
CRj ≤(log T + 1)(F(θ1) −F(θ∗)) + LD2(log T + 1)2
2
+ LD2(log T + 1)2
2
+
16LD2 + 16σD + 4M
 p
2T log(8T/δ) = ˜O(
√
T).
Theorem 1 illustrates that, under mild assumptions in a stochastic setting, VR-MCL attains a nearly
optimal ˜O(
√
T) regret bound w.h.p. for the online optimization problem. This theoretically demon-
strates the effectiveness of the proposed VR-MCL algorithm. Detailed proof see Appendix A.4.
5
EXPERIMENTS
Datasets and Settings. To verify the effectiveness of the proposed VR-MCL, we conduct compre-
hensive experiments on commonly used datasets Seq-CIFAR10, as well as the longer task sequences
6
Published as a conference paper at ICLR 2024
Table 2:
Performance of Seq-CIFAR10 and longer task sequences Seq-CIFAR100, Seq-
TinyImageNet with 95% confidence interval on reduced ResNet-18. The memory buffer size is
set as 1000 (i.e., |M|=1000). All reported numbers are the average of 5 runs. Shaded areas are our
methods, and ‘–’ indicates the result is omitted due to high instability.
Seq-CIFAR10
Seq-CIFAR100
Seq-TinyImageNet
Method
AAA
Acc
AAA
Acc
AAA
Acc
SGD
34.85 ±1.71
16.96 ±0.60
11.63 ±0.38
5.27 ±0.28
8.99 ±0.39
3.86 ±0.20
LWF
35.31 ±0.98
18.84 ±0.10
11.98 ±0.40
5.63 ±0.37
9.21 ±0.37
4.01 ±0.29
A-GEM
38.66 ±0.79
18.46 ±0.17
13.15 ±0.23
6.02 ±0.17
9.81 ±0.31
4.07 ±0.18
GEM
38.67 ±0.77
18.49 ±0.15
15.18 ±0.38
8.30 ±0.58
10.99 ±0.32
5.15 ±0.31
ER
55.53 ±2.58
43.83 ±4.84
23.19 ±0.38
16.07 ±0.88
19.45 ±0.40
11.13 ±0.39
DER
45.85 ±1.62
29.87 ±2.95
13.35 ±0.36
6.12 ±0.18
10.35 ±0.33
4.08 ±0.13
DER++
64.22 ±0.70
52.29 ±1.86
19.88 ±0.43
11.79 ±0.65
14.75 ±0.15
8.26 ±0.34
CLSER
63.02 ±1.54
52.80 ±1.66
25.46 ±0.57
17.88 ±0.69
19.43 ±0.37
11.09 ±0.24
OCM
66.14 ±0.95
53.39 ±1.00
22.54 ±0.79
14.40 ±0.82
10.45 ±0.65
4.53 ±0.54
ER-OBC
65.82 ±0.91
54.85 ±2.16
25.54 ±0.25
17.21 ±0.92
20.11 ±0.31
11.51 ±0.18
On-EWC
38.44 ±0.50
17.12 ±0.51
11.81 ±0.42
5.88 ±0.31
9.38 ±0.17
3.65 ±0.22
IS
37.33 ±0.23
17.39 ±0.19
12.32 ±0.22
5.20 ±0.18
8.73 ±0.25
3.33 ±0.28
MER
50.99 ±0.65
36.92 ±2.42
–
–
–
–
La-MAML
42.98 ±1.60
33.43 ±1.21
12.55 ±0.39
11.78 ±0.65
11.10 ±0.70
6.74 ±0.36
VR-MCL
66.97 ±1.58
56.48 ±1.79
27.01 ±0.48
19.49 ±0.69
21.26 ±0.53
13.27 ±0.39
Seq-CIFAR100 and Seq-TinyImageNet (Buzzega et al., 2020).
Specifically, the Seq-CIFAR10
dataset comprises 5 tasks, with each task containing 2 classes. In contrast, Seq-CIFAR100 consists
of 10 tasks, each with 10 classes, while Seq-TinyImageNet includes 20 tasks, each encompassing
10 classes. In this paper, we focus on the following settings of continual learning:
• Online CL: the data is fed to the model in small batches and trained in a single pass.
• Class Incremental: the model cannot get the oracle task index during testing.
Our evaluation includes the metrics and experimental settings following the previous works on on-
line CL with a single-head (Caccia et al., 2021; Ji et al., 2020; Shim et al., 2021). We choose the
final Averaged accuracy (Acc) across all tasks after sequential training on each task as the main
metric for comparing approaches. Moreover, under online CL, we use the Averaged Anytime Ac-
curacy (AAA) Caccia et al. (2021) to evaluate the model through the stream tasks. Let AAj denote
the test average accuracy after training on Tj, then the evaluation metrics of AAA and Acc are:
AAA = 1
N
PN
j=1(AAj), Acc = AAN.
Baselines and Training Details. To more effectively validate the efficacy of VR-MCL, which
improves the Hessian component and thus better utilizes second-order information, we select key
regularization-based CL methods (Husz´ar, 2017; Zenke et al., 2017) and Meta-CL methods (Riemer
et al., 2019; Gupta et al., 2020). We also opt for other representative SOTA CL methods (Chaudhry
et al., 2018; Lopez-Paz & Ranzato, 2017; Rolnick et al., 2019; Buzzega et al., 2020; Arani et al.,
2021; Guo et al., 2022; Chrysakis & Moens, 2023) as baselines to further highlight the supe-
rior performance of VR-MCL. For detailed information on each baseline method, please refer to
Appendix C. For fair comparison among different CL methods, we train all the models using
the Stochastic Gradient Descent (SGD) optimizer under the online continual learning. Follow-
ing (Chrysakis & Moens, 2023; Lopez-Paz & Ranzato, 2017) and (Gupta et al., 2020) we utilize
two backbones: reduced ResNet-18 and a smaller network named PcCNN which has only 3 conv-
layers. For the reduced ResNet18, we set both the batch size and the replay batch size (i.e., the batch
size sampled from the memory buffer M) as 32. For the smaller network PcCNN, we set both the
batch size and the replay bath size as 10. The momentum ratio r and the learning rate α of VR-MCL
are both set as 0.25 for all experiments. For other training details please check Appendix C.
To assess the effectiveness of VR-MCL, we perform a comprehensive set of experiments and con-
duct ablation studies to address the following six key questions:
Question 1: How does VR-MCL perform on online CL benchmarks? We report results with
a 95% confidence interval on Seq-CIFAR10 and the longer task sequences Seq-CIFAR100, Seq-
TinyImageNet in Table 2. The methods in the bottom row are the methods utilizing second-order
information while the top row are the methods of other representative CL methods. It is evident that
the proposed VR-MCL substantially enhances the performance of CL methods employing second-
7
Published as a conference paper at ICLR 2024
Table 3: Performance of Seq-CIFAR100 with 95% confidence interval (CI) on different memory
buffer sizes |M|. The backbone is reduced ResNet-18, and all numbers are the average of 5 runs.
Shaded areas are our methods, and ‘–’ indicates the result is omitted due to high instability.
|M|=200
|M|=600
|M|=1000
Method
AAA
Acc
AAA
Acc
AAA
Acc
SGD
11.63 ±0.38
5.60 ±0.28
11.63 ±0.38
5.60 ±0.28
11.63 ±0.38
5.60 ±0.28
LWF
12.01 ±0.42
5.84 ±0.21
12.12 ±0.41
5.97 ±0.33
12.01 ±0.43
5.77 ±0.32
A-GEM
13.45 ±0.29
6.25 ±0.25
13.36 ±0.21
6.00 ±0.16
13.15 ±0.36
6.12 ±0.18
GEM
15.41 ±0.21
7.92 ±0.35
14.92 ±0.24
7.89 ±0.31
15.18 ±0.38
8.30 ±0.58
ER
18.52 ±0.63
10.15 ±0.42
21.73 ±0.25
13.63 ±0.60
23.19 ±0.38
16.07 ±0.88
DER
13.44 ±0.31
6.12 ±0.12
13.57 ±0.27
6.23 ±0.12
13.35 ±0.36
6.12 ±0.18
DER++
17.91 ±0.47
9.59 ±0.42
19.13 ±0.27
10.45 ±0.30
19.88 ±0.43
11.79 ±0.65
CLSER
20.56 ±0.35
11.35 ±0.36
23.63 ±0.33
16.55 ±0.35
25.46 ±0.57
17.88 ±0.69
OCM
–
–
18.73 ±0.10
8.40 ±0.37
22.54 ±0.79
14.40 ±0.82
ER-OBC
19.53 ±0.28
10.38 ±0.37
23.96 ±0.40
15.83 ±0.41
25.54 ±0.25
17.21 ±0.92
On-EWC
11.81 ±0.42
5.88 ±0.31
11.81 ±0.42
5.88 ±0.31
11.81 ±0.42
5.88 ±0.31
IS
12.32 ±0.22
5.20 ±0.18
12.32 ±0.22
5.20 ±0.18
12.32 ±0.22
5.20 ±0.18
La-MAML
8.48 ±0.34
6.95 ±0.27
10.07 ±1.10
9.05 ±0.58
12.55 ±0.39
11.78 ±0.65
VR-MCL
22.41 ±0.59
13.27 ±0.40
25.97 ±0.47
17.15 ±0.59
27.01 ±0.48
19.49 ±0.69
order information, and also outperforms other representative SOTA CL methods by nearly 2%. It
is noteworthy that the performance of VR-MCL is superior to both MER (Riemer et al., 2019)
and La-MAML (Gupta et al., 2020), and significantly better than IS (Zenke et al., 2017) and On-
EWC (Husz´ar, 2017). This observation aligns with our analysis in Table 1.
Question 2: How does the performance gain from VR-MCL vary with buffer size |M|?
In
large-scale CL, it is impractical to store a large number of examples from previous tasks due to
storage constraints. Therefore, an ideal CL method should still perform well even with a modest
memory buffer. To investigate if VR-MCL is still effective with a smaller buffer size, we conduct
experiments on Seq-CIFAR100 with different |M| as shown in Table 3. The results show that the
performance of most methods improves with the increase of memory buffer size |M|. And our
VR-MCL consistently outperforms other baselines, suggesting that VR-MCL is effective even with
modest buffer sizes. Note that GEM (Lopez-Paz & Ranzato, 2017) and A-GEM (Chaudhry et al.,
2018) are insensitive to the buffer size complying with the results in Shim et al. (2021).
Question 3: How effective is VR-MCL in dealing with increasingly non-stationary Settings
(i.e., imbalanced CL)? In the imbalanced CL setting, where the number of samples varies across
tasks, an imbalance issue arises in the samples stored in M. This introduces a high variance during
the calculation process of the hyper-gradient, thereby negatively affecting the impact on the implicit
Hessian. To assess the effectiveness of VR-MCL in addressing these challenges, we choose differ-
ent imbalanced settings and explore the performance of various methods under this setup in Table 4.
Specifically, we choose three imbalanced settings. The Normal means the total number of samples
per streaming task from high to low, and the Reversed is the opposite. The Random setting, on the
other hand, represents a situation where there is no specific pattern in the number of samples per
streaming task. From Table 4, we can see that the performance of most models degrades in this
challenging setup. However, VR-MCL significantly outperforms other baselines and even surpasses
methods such as CBRS (Chrysakis & Moens, 2020) and Coresets (Borsos et al., 2020), which are
specifically designed for imbalanced CL. This suggests that the VR-MCL has excellent performance
under the challenged imbalanced CL setting. The 95% CI for Table 4, as well as additional experi-
mental results for different datasets and various imbalance ratios, can be found in Appendix D.4.
Question 4: Whether the proposed method VR-MCL is still effective using different back-
bones? To verify whether the improvement of VR-MCL is related to the backbone network, we
change the reduced ResNet-18 as a shallow network PcCNN following Gupta et al. (2020). The
outcomes are provided in Table 5. It can be observed that VR-MCL shows a nearly 3% improve-
ment over the other best method in terms of both the AAA and Acc metrics. This indicates that the
proposed VR-MCL remains effective across different backbone architectures.
Question 5: How does VR-MCL compare to other variance reduction methods?
As dis-
cussed in Sec. 4.1, most existing popular variance reduction methods like SAG (Gower et al., 2020),
SAGA (Defazio et al., 2014), and SVRG (Johnson & Zhang, 2013) cannot be directly applied since
they need to compute the full-batch gradient which is not accessible in online CL. Therefore, we
8
Published as a conference paper at ICLR 2024
compare VR-MCL with other techniques that promise to reduce the variance: 1) VR-MCL1, which
simply increases the replay batch size; 2) VR-MCL2, which incorporates SGD with the naive mo-
mentum term. The outcomes shown in Table 6 indicate that: 1) Other variance reduction techniques
are also effective, supporting our analysis in Sec. 4; 2) Our proposed momentum-based VR-MCL
significantly outperforms other variance techniques, further demonstrating its effectiveness.
Table 4: Performance on the imbalanced Seq-CIFAR10
(|M|=1000) with imbalance ratio γ=2. The Reversed
means the total number of samples per streaming task
from low to high. Full table see Appendix D.6.
γ =2 (Normal)
γ =2 (Reversed)
γ =2 (Random)
Methods
AAA
Acc
AAA
Acc
AAA
Acc
SGD
36.64
16.46
35.57
17.54
34.62
17.33
A-GEM
38.50
17.64
36.72
17.43
37.79
17.85
GEM
41.29
17.90
38.46
18.37
39.63
18.00
ER
52.50
33.10
46.58
28.49
43.78
31.10
DER
46.95
16.82
40.62
18.63
41.44
18.78
DER++
62.02
44.14
58.22
39.21
60.25
42.83
CLS-ER
61.37
47.75
54.92
40.51
56.60
47.48
On-EWC
38.85
16.92
37.41
17.79
37.26
14.35
CBRS
59.07
43.81
57.57
44.20
58.16
44.66
Coresets
61.11
45.37
58.12
45.80
58.56
45.63
La-MAML
36.64
29.17
32.08
31.17
42.52
31.24
VR-MCL
65.06
49.82
61.16
51.36
61.91
50.74
Table 5: Performance of Seq-CIFAR10
with 95% confidence interval on the
small network PcCNN with |M|=200.
The results are the average of 5 runs.
Method
AAA
Acc
SGD
37.84 ±0.03
17.58 ±0.20
A-GEM
41.84 ±0.44
18.50 ±0.38
GEM
45.07 ±0.52
22.52 ±1.02
ER
54.64 ±0.91
32.46 ±1.58
DER
58.06 ±0.18
38.21 ±0.29
DER++
56.07 ±0.32
34.28 ±0.79
CLS-ER
58.03 ±0.75
39.38 ±1.44
ER-OBC
56.75 ±0.68
38.10 ±1.28
On-EWC
38.72 ±0.58
17.05 ±0.08
MER
55.50 ±0.38
32.01 ±1.22
La-MAML
46.36 ±0.98
29.45 ±0.51
VR-MCL
60.88 ±0.22
43.41 ±0.30
Table 6: Performance with 95% CI using various variance reduction techniques, where |M| = 1000.
Seq-CIFAR10
Seq-CIFAR100
Seq-TinyImageNet
Method
AAA
Acc
AAA
Acc
AAA
Acc
La-MAML
42.98 ±1.60
33.43 ±1.21
12.55 ±0.39
11.78 ±0.65
11.10 ±0.70
6.74 ±0.36
VR-MCL1
66.74 ±2.49
55.02 ±3.70
26.15 ±0.53
19.02 ±0.45
20.39 ±1.17
12.09 ±1.30
VR-MCL2
62.87 ±1.70
53.20 ±1.52
20.15 ±1.52
16.05 ±2.04
15.96 ±0.51
9.82 ±0.45
VR-MCL(ours)
66.97 ±1.58
56.48 ±1.79
27.01 ±0.48
19.49 ±0.69
21.26 ±0.53
13.27 ±0.39
0
500
1000
1500
2000
Iterations
0.00
0.25
0.50
0.75
1.00
1.25
Relative Gradient Variance
Meta-CL
VR-MCL
Figure 3: The relative variance of Meta-CL
and the proposed VR-MCL.
Question 6: Whether the proposed method VR-
MCL can effectively reduce the variance of gra-
dients? To explore this problem, following (Yang &
Kwok, 2022), we choose the relative variance metric
(i.e., E∥gθt−E(gθt)∥2
∥E(gθt)∥2
), which can eliminate the influ-
ence of the gradient value on the variance. The out-
comes are shown in Fig. 3, where the Meta-CL refers
to La-MAML (Gupta et al., 2020). It can be seen that
the proposed VR-MCL indeed has less variance dur-
ing training, complying with the analysis in Eqn. (5).
Other Results. Due to space constraints, additional comprehensive experiments have been pro-
vided in Appendix D, including other evaluation metrics (i.e., forgetting measure), hyperparameter
selection (i.e., the selection of the momentum ratio r and learning rate α), and training time analysis.
6
CONCLUSION
In this paper, we revisited Meta-CL and first bridged it with regularization-based methods from the
Hessian matrix approximation perspective. Specifically, Meta-CL, through the use of hypergradient,
implicitly approximates the Hessian in an online manner. While this approach benefits from timely
adaptation, it also grapples with high variance resulting from random memory buffer sampling.
Building on this understanding, we proposed the VR-MCL method, which effectively reduces the
variance of the hypergradient and exhibits superior performance under the online continual learning
setting. We provide theoretical proof that VR-MCL imposes a regularization term on the implicitly
estimated Hessian matrix. This prevents updates from moving excessively in the wrongly estimated
low-curvature directions and thus has a more accurate iterative update rule shown in Table 1. More-
over, to enhance comprehension, we also provide the regret bound of our VR-MCL.
9
Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank all the anonymous reviewers for their constructive suggestions on improving this paper.
This research was sponsored by the National Key R&D Program of China (2020YFA0713900), the
RMGS 9229073, the China NSFC projects under contract 62272375, 12226004 and 62306233, the
Young Elite Scientists Sponsorship Program by CAST 2023QNRC001.
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection
for online continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Zeyuan Allen-Zhu and Elad Hazan. Variance reduction for faster non-convex optimization. In
International Conference on Machine Learning, pp. 699–707. PMLR, 2016.
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general contin-
ual learning method based on complementary learning system. In International Conference on
Learning Representations, 2021.
Zal´an Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual
learning and streaming. Advances in Neural Information Processing Systems, 2020.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark expe-
rience for general continual learning: a strong, simple baseline. Advances in Neural Information
Processing Systems, 33:15920–15930, 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky.
New insights on reducing abrupt representation change in online continual learning. International
Conference on Learning Representations, 2021.
Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient
lifelong learning with a-gem. In International Conference on Learning Representations, 2018.
Aristotelis Chrysakis and Marie-Francine Moens. Online continual learning from imbalanced data.
In International Conference on Machine Learning, pp. 1952–1961. PMLR, 2020.
Aristotelis Chrysakis and Marie-Francine Moens. Online bias correction for task-free continual
learning. In International Conference on Learning Representations, 2023.
Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd.
Advances in Neural Information Processing Systems, 32, 2019.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method
with support for non-strongly convex composite objectives.
Advances in Neural Information
Processing Systems, 27, 2014.
Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal non-convex
optimization via stochastic path-integrated differential estimator. Advances in Neural Information
Processing Systems, 31, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation
of deep networks. In International conference on machine learning, pp. 1126–1135. PMLR, 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In
International Conference on Machine Learning, pp. 1920–1930. PMLR, 2019.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences,
3(4):128–135, 1999.
Robert M Gower, Mark Schmidt, Francis Bach, and Peter Richt´arik. Variance-reduced methods for
machine learning. Proceedings of the IEEE, 108(11):1968–1983, 2020.
10
Published as a conference paper at ICLR 2024
Yanan Gu, Xu Yang, Kun Wei, and Cheng Deng. Not just selection, but exploration: Online class-
incremental continual learning via dual view consistency. In Proceedings of the IEEE/CVF Con-
ference on Computer Vision and Pattern Recognition, pp. 7442–7451, 2022.
Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information
maximization. In International Conference on Machine Learning, 2022.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning.
Advances in Neural Information Processing Systems, 33:11588–11598, 2020.
Ferenc Husz´ar.
On quadratic penalties in elastic weight consolidation.
arXiv preprint
arXiv:1712.03847, 2017.
Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances
in Neural Information Processing Systems, 32, 2019.
Xu Ji, Joao Henriques, Tinne Tuytelaars, and Andrea Vedaldi. Automatic recall machines: Internal
replay, continual learning and the brain. arXiv preprint arXiv:2006.12323, 2020.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance
reduction. Advances in Neural Information Processing Systems, 26, 2013.
Robert W Keener. Theoretical statistics: Topics for a core course. Springer, 2010.
Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang.
A near-optimal algorithm for stochastic bilevel optimization via double-momentum. Advances in
Neural Information Processing Systems, 34:30271–30283, 2021.
Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with partition-
ing reservoir sampling. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow,
UK, August 23–28, 2020, Proceedings, Part XIII 16, pp. 411–428. Springer, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcom-
ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences,
114(13):3521–3526, 2017.
Hyunseo Koh, Minhyuk Seo, Jihwan Bang, Hwanjun Song, Deokki Hong, Seulki Park, Jung-Woo
Ha, and Jonghyun Choi. Online boundary-free continual learning by scheduled data prior. In The
Eleventh International Conference on Learning Representations.
Zhengfeng Lai, Chao Wang, Sen-ching Cheung, and Chen-Nee Chuah. Sar: Self-adaptive refine-
ment on pseudo labels for multiclass-imbalanced semi-supervised learning. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4091–4100, 2022a.
Zhengfeng Lai, Chao Wang, Henrry Gunawan, Sen-Ching S Cheung, and Chen-Nee Chuah.
Smoothed adaptive weighting for imbalanced semi-supervised learning:
Improve reliability
against unknown distribution data. In International Conference on Machine Learning, pp. 11828–
11843. PMLR, 2022b.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis
and machine intelligence, 40(12):2935–2947, 2017.
David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning.
Advances in Neural Information Processing Systems, 30, 2017.
James L McClelland, Bruce L McNaughton, and Randall C O’Reilly. Why there are complementary
learning systems in the hippocampus and neocortex: insights from the successes and failures of
connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The
sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109–165.
Elsevier, 1989.
11
Published as a conference paper at ICLR 2024
Aryan Mokhtari, Hamed Hassani, and Amin Karbasi. Stochastic conditional gradient methods:
From convex minimization to submodular maximization. The Journal of Machine Learning Re-
search, 21(1):4232–4280, 2020.
Lam M Nguyen, Jie Liu, Katya Scheinberg, and Martin Tak´aˇc. Sarah: A novel method for machine
learning problems using stochastic recursive gradient. In International Conference on Machine
Learning, pp. 2613–2621. PMLR, 2017.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual
lifelong learning with neural networks: A review. Neural Networks, 113:54–71, 2019.
Iosif Pinelis. Optimum bounds for the distributions of martingales in banach spaces. The Annals of
Probability, pp. 1679–1706, 1994.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah.
itaml: An incremental task-agnostic meta-learning approach. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 13588–13597, 2020.
Jathushan Rajasegaran, Chelsea Finn, and Sergey Levine. Fully online meta-learning without task
boundaries. arXiv preprint arXiv:2202.00263, 2022.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald
Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer-
ence. In International Conference on Learning Representations, 2019.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations
for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 31,
2018.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience
replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. On-
line class-incremental continual learning with adversarial shapley value. In Proceedings of the
AAAI Conference on Artificial Intelligence, volume 35, pp. 9630–9638, 2021.
Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis Titsias.
Information-
theoretic online memory selection for continual learning. In International Conference on Learn-
ing Representations.
Johannes Von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas
Zucchet, and Jo˜ao Sacramento. Learning where to learn: Gradient sparsity in meta and continual
learning. Advances in Neural Information Processing Systems, 2021.
Ling Xiao Wang, Kevin Huang, Tengyu Ma, Quanquan Gu, and Jing Huang. Variance-reduced
first-order meta-learning for natural language processing tasks. In Proceedings of the 2021 Con-
ference of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies, pp. 2609–2615, 2021.
Quanziang Wang, Renzhen Wang, Yichen Wu, Xixi Jia, and Deyu Meng. Cba: Improving online
continual learning via continual bias adaptor. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pp. 19082–19092, 2023a.
Renzhen Wang, Xixi Jia, Quanziang Wang, Yichen Wu, and Deyu Meng.
Imbalanced semi-
supervised learning with bias adaptive classifier. In 11th International Conference on Learning
Representations (ICLR 2023), 2023b.
Yichen Wu, Jun Shu, Qi Xie, Qian Zhao, and Deyu Meng. Learning to purify noisy labels via meta
soft label corrector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35,
pp. 10388–10396, 2021.
Yichen Wu, Long-Kai Huang, and Ying Wei. Adversarial task up-sampling for meta-learning. Ad-
vances in Neural Information Processing Systems, 35:31102–31115, 2022.
12
Published as a conference paper at ICLR 2024
Hansi Yang and James Kwok. Efficient variance reduction for meta-learning. International Confer-
ence on Machine Learning, 2022.
Yang Yang, Da-Wei Zhou, De-Chuan Zhan, Hui Xiong, and Yuan Jiang. Adaptive deep models
for incremental learning: Considering capacity scalability and sustainability. In Proceedings of
the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.
74–82, 2019.
Yang Yang, Da-Wei Zhou, De-Chuan Zhan, Hui Xiong, Yuan Jiang, and Jian Yang. Cost-effective
incremental deep model: Matching model capacity with the least sampling. IEEE Transactions
on Knowledge and Data Engineering, 2021.
Dong Yin, Mehrdad Farajtabar, Ang Li, Nir Levine, and Alex Mott. Optimization and generaliza-
tion of regularization-based continual learning: a loss approximation viewpoint. arXiv preprint
arXiv:2006.10974, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence.
In International Conference on Machine Learning, pp. 3987–3995. PMLR, 2017.
13
Published as a conference paper at ICLR 2024
A
DETAILED PROOF
A.1
PROOF OF PROPOSITION 1
Proposition 1. In regularization-based continual learning, if the model parameter θ is searched
within the neighborhood set ∪j−1
i=1N i with N i = {θ : d(θ, ˆθi) < δi}, then the iterative update rule of θ
approximately is
θ :≈θ −α(H1 + H2 + · · · + Hj−1)−1∇θLj(θ)
Proof. The empirical loss L on T [1:j] can be approximate as,
L =
j−1
X
i=1
Li(θ) + Lj(θ) ≈Lprox
j−1 (θ) + Lj(θ)
(1)
=
j−1
X
i=1
Li(ˆθi)
| {z }
(a)
+ (θ −ˆθi)⊤∇θLi(ˆθi)
|
{z
}
(b)
+ 1
2(θ −ˆθi)⊤Hi(θ −ˆθi)
|
{z
}
(c)
+Lj(θ)
(2)
≈
j−1
X
i=1
1
2(θ −ˆθi)⊤Hi(θ −ˆθi) + Lj(θ)
Here, (1) is expand each loss Li using Taylor series at point ˆθi to compute Lprox
j−1 (θ), where i =
1, ..., j−1. Step (2) is due to the term (a) is not related to θ and thus can be discarded. Furthermore,
since the training process at the end of each task (i.e., ˆθi) is usually converged, the gradient ∇θLi(ˆθi)
is near zero and the term (b) can be ignored in practice. Then we can derive ∂L
∂θ as,
∂L
∂θ =
j−1
X
i=1
Hi(θ −ˆθi) + ∇θLj(θ).
If the parameters θ are searched in the neighborhood set ∪j−1
i=1N i, where N i = {θ : d(θ, ˆθi) < δi},
then we can make the approximation Pj−1
i=1 Hi(θ −ˆθi) ∼(Pj−1
i=1 Hi)(θ −ˆθj−1) Husz´ar (2017).
Let ∂L
∂θ = 0, we can obtain the iterative update formula of T j as:
θ :≈θ −α(H1 + H2 + · · · + Hj−1)−1∇θLj(θ),
where α = 1 to maintain formal consistency with other iterative rules in Table 1, (H1 + H2 + · · · +
Hj−1) are the Hessian matrices of previous tasks computed at the end of training of each T i, and
∇θLj(θ) is the gradient of the j-th task.
A.2
PROOF OF PROPOSITION 2
Proposition 2. For MCL with single inner step adaption, suppose that θ(K) is located in the ϵ-
neighborhood N(θ∗, ϵ) of the optimal model parameter θ∗= argminθ L[1:j](θ(K)), L is µ-smooth,
and β <
p
δ/|∇θLj(θ) −(∇θLj(θ))2| where δ is a small number. Thus, the iterative update rule
approximately is
θ :≈θ −α(Hj
M)−1∇θLj(θ),
Proof. (Single inner step adaption) For simplicity, we analyze the MCL with single inner step
adaption (i.e., K = 1) at first. The objective function of one step inner-loop MCL could be formu-
lated as,
θ∗= argminθL[1:j](θ(K)),
s.t. θ(K) = θ−β∇θLj(θ).
Then the derivative of loss L[1:j](θ(K)) over θ is,
∂L[1:j](θ(K))
∂θ
= ∂L[1:j](θ(K))
∂θ(K)
∂θ(K)
θ
.
(1)
14
Published as a conference paper at ICLR 2024
For the first term
∂L[1:j](θ(K))
∂θ(K)
, we take the Taylor expansion at θ,
∂L[1:j](θ(K))
∂θ(K)
= ∇θ(K)L[1:j](θ(K)) ≈∇θ(K)L[1:j](θ)+Hj
M(θ(K)−θ) + (θ(K)−θ)T ⊗T⊗(θ(K)−θ),
(2)
where Hj
M = ∇2
θ(K)L[1:j](θ) and T denote the Hessian matrix and the third-order symmetric tensor,
respectively, and ⊗represents the Kronecker product.
Since we hope to find the optimal parameters θ∗so as to achieve the lowest L[1:j], which means
L[1:j](θ(K))
∂θ
= 0. According to Eqn. (1), that is approximately to let
∂L[1:j](θ(K))
∂θ(K)
= 0.
Suppose δ is a small number, β <
p
δ/|∇θLj(θ) −(∇θLj(θ))2|, and there exists an ϵ-neighborhood
N(·) such that parameters θ∗∈N(θ−β∇θLj(θ), ϵ). Since L is µ-smooth, which means ∥∇L(θ)−
∇L(θ
′)∥2 ≤µ∥θ −θ
′∥2. Then we can get the following equations according to Eqn. (2),
∂L[1:j](θ∗)
∂θ∗
= ∇θ∗L[1:j](θ∗)
≈∇θ∗L[1:j](θ) + ∇2
θ∗L[1:j](θ)(θ∗−θ) + (θ∗−θ)T ⊗T ⊗(θ∗−θ)
≈∇θ(K)L[1:j](θ) + ∇2
θ(K)L[1:j](θ)(θ∗−θ) + (θ(K)−θ)T ⊗T ⊗(θ(K)−θ) + o(ϵ)
= ∇θ(K)L[1:j](θ) + ∇2
θ(K)L[1:j](θ)(θ∗−θ) + C ⊙(θ(K)−θ)T (θ(K)−θ) + o(ϵ)
≈o(µϵ) + ∇2
θ(K)L[1:j](θ)(θ∗−θ) + β2(∇θLj(θ))2 + o(ϵ)
≈o(µϵ) + ∇2
θ(K)L[1:j](θ)(θ∗−θ) + β2∇θLj(θ) + o(δ) + o(ϵ)
≈∇2
θ(K)L[1:j](θ)(θ∗−θ) + β2∇θLj(θ) + o(δ) + o(ϵ) + o(µϵ) = 0,
(3)
where C means a constant vector, ⊙denotes the element-wise multiplication operator. Then it is
easy to get θ := θ −β2(∇2
θ(K)L[1:j](θ))−1∇θLj(θ) = θ −α(Hj
M)−1∇θLj(θ), where α = β2 and
Hj
M = ∇2
θ(K)L[1:j](θ) .
(K inner steps adaption) For MCL with K steps of inner-loop updating, θ(K) = θ −β∇Lj
(1)(θ) −
β∇Lj
(2)(θ(1)) −... −β∇Lj
(K)(θ(K−1)). Similar to Eqn. (3), we can get
θ := θ −α(Hj
M)−1(
K−1
X
i=0
∇Lj
(i+1)(θ(K−1))),
where θ(i) denotes the model parameters after the i-th inner loop adaptation and θ(0) = θ. It is worth
noting that in MCL with multiple updating steps, the hypergradient incorporates more gradients
from the inner loop, which is the main difference from the one-step inner loop MCL. However, this
difference does not affect the variance reduction analysis presented in Proposition 3.
A.3
PROOF OF PROPOSITION 3
A.3.1
PROOF OF LEMMA 1
Lemma 1. (The linear combination of two invertible matrices is invertible under finite conditions.)
Let A = [a1, a2, ..., an] ∈Rn×n and B = [b1, b2, ..., bn] ∈Rn×n denote two square matrices,
where ai ∈Rn, bi ∈Rn are the i-th column vector of the matrix A and B. Suppose: 1) A and B
are invertible ; 2) for any λi ̸= 0, i = {1, 2, ..n}, (Pn
i=1 λiai)(Pn
i=1 λibi) ≥0, then the linear
combination of A and B (i.e., rA + (1 −r)B) is invertible, r ∈(0, 1).
Proof. (By contradiction.)
(Step-1) We assume rA+(1−r)B, where r ∈(0, 1), is not invertible, which means the square
matrix rA + (1 −r)B = [ra1 + (1 −r)b1, ..., ran + (1 −r)bn] is singular.
15
Published as a conference paper at ICLR 2024
(Step-2) Then there must exists λi ̸= 0, i = {1, 2, ..n}, such that Pn
i=1 λi(rai+(1−r)bi) = 0.
That is equivalent to r Pn
i=1 λiai + (1 −r) Pn
i=1 λibi = 0. According to condition 2, it can deduce
that Pn
i=1 λiai = Pn
i=1 λibi = 0. This is contradict with the condition that A and B are invertible.
(Step-3) Since assuming rA + (1 −r)B is not invertible leads to a contradiction, therefore,
rA + (1 −r)B is convertible.
A.3.2
PROOF OF PROPOSITION 3
Proposition 3. Assume that the batch size for inner step adaptation is sufficiently large. Let Hj
Mb =
∇2
θb(K)L[1:j]b(θb)=[h1
b, h2
b,· · ·, hD
b ] denote the Hessian at θb calculated on ϵb, where hd
b is the d-th
column vector of Hj
Mb. Similarly, bHj
Mb−1 =[ˆh1
b−1,· · ·, ˆhD
b−1] denotes the Hessian for the momentum
term at the (b−1)-th iteration. If (PD
d=1 λdhd
b)(PD
d=1 λdˆhd
b−1) ≥0 holds for any λd ̸= 0, d =
{1, 2,· · ·, D}, then we have the following iterative update rule for VR-MCL,
θ :≈θ −α(Hj
VR)−1∇θLj(θ),
where (Hj
VR)−1 =(Hj
Mb)−1+r(( bHj
Mb−1)−1−(H
j
Mb−1)−1) with H
j
Mb−1 denoting the Hessian at θb−1 cal-
culated on ϵb. The eigenvalues of (Hj
VR)−1 are approximately given by v−1
b +r((ˆvb−1)−1−(vb−1)−1),
where vb, ˆvb−1, and vb−1 denote the eigenvalues of Hj
Mb, bHj
Mb−1, and H
j
Mb−1, respectively.
Proof. Let gϵb
θb denote the gradient of the b-th outer step. In proposition 2, the updating formula of
MCL is θ := θ −α(Hj
M)−1∇θLj(θ) which means the gradient gϵb
θb can be rewrite in the form of
(Hj
Mb)−1∇θLj(θ). When we adopt the variance reduction method, let m denote the momentum
part, then we can get,
m1 = gϵ1
θ1 = (Hj
M1)−1∇θLj(θ1),
m2 = gϵ2
θ2 + r(m1 −gϵ2
θ1) = (HM j
2 )−1∇θLj(θ2) + r((Hj
M1)−1∇θLj(θ) −(H
j
M1)−1∇θL
j(θ1)),
· · ·
mb = gϵb
θb + r(mb−1 −gϵb
θb−1),
where (H
j
M1)−1 and ∇θL
j(θ) are the Hessian and corresponding gradient in gϵ2
θ1. When there
has a sufficiently large batch size on the current task, then ∇θLj(θ2) ≈∇θL
j(θ1) ≈∇θLj(θ1).
Therefore, we can rewrite m2 as,
m2 = gϵ2
θ2 + r(m1 −gϵ2
θ1) = {(Hj
M2)−1 + r((Hj
M1)−1 −(HM1)−1)}∇θLj(θ2).
Since (H
j
M1)−1 is highly related with (H
j
M2)−1, we can get,
(Hj
M2)−1 + r((Hj
M1)−1 −(H
j
M1)−1) ≈(1 −r)(Hj
M2)−1 + r(Hj
M1)−1
• For m2, Hj
M1 = [h1
1, h2
1, ..., hD
1 ] and Hj
M2 = [h1
2, h2
2, ..., hD
2 ] are also highly related. Based on the
assumption that for any λd ̸= 0, d = {1, 2, ..D}, (PD
d=1 λdhd
1)(PD
d=1 λdhd
2) ≥0. According to
Lemma 1, we can get r(Hj
M1)−1 + (1 −r)(Hj
M2)−1 is invertible, and thus there exist a matrix
( bHM2)−1 = r(HM1)−1 + (1 −r)(HM2)−1.
• For mb, the previous assumption also holds, which is for Hj
Mb = [h1
b, ..., hD
b ], bHj
M(b−1) =
[ˆh1
b−1, ..., ˆhD
b−1] , for any λd ̸= 0, d = {1, 2, ..D}, (PD
d=1 λdhd
b)(PD
d=1 λdˆhd
b−1) ≥0. Ac-
cording to this recurrence relation, we can get ( bHMb)−1 = r( bHMb−1)−1 + (1 −r)(HMb)−1. Let
Hj
VR = bHMb, we can get the following iterative update rule of VR-MCL,
θ :≈θ −α(Hj
VR)−1∇θLj(θ),
16
Published as a conference paper at ICLR 2024
A.4
REGRET BOUND OF VR-MCL
This section aims to establish the regret bound of VR-MCL. Our method involves three key parts:
Firstly, we define the objective form of Regret Bound in continual learning. Secondly, we prove
that updating the objective of VR-MCL is equivalent to updating the latent optimal loss function, by
bounding the gradient of VR-MCL with the latent optimal gradients. Finally, we derive the regret
bound of VR-MCL based on the aforementioned steps.
(Part I) We first define the Regret Bound in continual learning.
Definition 1 (The Regret Bound in CL). Consider an online learning scenario where an agent
makes a decision by choosing an action θt ∈Rd at each time step t = 1, ..., T, and experiences a
corresponding loss Lt(θt). The primary objective of the agent is to minimize the difference between
its cumulative loss and that of the best action in hindsight, which is commonly known as the regret.
RT = PT
t=1 Lt (θt) −minθ∈Θ
PT
t=1 Lt(θ)
Similarly, in MCL, for the current task T j, the goal is to minimize CRj = ˜F(θ) −F(θ∗), where
˜F(θ) := Pj
i=1 Li(ˆθj)
and
F(θ∗) = minθ
Pj
i=1 Li(θ).
This proof is grounded on certain assumptions, which are listed below:
Assumption 1. The compact convex set C ⊆Rd has diameter D, i.e., ∀θ, θ
′ ∈C, ∥θ −θ
′∥≤D.
Assumption 2. The function F(·) is G-Lipschitz, µ-strongly convex, φ-smooth (i.e., has φ-Lipschitz
gradients) and has κ-Lipschitz hessian. Then, according to the Theorem 1 in Finn et al. (2019), when
the step size α < { 1
2φ,
µ
8κG},the gradient ∇˜F(x, ϵt) is L-Lipschitz (L = 9φ
8 ) over the constraint set
C, that is
∥∇˜F(θ, ϵt) −∇˜F(θ
′, ϵt)∥≤L∥θ −θ
′∥= 9φ
8 ∥θ −θ
′∥, ∀θ, θ
′ ∈C.
Assumption 3. Let ˜Ft denote the t-th training step, then the distance between our computed
∇˜Ft(θ, ϵt) and the gradient derived from the optimal function is bounded over the constraint set
C, i.e., for any θ ∈C, t ∈{1, ..., T}, there exists σ2 < ∞such that with probability 1,
∥∇˜Ft(θ, ϵt) −∇F(θ)∥2 ≤σ2
Assumption 4. Let Ft(θ) mean the optimal t-th training step, and the difference of Ft(θ) and F(θ)
is bounded over the constraint set C, i.e., ∀θ ∈C, t ∈{1, ..., T}, there exists M 2 < ∞such that
with probability 1,
∥Ft(θ) −F(θ)∥2 ≤M 2
(Part II) In the following, we will demonstrate that updating the objective function of VR-MCL can
be regarded as updating the latent optimal loss function. This is achieved by bounding the gradient
of VR-MCL with the latent optimal gradients. Specifically, the updating gradient can be expressed
as mt = ∇˜F(θt, ϵt) + r(mt−1 −∇˜F(θt−1, ϵt)). Let α denote the small learning rate and define
∆t = mt −∇F(θt), our aim is to prove that ∆t is bounded. Here, we suppose the momentum ratio
r = (1 −ρt), the learning rate α = ηt, and ρt = ηt = 1/(t + 1)γ, γ ∈(0, 1].
Lemma 2. If Assumptions 1,2,3 are satisfied, ∀t ≥1, δ0 ∈(0, 1), we have with probability at least
1 −δ0,
∥∆t∥≤2

2LD +
3γσ
3γ−1

(t + 1)−γ/2p
2 log (4/δ0)
Lemma 2 demonstrates that the gradient approximation error ∥∆t∥is bounded with high probability
as t increases.
17
Published as a conference paper at ICLR 2024
Proof. We derive the formulation of ∆t at first.
∆t = mt −∇F(θt)
= (1 −ρt)∆t−1+(1 −ρt)(∇˜Ft(θt, ϵt)−∇˜Ft(θt−1, ϵt)−(∇F(θt)−∇F(θt−1)))
+ ρt(∇˜Ft(θt, ϵt)−∇F(θt))
=
τY
k=2
(1−ρk)∆1 +
t
X
τ=2
tY
k=τ
(1−ρk)(∇˜Fτ(θτ, ϵτ)−∇˜Fτ(θτ−1, ϵτ) −(∇F(θτ)−∇F(θτ−1)))
+
t
X
τ=2
ρτ
tY
k=τ+1
(1 −ρk)(∇˜Ft(θτ, ϵτ) −∇F(θτ))
Let ∆t = Pt
τ=1 ζt,τ, where ζt,1 = rt−1∆1 and for τ > 1, ζt,τ = Qt
k=τ(1 −ρk)(∇˜Fτ(θτ, ϵτ) −
∇˜Fτ(θτ−1, ϵτ) −(∇F(θτ) −∇F(θτ−1))) + ρτ
Qt
k=τ+1(1 −ρk)(∇˜Fτ(θτ, ϵτ) −∇F(θτ)). Recall
∆1 = ∇˜F(θ1, ϵ1)−∇F(θ1). We observe that E [ζt,τ∥Fτ−1] = 0 when Fτ−1 is the σ-field generate
by {L1, ϵ1, ..., Lτ−1, ϵτ−1}. Therefore, {ζt,τ}t
τ=1 is a martingale difference sequence. Then, we
will derive upper bounds of ∥ζt,τ∥. We initially proved the following results,
tY
k=τ
(1 −ρk) =
tY
k=τ

1 −
1
(k + 1)γ

=
tY
k=τ
(k + 1)γ −1
(k + 1)γ
≤
tY
k=τ
kγ
(k + 1)γ =
τ γ
(t + 1)γ ,
where the inequality holds from the concavity of h(x) = xγ for any x ≥0. Based on this inequality
we can bound ∥ζt,1∥as follows,
∥ζt,1∥≤
2γ
(t + 1)γ

∇˜F1(θ1, ϵ1), ∇F(θ1)

 ≤
2γσ
(t + 1)α
def
= ct,1,
where the second inequality holds according to Assumption 3. When τ ≥1, we can get,
∥ζt,τ∥≤
tY
k=τ
(1 −ρk)(

∇˜Fτ(θτ, ϵτ) −∇Fτ(θτ−1, ϵτ)

 + ∥∇F(θτ) −∇F(θτ−1)∥)
+ ρτ
tY
k=τ+1
(1 −ρk)∥∇˜Fτ(θτ, ϵτ) −∇F(θτ)∥
≤2L∥θτ −θτ−1∥
tY
k=τ
(1 −ρk) + σρτ
tY
k=τ+1
(1 −ρk)
≤2LDρτ−1
tY
k=τ
(1 −ρk) + σρτ
tY
k=τ+1
(1 −ρk)
(4)
We can further derive the final term as follows,
ρτ
tY
k=τ+1
(1 −ρk) =
ρτ
ρτ−1 (1 −ρτ)

ρτ−1
tY
k=τ
(1 −ρk)
!
≤
1
1 −ρτ

ρτ−1
tY
k=τ
(1 −ρk)
!
≤
1
1 −1/3γ

ρτ−1
tY
k=τ
(1 −ρk)
!
≤
3γ
3γ −1

ρτ−1
tY
k=τ
(1 −ρk)
!
.
(5)
By plugging Eqn. (5) into Eqn. (4, we can get, ∀τ ≥1
∥ζt,τ∥≤(2LD +
3γσ
3γ −1)ρτ−1
tY
k=τ
(1 −ρk)
def
= ct,τ
According to Theorem 3.5 in Pinelis (1994), we can get ∀λ ≥0,
P (∥∆t∥≥λ) ≤4 exp

−
λ2
4 Pt
τ=1 c2
t,τ
!
,
(6)
18
Published as a conference paper at ICLR 2024
where ct,1 is defined in Eqn. (A.4) and ct,τ, (τ ≥1) is defined in Eqn. (4). Then we can get,
t
X
τ=1
c2
t,τ = c2
t,1 +
t
X
τ=2
c2
t,τ =
22γσ2
(t + 1)2γ +

2LD +
3γσ
3γ −1
2
t
X
τ=2

ρτ−1
tY
k=τ
(1 −ρk)
!2
≤
22γσ2
(t + 1)2γ +

2LD +
3γσ
3α−1
2
(t + 1)γ
≤
(
√
2)γσ
2
(t + 1)γ
+

2LD +
3γσ
3γ−1
2
(t + 1)γ
≤
2

2LD +
3γσ
3γ−1
2
(t + 1)γ
,
(7)
where the last inequality is because (
√
2)γ ≤3γ/(3γ −1) ∀γ ∈(0, 1]. Substituting Eqn. (7) into
Eqn. (6) and setting λ = 2

2LD +
3γσ
3γ−1

(t+1)−γ/2p
2 log (4/δ0) for some δ0 ∈(0, 1), we have
with probability at least 1 −δ0,
∥∆t∥≤2

2LD +
3γσ
3γ−1

(t + 1)−γ/2p
2 log (4/δ0),
which is the expected result.
(Part III) Next, we will establish the regret bound CRj for continual learning.
Theorem 1 (Regret Bound of VR-MCL). If F is convex and all aforementioned four Assumptions
are satisfied, then with probability at least 1 −δ for any δ ∈(0, 1),
CRj ≤(log T + 1)(F(θ1) −F(θ∗)) + LD2(log T + 1)2
2
+ LD2(log T + 1)2
2
+
16LD2 + 16σD + 4M
 p
2T log(8T/δ) = ˜O(
√
T)
Proof. We initially define a sequence st = ˜Ft(θt)−Ft(θ∗)−(F(θt)−F(θ∗)), t = 1, ..., T. It can be
observed that E [st | Ft−1] = 0, where Ft−1 is the σ-algebra generated by {F1, ϵ1, ..., Ft−1, ϵt−1}.
It means that {st}T
t=1 is a martingale difference sequence. According to Assumption 4, we have
∥st∥= ∥˜Ft(θt) −Ft(θ∗) −(F(θt) −F(θ∗)∥≤2M
According to Theorem 3.5 in Pinelis (1994), we can get
P

∥
T
X
t=1
st∥≥λ
!
≤4 exp

−
λ2
16TM 2

,
where λ > 0. By setting λ = 4M
p
Tlog(8/δ) , we have with probability at least 1 −δ/2,
T
X
t=1
st =
T
X
t=1
( ˜Ft(θt) −Ft(θ∗)) −
T
X
t=1
(F(θt) −F(θ∗)) ≤4M
p
Tlog(8/δ).
By rearranging the above terms, we get
CRj = ˜F(θ) −F(θ∗) =
T
X
t=1
( ˜Ft(θt) −Ft(θ∗)) ≤
T
X
t=1
(F(θt) −F(θ∗)) + 4M
p
Tlog(8/δ). (8)
19
Published as a conference paper at ICLR 2024
Then according to Lemma 2 in (Mokhtari et al., 2020), we can derive the first term in Eqn. (8) as
follows,
F(θt) −F(θ∗) = (1 −ηt)(F(θt−1) −F(θ∗)) + ηtD∥∆t−1∥2 + LD2η2
t
2
≤
t−1
Y
τ=1
(1 −ητ)(F(θ1) −F(θ∗)) +
t−1
X
τ=1
ητ(D∥∆τ∥2 + LD2ητ
2
)
t−1
Y
k=τ+1
(1 −ηk)
= 1
t (F(θ1) −F(θ∗)) +
t−1
X
τ=1
1
τ + 1(D∥∆τ∥2 +
LD2
2(τ + 1))τ + 1
t
= 1
t (F(θ1) −F(θ∗)) + 1
t
t−1
X
τ=1
(D∥∆τ∥2 +
LD2
2(τ + 1))
≤1
t (F(θ1) −F(θ∗)) + D
t
t−1
X
τ=1
∥∆τ∥2 + LD2 log t
2t
(9)
Summing Eqn. (9) from t = 1 to T, we obtain,
T
X
t=1
F(θt) −F(θ∗) ≤
T
X
t=1
1
t (F(θ1) −F(θ∗)) +
T
X
t=1
t−1
X
τ=1
D
t ∥∆τ∥2 +
T
X
t=1
LD2 log t
2t
≤
T
X
t=1
1
t (F(θ1) −F(θ∗)) +
T
X
t=1
t−1
X
τ=1
D
t ∥∆τ∥2 + LD2 log T
2
T
X
t=1
1
t
≤(log T + 1)(F(θ1) −F(θ∗)) +
T
X
t=1
t−1
X
τ=1
D
t ∥∆τ∥2 + LD2(log T + 1)2
2
(10)
According to Lemma 2, we can obtain the second term is,
T
X
t=1
t−1
X
τ=1
D
t ∥∆τ∥≤4(LD2 + σD)
p
2 log(8T/σ)
T
X
t=1
t−1
X
τ=1
1
t√τ + 1
≤4(LD2 + σD)
p
2 log(8T/σ)
T
X
t=1
2
√
t
t
≤16(LD2 + σD)
p
2T log(8T/σ)
(11)
Substituting Eqn. (11) into Eqn. (10), we have with probability at least 1 −δ/2,
T
X
t=1
F(θt)−F(θ∗) ≤log(T+1)(F(θ1)−F ∗(θ))+LD2(log T + 1)2
2
+16(LD2+σD)
p
2T log(8T/δ)
(12)
Combining Eqn. (12) with Eqn. (8) we have with probability at least 1 −δ,
CRj ≤(log T + 1)(F(θ1) −F(θ∗)) + LD2(log T + 1)2
2
+ LD2(log T + 1)2
2
+
16LD2 + 16σD + 4M
 p
2T log(8T/δ) = ˜O(
√
T)
B
RELATED WORKS
Regularization-based methods (or parameter regularization methods) usually construct a
quadratic regularization term to slow down the learning of weights that are important to prior tasks.
As we know, the Fisher information matrix is equivalent to the Hessian matrix if the loss function
20
Published as a conference paper at ICLR 2024
is negative log-likelihood and we get a ground truth probabilistic model (Chapter 4.5 of (Keener,
2010)). In this way, EWC (Kirkpatrick et al., 2017) and on-EWC (Husz´ar, 2017), as the first work
applying parameter regularization, utilize the diagonal Fisher information matrix to approximate
the Hessian matrix, while Kronecker factored Laplace approximation (KFLA) (Ritter et al., 2018)
exploits Kronecker factored Laplace to compute the Fisher information matrix with off-diagonal el-
ements. To account for the trajectory of model training, IS (Zenke et al., 2017) proposes an online
estimation of a diagonal matrix and assigns weights based on its contribution to loss decay. On the
other hand, in class-incremental learning, IADM (Yang et al., 2019) and CE-IDM (Yang et al., 2021)
report that different layers have varying characteristics. Shallow layers with limited representation
tend to converge faster, while deep layers with powerful discrimination abilities tend to converge
more slowly. Therefore, based on EWC (Kirkpatrick et al., 2017), IADM (Yang et al., 2019) learns
an online layer-wise importance matrix.
Rehearsal-based methods address catastrophic forgetting by replaying previous task samples from
a memory buffer. Experience Replay (ER) (Rolnick et al., 2019) directly sample data from previous
tasks and put them jointly train with current data. On the basis of ER, recent studies have further
extended this idea. Meta Experience Replay (MER) (Riemer et al., 2019) views replay as a meta-
learning problem for maximizing the transfer from previous tasks and minimizing the interference.
Gradient-based Sample Selection (GSS) (Aljundi et al., 2019) pays attention to the samples stored
in the memory module and hopes to increase the diversity of samples in the gradient space. Gradient
Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) and its lightweight variant Averaged-GEM
(A-GEM) (Chaudhry et al., 2018) formulate optimization constraints such that gradients between
current and old training data are aligned to further determine the final direction of optimization.
Dark Experience Replay (DER++) (Buzzega et al., 2020) adds the knowledge distillation part on top
of ER to regularize the logits of samples stored in the memory buffer. CLSER (Arani et al., 2021)
introduces an innovative approach to episodic replay (ER) by presenting a dual memory framework.
This framework incorporates both short-term and long-term semantic memories, allowing for their
interaction with the episodic memory. CBA (Wang et al., 2023a) proposes to address the issue of
recency bias by introducing a specially designed bias attractor, guided by the memory buffer M.
Meta-Continual Learning (Meta-CL) is a class of methods that apply meta-learning (Finn et al.,
2017; Wu et al., 2022; 2021; Wang et al., 2023b) in continual learning. From the bi-level optimiza-
tion perspective to analyze meta-learning, its inner-loop (or lower-level) optimization usually fits the
training data, while the outer-loop (higher-level) centralized test data aims to increase the model’s
generalization ability. Therefore, meta-learning is well suited for continual learning, as we want
the model to fit the current task and still perform well on all observed tasks (i.e., previous and cur-
rent tasks). Meta Experience Replay (MER) (Riemer et al., 2019), inspired by GEM (Lopez-Paz &
Ranzato, 2017), utilizes replay to incentivize the alignment of gradients between old and new tasks
to further maximize the transfer from previous tasks and minimize the interference. Online-aware
Meta Learning (OML) (Javed & White, 2019) adopts a pre-training algorithm to learn an optimal
representation offline, which is frozen when training in the downstream tasks. However, this offline
training manner and assumption that the training data is a correlated data stream limit its applicabil-
ity to practical scenarios. Besides, MER (Riemer et al., 2019) and OML (Javed & White, 2019) use a
single sample at one time during the inner-loop optimization, which is time-consuming and not suit-
able for large-scale CL. To alleviate these problems, Look-ahead Meta learning (La-MAML) (Gupta
et al., 2020) proposed an online gradient-based meta-learning algorithm and proved it is equivalent
to a simplified version of gradient alignment. On the other hand, some Meta-CL algorithms (Ra-
jasegaran et al., 2020; 2022) focus on combining meta-learning with task-agnostic continual learning
that does not require task information during training and testing. Besides, the variance reduction
has also been integrated with first-order meta-learning (Yang & Kwok, 2022; Wang et al., 2021).
In this work, we focus on the online continual learning setting and provide a variance reduction
Meta-CL (second-order) based on our novel understanding of Meta-CL, it also links Meta-CL with
regularization-based methods.
Online and Imbalanced Continual learning, different from the traditional offline CL setting, are
more challenging CL settings that have not been explored as much. For online CL scenarios, the
initial methods still have been developed based on similar ways in offline CL (Lopez-Paz & Ranzato,
2017; Chaudhry et al., 2018; Aljundi et al., 2019). Recently, there has been increasing interest in
developing online CL methods that can fully utilize the single-pass data stream to decrease the
performance difference between online and offline CL. OCDVC (Gu et al., 2022) advocates for
21
Published as a conference paper at ICLR 2024
the dual view consistency strategy, while SDP (Koh et al.) utilizes knowledge distillation to fully
leverage the data stream. OBC (Chrysakis & Moens, 2023) proposes a simple approach to mitigate
the bias of online CL by changing the optimization manner of the output layer model. OCM (Guo
et al., 2022) combines these motivations and aims to reduce feature bias and fully utilize streaming
data through mutual information maximization. However, none of these methods directly focus on
the performance drop between online and offline CL and analyze the underlying reasons, which is a
motivation of this work.
For imbalanced CL, where the sample number of each task is different (Lai et al., 2022b;a), most
works aim to directly design a sampling strategy that can balance the distribution of samples re-
trieved from the memory buffer that stores data from previous tasks. CBRS (Chrysakis & Moens,
2020) introduced a novel mechanism for class-balancing reservoir sampling based on the sample
distribution within the buffer. PRS (Kim et al., 2020) utilizes current data stream statistics to deter-
mine the sample-in and sample-out process, maintaining a balanced replay memory. Additionally,
InfoRS (Sun et al.) introduces a stochastic information theoretic reservoir sampler to select and store
the informative samples. Coresets (Borsos et al., 2020) employs the bi-level optimization technique
for the same purpose. Different from these methods, our proposed approach takes a different per-
spective. It aims to conduct a comprehensive analysis of the factors that contribute to the substantial
performance drop observed in both imbalanced and online settings compared to offline learning.
By understanding these underlying factors, we propose the VR-MCL to the model performance in
imbalanced CL scenarios.
Variance Reduction is a widely used technique for reducing the variance in stochastic gradients
and accelerating the optimization process. Early research on variance reduction methods, such as
SAG Gower et al. (2020), SAGA (Defazio et al., 2014), and SVRG (Johnson & Zhang, 2013), was
primarily focused on strongly convex optimization problems. However, in recent years, these meth-
ods have been extended to general non-convex optimization problems (Allen-Zhu & Hazan, 2016;
Nguyen et al., 2017; Fang et al., 2018). Nonetheless, these methods typically require the expensive
computation of the full-batch gradient by processing all samples in a task, which is impractical in
the context of online continual learning where samples are processed in mini-batches and the com-
plete sample set is unavailable. To alleviate this expensive computation problem, momentum-based
variance reduction algorithms (Cutkosky & Orabona, 2019; Khanduri et al., 2021) are proposed.
Specifically, these methods incorporate a momentum term to compute the stochastic gradients at two
consecutive iterations on the same set of stochastic samples, thereby replacing the time-consuming
computation of the full-batch gradient. Additionally, these methods have the same asymptotic con-
vergence rate as other variance reduction methods.
C
IMPLEMENTATION DETAILS
C.1
EXPERIMENTAL SETTINGS
Imbalanced Settings. We conducted imbalanced online continual learning experiments on Seq-
cifar10, consisting of five tasks, each with two classes. As shown in Fig. 4, when the imbalance
ratio γ = 2, the number of samples for each class across all tasks is [5000, 4629, 4286, 3968, 3674,
3401, 3149, 2916, 2700, 2500]. We also considered the reversed version, where the number of
samples for each class is [2500, 2700, 2916, 3149, 3401, 3674, 3968, 4286, 4629, 5000]. Without
loss of generality, we additionally sampled a random version with [2700, 2500, 5000, 4286, 3674,
3968, 3149, 3401, 2916, 4629] samples per class across all tasks.
Baselines.
We compared the proposed VR-MCL to various baseline approaches, including
regularization-based methods, rehearsal-based methods, meta-continual learning methods, and spe-
cific CL methods designed for imbalanced and online settings.
Regularization-based methods:
• On-EWC: (Husz´ar, 2017). It approximates each Hessian matrix by a diagonal Fisher matrix
calculated at the end of each task training. The larger Fisher information value means the
corresponding weights are more important to the old tasks and thus should undergo less change
in future tasks.
22
Published as a conference paper at ICLR 2024
1
2
3
4
5
6
7
8
9
10
Class index
0
1000
2000
3000
4000
5000
 Sample size
Imbalance ratio =2
(a) Traditional imbalance.
1
2
3
4
5
6
7
8
9
10
Class index
0
1000
2000
3000
4000
5000
 Sample size
Imbalance ratio =2 (Reverse)
(b) Reverse imbalance.
1
2
3
4
5
6
7
8
9
10
Class index
0
1000
2000
3000
4000
5000
 Sample size
Imbalance ratio =2 (Random)
(c) Random imbalance.
Figure 4: Different imbalance settings of Seq-CIFAR10, where the imbalance ratio γ = 2.
• IS: (Zenke et al., 2017).
It introduces the intelligent synapse.
Each synapse accumulates
task-relevant information over time so as to rapidly store new memories without forgetting the
learned ones. From the Hessian approximation perspective, it constructs a diagonal regulariza-
tion matrix (i.e., a generalization form of the Hessian matrix integrating over time) to estimate
the Hessian matrix.
• LWF: (Li & Hoiem, 2017). It is similar to joint training, but the key difference is that LWF
does not require data and labels from the old tasks, instead employing distillation techniques.
Rehearsal methods:
• GEM and A-GEM (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018). They formulate
optimization constraints such that gradients between current and old training data are aligned to
determine the final direction of optimization.
• ER (Rolnick et al., 2019). Training current task with data sampled from the memory buffer
which stores the examples from previous tasks.
• DER/DER++ (Buzzega et al., 2020). Adding the distillation loss of logits on the top of ER.
• CLSER (Arani et al., 2021). The dual memory experience replay (ER) method maintains short-
term and long-term semantic memories that interact with episodic memory.
CL methods focus on imbalance and online settings:
• CBRS (Chrysakis & Moens, 2020). It introduces a novel mechanism for class-balancing reser-
voir sampling based on the sample distribution within the buffer to ensure more balanced sam-
ples are stored.
• Coresets (Borsos et al., 2020).
It presents a novel coreset construction technique through
cardinality-constrained bilevel optimization, which is effective in the imbalanced CL setting.
• OCM (Guo et al., 2022). It proposes to reduce feature bias and fully exploit streaming data
through mutual information maximization. Note that, for a fair comparison, we choose its
OCM (no local augmentation) for evaluation.
• ER-OBC (Chrysakis & Moens, 2023). It proposes a simple approach to mitigate bias in online
CL by modifying the optimization process of the output layer model.
Meta-CL methods:
• MER (Riemer et al., 2019). It utilizes replay examples to incentivize the alignment of gradients
between old and new tasks to further maximize the transfer from previous tasks and minimize
interference.
• La-MAML (Gupta et al., 2020). It proposed an online gradient-based meta-learning algorithm
and proved it is equivalent to a simplified version of MER. Note that in the online scenario
of La-MAML (Gupta et al., 2020), the mini-batch samples instead of the whole task could be
trained multiple times, which is not the strict online continual learning. In our experiments, all
methods follow the standard online setting where all samples can only be trained once.
Hyper-parameters. The hyperparameters used for each experimental setting are listed in Table 7.
Let lr denote the learning rate, bs denote the batch size and mbs means the minibatch size (i.e.,
the batch size drawn from the buffer). It is worth noting that we used the same hyperparameters
across different datasets and buffer sizes, which shows our method has good generalizability without
requiring extensive hyperparameter tuning for each specific setting.
23
Published as a conference paper at ICLR 2024
Table 7:
Hyperparameters on all the three datasets including Seq-Cifar10/100 and Seq-
TinyImgaeNet.
Backbone
Epochs Buffer size Inner lr (β) Outer lr (α)
r
inner-bs
bs
mbs
PcCNN
1
200
0.1
0.25
0.25
2
10
10
PcCNN
1
600
0.1
0.25
0.25
2
10
10
PcCNN
1
1000
0.1
0.25
0.25
2
10
10
Reduced ResNet-18
1
200
0.1
0.25
0.25
8
32
32
Reduced ResNet-18
1
600
0.1
0.25
0.25
8
32
32
Reduced ResNet-18
1
1000
0.1
0.25
0.25
8
32
32
C.2
THE MASK TRAINING OF THE PROPOSED VR-MCL
For the implementation of the VR-MCL, we adopt mask training different from La-MAML (i.e., one
mask training strategy on the final output layer). In this subsection, we will first introduce two mask
training approaches: Mask-LM, which is the original mask training in La-MAML, and Mask-VR,
which is the proposed mask training used in VR-MCL. We will then provide both mathematical and
experimental analysis to compare the two mask training methods.
�1
�2
�3
�4
�5
�1
�2
�3
�4
�5
�1
�2
�3
�4
�5
Inner loop
  Outer loop
   (Mask-VR)
����
����
→
���→
→
�1
�2
�3
�4
�5
�1
�2
�3
�4
�5
Outer loop
(Mask-LM)
����
���→
→
Unseen
  tasks
Previous
tasks
(not masked) Preserved predicted probability
Predict probability of one class
Figure 5: The illustration of the two mask training approaches, where the current training task is T 3.
Introduction of the two Mask training methods As the output layer is shared among all tasks,
meta-continual learning methods have the problem of a mismatch between the limited rehearsal
data from previous tasks T [1:j−1] and the larger incoming data from the current task T j. To solve
this problem, a masked softmax loss is applied by La-MAML Gupta et al. (2020). As shown in
Figure 5, La-MAML applies a mask (i.e., Mask-LM) that only keeps the logits of the corresponding
classes within a task to both the loss in the inner-loop adaptation and outer-loop optimization. The
Mask-LM improves the inner-loop adaptation for the current tasks and mitigates the interference
between the limited data sampled from the memory buffer M and the larger incoming data from
the current task. However, the employment of the Mask-LM in the outer-loop still has a limitation.
The left part of Mask-LM concentrates solely on minimizing the intra-task loss, without consider-
ing the interdependencies between different classes from various tasks. This oversight significantly
reduces the model’s efficacy in the class incremental setting. To address this issue, we propose a
modification (i.e., Mask-VR) inspired by (Caccia et al., 2021), where we mask all seen classes that
appeared in the memory buffer M. This modification enables the model to distinguish samples
belonging to different tasks, which is shown in the left part of Mask-VR. Mathematical and Ex-
perimental Analysis. The main idea of Mask-VR to enlarge the preserved logits is to introduce the
negative gradients so as to get more concise class prototypes and thus achieve better performance.
To investigate the impact of the proposed Mask-VR, we conducted experiments and demonstrated
empirically that it can effectively reduce the gradient variance. Furthermore, our findings suggest
that the Mask-VR and the proposed variance reduction method can be synergistically combined to
effectively reduce the gradient variance.
We will begin by conducting the mathematical analysis. Note that the following mathematical
symbols are solely for the purpose of illustrating the Mask-VR, they are not aligned with the math-
ematical symbols used in the main text.
24
Published as a conference paper at ICLR 2024
Let Cseen refer to all seen classes, and |Cseen| is the number of classes the model has seen. In
Figure 5, the current training task is T 3 and |Cseen| = 6. Let the model architecture consists of two
parts: feature extractor hθ and classifier fϕ and the output of the model before the softmax operation
is o(x; θ, ϕ) = fϕ(hθ(x)). Then the cross-entropy loss in continual learning Lce is,
Lce(o(x; θ, ϕ)) = −
|Cseen|
X
i=1
lcilog(pci),
pci =
exp(oci)
P|Cseen|
s=1
exp(ocs)
where x is the training sample, lci ∈{0, 1} is the one-hot label of class ci, and o(x; θ, ϕ) =
[oc1, oc2, ..., oc|Cseen|] is the corresponding logit values of x. Let xi mean the sample of class ci,
then the gradients on the classifier fϕ is,
∂Lce
o
xi; θ, ϕ

∂oci
= pci −1,
∂Lce
o
xi; θ, ϕ

∂ocj
= pcj,
∂Lce
o
xi; θ, ϕ

∂ϕci
= ∂Lce
o
xi; θ, ϕ

∂oci
∂oci
∂ϕci
= (pci −1) hθ(xi),
[Positive gradient]
∂Lce
o
xi; θ, ϕ

∂ϕcj
= ∂Lce
o
xi; θ, ϕ

∂ocj
∂ocj
∂ϕcj
= pcjhθ(xi),
[Negative gradient]
Let W denote the matrix with all class prototypes {wc}c∈Cseen, then W ◦hθ = fϕ(hθ(x)). Ac-
cording to the above equation, we can get in Mask-LM, only wci receives a positive gradient from
xi, and there are no negative gradients due to preserving only a portion of the logits. However, in
Mask-VR, both positive and negative gradients are present, similar to the concept of positives and
negatives in contrastive loss. Consequently, the class prototypes W learned with Mask-VR exhibit
superior performance compared to those learned with Mask-LM.
0
500
1000
1500
2000
Iterations
0.00
0.25
0.50
0.75
1.00
1.25
Relative Gradient Variance
La-MAML
MCL
VR-MCL1
VR-MCL2
Figure 6: Relative gradient variance of update during training. The subscript 1 represents training
with Mask-LM, while the subscript 2 represents training with Mask-VR.
Ablation studies of the Mask-Training. To better understand the effectiveness of the Mask-VR,
and the proposed VR-MCL, we compare the performance of La-MAML, MCL(La-MAML with
Mask-VR), as well as VR-MCL1 (i.e., VR-MCL with Mask-LM) and VR-MCL2 (i.e., VR-MCL
with Mask-VR), as shown in Table 8. Moreover, we also plot the relative gradient variance of La-
MAML, MCL, VR-MCL1, and VR-MCL2, as shown in Fig. 6. Based on the fact that the curves
with variance reduction (VR) are all lower than those without VR, it can be concluded that: 1) the
proposed VR technique indeed can reduce the variance, 2) the effects of VR and Mask-VR can be
superimposed and work together.
Table 8: Performance of La-MAML, MCL, VR-MCL1 and VR-MCL2 on Seq-CIFAR10.
Method
La-MAML
MCL
VR-MCL1
VR-MCL2
Acc
33.43 ±1.21
52.40 ±2.13
35.66 ±1.13
56.48 ±1.79
AAA
42.98 ±1.60
65.87 ±1.26
50.02 ±1.80
66.97 ±1.58
25
Published as a conference paper at ICLR 2024
D
OTHER EXPERIMENTAL RESULTS.
D.1
THE TRAINING TIME ANALYSIS OF VR-MCL
To analyze the time complexity of the proposed algorithm, we compared the training time of the
proposed VR-MCL with relevant Meta-CL methods on the Seq-CIFAR-10 dataset using reduced
ResNet-18, as shown in Table 9. It is evident that our proposed VR-MCL method demonstrates
a substantial reduction in training time compared to MER. To assess the value of the additional
training time required by VR-MCL, we conducted experiments by extending the training epoch of
La-MAML from 1 to 2. Surprisingly, despite the comparable training time with VR-MCL, La-
MAML achieved a performance of only 38.89%, which is extremely lower than the performance of
VR-MCL (i.e., 56.48%). This outcome serves as strong evidence for the superiority of VR-MCL in
terms of both training efficiency and performance on Seq-CIFAR10.
Table 9: Training time comparison of different meta-continual learning methods on Seq-CIFAR10
under the online setting (i.e., the training epoch of each task is set as one.)
Method
La-MAML
MER
VR-MCL
Training Epochs
1
1
1
Training Time (s)
750.61
20697.70
1297.10
Training Epochs
2
1
1
Training Time (s)
1511.54
20697.70
1297.10
AAA
53.21%
50.99%
66.97%
Acc
38.89%
36.92%
56.48%
D.2
OTHER EVALUATION METRICS
To further evaluate the effectiveness of the proposed VR-MCL, we provide the backward transfer
(BWT) metric in Table 10, which measures the performance degradation in subsequent tasks. It can
be observed our VR-MCL achieves the best results. It is worth noting that in Table 10, La-MAML
is excluded from the analysis due to the extremely negative impact of Mask-LM utilization on its
newest training task (i.e., T j) performance (Caccia et al., 2021). This impact results in large and
even a positive backward transfer (BWT) value for La-MAML.
Table 10: The Backward Transfer of Seq-CIFAR10 and longer task sequences Seq-CIFAR100, Seq-
TinyImageNet with 95% confidence interval on reduced ResNet-18 under the online setting. The
memory buffer size is set as 1000 (i.e., |M|=1000). All reported numbers are the average of 5 runs.
Shaded areas are our methods, and ‘–’ indicates the result was omitted due to high instability.
CIFAR-10
CIFAR-100
Tiny-ImageNet
Method
BWT
BWT
BWT
SGD
-61.48 ±1.58
-44.53 ±0.38
-34.59 ±0.35
On-EWC
-65.94 ±2.08
-42.45 ±1.30
-33.05 ±0.56
IS
-63.38 ±1.19
-39.54 ±1.58
-23.76 ±1.27
A-GEM
-67.06 ±1.11
-48.74 ±0.37
-37.93 ±1.04
GEM
-62.72 ±2.00
-42.27 ±1.90
-36.21 ±0.79
ER
-33.97 ±7.07
-31.24 ±1.86
-31.07 ±1.91
DER
-62.17 ±3.48
-50.21 ±1.32
-39.50 ±0.60
DER++
-24.65 ±2.21
-46.74 ±1.13
-39.16 ±1.34
CLSER
-23.61 ±1.54
-37.02 ±1.35
-34.58 ±0.85
OCM
-22.02 ±1.92
-17.91 ±0.74
-11.99 ±0.86
ER-OBC
-34.71 ±0.81
-5.39 ±1.83
-6.20 ±1.27
MER
-28.77 ±1.25
–
–
VR-MCL
-20.92 ±1.94
-0.16 ±0.10
-2.86 ±0.83
26
Published as a conference paper at ICLR 2024
D.3
HYPERPARAMETER SELECTION
We utilized a grid search technique to select the optimal hyperparameters, including the learning rate
α and the momentum ratio r, as illustrated in Table 11 and Table 12. The results demonstrate that our
proposed method achieves the best performance when both the momentum ratio r and learning rate α
are set to 0.25. Moreover, it is worth noting that there is small performance difference observed when
different hyperparameters are chosen. This suggests that the proposed VR-MCL method exhibits
robustness and is not highly sensitive to the selection of hyperparameters.
Table 11: Reduced ResNet18 performance with
different momentum ratios r on Seq-CIFAR10,
using a fixed learning rate (α = 0.25).
r
0.15
0.25
0.35
Acc
55.46 ±0.44
56.48 ±1.79
54.14 ±1.50
AAA
66.46 ±1.23
66.97 ±1.58
63.92 ±0.89
Table 12: Reduced ResNet18 performance with
different learning rates α on Seq-CIFAR10, us-
ing a fixed momentum ratio (r = 0.25).
α
0.15
0.25
0.35
Acc
54.466 ±1.94
56.48 ±1.79
55.91 ±1.52
AAA
66.06 ±1.05
66.97 ±1.58
66.33 ±1.33
D.4
OTHER IMBALANCED CL EXPERIMENTS
Due to the page limit, the full results of Table 4 are shown in Table 13. To further explore the
performance of VR-MCL in imbalanced settings, we select the more realistic random imbalanced
scenario and present the results for a higher imbalance ratio, specifically γ = 5, in Table 14. For
simplicity, we select the SOTA methods from Table 13 as the comparable methods. The results
clearly demonstrate that the proposed VR-MCL surpasses other methods in the case of γ = 5,
providing further evidence of the effectiveness of the variance reduction technique in addressing
imbalanced CL.
To further demonstrate the effectiveness of the proposed VR-MCL under imbalanced settings, we
performed additional verification on various imbalance settings using the Seq-CIFAR100 dataset.
The results of these experiments are presented in Table 15.
Table 13: Reduced ResNet-18 performance on the imbalanced Seq-CIFAR10 (|M|=1000) with
imbalance ratio γ=2.
γ =2
γ =2 (Reversed)
γ =2 (Random)
Methods
AAA
ACC
AAA
ACC
AAA
ACC
SGD
36.64 ±0.42 16.46 ±0.31 35.57 ±0.70 17.54 ±0.16 34.62 ±1.31 17.33 ±0.34
On-EWC
38.85 ±0.18 16.92 ±0.31 37.41 ±0.26 17.79 ±0.17 37.26 ±0.49 14.35 ±1.85
A-GEM
38.50 ±0.25 17.64 ±0.43 36.72 ±0.33 17.43 ±0.47 37.79 ±0.45 17.85 ±0.22
GEM
41.29 ±0.42 17.90 ±0.65 38.46 ±1.13 18.37 ±0.25 39.63 ±1.38 18.00 ±0.73
ER
52.50 ±0.74 33.10 ±2.17 46.58 ±2.29 28.49 ±3.46 43.78 ±2.85 31.10 ±5.38
DER
46.95 ±1.54 16.82 ±0.98 40.62 ±1.69 18.63 ±0.61 41.44 ±0.65 18.78 ±0.23
DER++
62.02 ±0.52 44.14 ±2.77 58.22 ±0.85 39.21 ±3.81 60.25 ±0.58 42.83 ±2.60
CLSER
61.37 ±0.69 47.75 ±0.70 54.92 ±0.55 40.51 ±1.15 56.60 ±1.51 47.48 ±0.70
CBRS
59.07 ±1.78 43.81 ±0.13 57.57 ±1.60 44.20 ±1.96 58.16 ±1.57 44.66 ±3.21
Coresets
61.11 ±1.34 45.37 ±0.98 58.12 ±1.37 45.80 ±1.89 58.56 ±2.17 45.63 ±1.89
La-MAML 36.64 ±1.54 29.17 ±0.91 32.08 ±2.37 31.17 ±1.59 42.52 ±2.81 31.24 ±2.01
VR-MCL
65.06 ±0.85 49.82 ±1.13 61.16 ±1.32 51.36 ±1.31 61.91 ±1.07 50.74 ±1.15
Table 14: Reduced ResNet-18 performance under the random imbalanced scenario with larger im-
balance ratio, i.e., γ = 5 (Random).
Method
CLS-ER
DER++
La-MAML
VR-MCL
Acc
35.66 ±1.56
40.93 ±0.69
27.46 ±4.65
41.55 ±3.15
AAA
53.93 ±0.61
49.98 ±2.34
36.85 ±1.33
54.52 ±1.97
27
Published as a conference paper at ICLR 2024
Table 15: Reduced ResNet-18 performance on the imbalanced Seq-CIFAR100 (|M|=1000) with
imbalance ratio γ=2.
Setting
Method
CLSER
DER++
La-MAML
VR-MCL
γ = 2
Acc
13.85 ±1.19
9.19 ±1.20
11.19 ±1.07
15.12 ±1.32
AAA
23.54 ±1.02
18.78 ±0.24
19.54 ±0.38
24.02 ±1.13
γ = 2 (Reverse)
Acc
12.89 ±0.78
9.66 ±0.03
11.86 ±0.60
14.98 ±1.65
AAA
18.72 ±1.22
15.60 ±0.60
16.04 ±0.53
20.88 ±1.16
γ = 2 (Random)
Acc
13.42 ±1.48
8.60 ±0.49
11.32 ±0.91
14.93 ±0.27
AAA
19.10 ±1.89
15.55 ±0.25
16.22 ±0.58
20.13 ±0.57
D.5
OTHER RESULTS USING THE PCCNN BACKBONE
In Table 16, we further investigate the performance gain over the memory buffer size on the small
network PcCNN. Combined with the results on reduced ResNet18, it is evident that the proposed
VR-MCL has a consistent and significant performance improvement in different buffer sizes and
various network structures.
Table 16: Performance of Seq-CIFAR10 with 95% confidence interval on PcCNN. All numbers are
the average of 5 runs. |M| denotes the memory buffer size.
|M|=200
|M|=600
|M|=1000
Method
AAA
Acc
AAA
Acc
AAA
Acc
SGD
37.84 ±0.03 17.58 ±0.20 37.84 ±0.03 17.58 ±0.20 37.84 ±0.03 17.58 ±0.20
On-EWC
38.72 ±0.58 17.05 ±0.08 38.72 ±0.58 17.05 ±0.08 38.72 ±0.58 17.05 ±0.08
A-GEM
41.84 ±0.44 18.50 ±0.38 41.54 ±0.14 18.45 ±0.37 41.73 ±0.61 18.78 ±0.38
GEM
45.07 ±0.52 22.52 ±1.02 45.11 ±0.34 22.45 ±0.89 44.92 ±0.20 22.10 ±0.60
ER
54.64 ±0.91 32.46 ±1.58 58.85 ±0.82 40.29 ±1.25 59.26 ±0.69 41.71 ±1.35
DER
58.06 ±0.18 38.21 ±0.29 58.40 ±0.11 38.46 ±0.45 57.90 ±0.12 38.88 ±0.39
DER++
56.07 ±0.32 34.28 ±0.79 62.13 ±0.21 43.89 ±0.20 63.55 ±0.16 47.20 ±0.67
CLSER
58.03 ±0.75 39.38 ±1.44 61.53 ±0.32 44.94 ±1.01 62.39 ±0.34 48.24 ±0.85
MER
55.50 ±0.38 32.01 ±1.22 63.20 ±0.43 44.69 ±0.43 65.75 ±0.63 51.04 ±0.78
La-MAML 46.36 ±0.98 29.45 ±0.51 47.22 ±0.87 32.78 ±1.53 47.22 ±0.87 32.51 ±1.18
VR-MCL
60.88 ±0.22 43.41 ±0.30 63.86 ±0.52 48.85 ±0.66 66.02 ±0.31 52.67 ±0.25
D.6
OFFLINE CL PERFORMANCE
As shown in Table 17, we also provide the performance of different methods under the offline CL
setting where we set the training epochs as 5.
D.7
INFLUENCE OF THE INNER BATCH SIZE
To investigate the practical implications of the inner batch size, we conducted experiments using
various inner batch sizes. The corresponding results are presented in Table 18. The findings indicate
that a larger inner batch size consistently leads to improved performance. However, it is worth noting
that the performance gains tend to plateau as the inner batch size increases. This can be attributed to
the online nature of the setting, where each sample is observed only once. A larger inner batch size
reduces the number of outer loop steps, which may potentially compromise the overall performance
improvement.
E
ALGORITHM.
As presented in Algo. 1, Steps 1-9 present the update of θ in the first iteration, where no momentum
term is available. Steps 2-6 present the inner loop and steps 7-8 present the estimate of hypergradient
in the outer loop. Steps 10-28 represent the update of θ in b-th iteration (b > 1). Steps 12-17 are
28
Published as a conference paper at ICLR 2024
Table 17: Offline CL (i.e., 5 training epochs) performance of Seq-CIFAR10 and longer task se-
quences Seq-CIFAR100, Seq-TinyImageNet with 95% confidence interval on reduced ResNet-18.
The memory buffer size is set as 1000 (i.e., |M|=1000).
Seq-CIFAR10
Seq-CIFAR100
Seq-TinyImageNet
Method
AAA
Acc
AAA
Acc
AAA
Acc
SGD
42.26 ±0.22 18.87 ±0.66 18.73 ±1.67
7.73 ±0.66 16.89 ±0.26
6.93 ±0.03
On-EWC
42.13 ±0.18 20.63 ±1.76 16.99 ±2.12
5.34 ±0.42 14.89 ±0.14
4.85 ±0.35
IS
39.84 ±0.11 17.49 ±0.49 15.56 ±0.58
4.98 ±0.65 10.92 ±1.44
2.65 ±0.41
A-GEM
42.13 ±0.14 19.12 ±0.15 18.69 ±1.81
7.64 ±1.42 16.73 ±0.47
6.50 ±0.35
GEM
48.28 ±0.83 23.75 ±3.91 21.56 ±3.57 10.71 ±2.31 18.12 ±0.11
7.98 ±0.33
ER
74.64 ±0.76 63.97 ±0.27 35.37 ±2.36 21.84 ±1.75 26.49 ±0.38 12.47 ±0.23
DER
67.38 ±1.54 59.95 ±1.96 18.73 ±2.46
7.67 ±1.12 16.96 ±0.44
6.68 ±0.22
DER++
75.50 ±2.02 67.16 ±1.62 39.15 ±0.54 24.03 ±0.38 25.91 ±0.63 11.54 ±0.71
CLSER
75.45 ±1.10 67.36 ±0.27 40.18 ±1.70 26.16 ±0.34 26.24 ±0.33 12.98 ±0.10
ER-OBC
75.93 ±0.95 65.53 ±1.42 41.15 ±0.53 29.02 ±0.99 29.01 ±0.93 16.27 ±0.44
La-MAML 60.55 ±0.48 43.58 ±0.56 31.05 ±0.28 19.67 ±1.46 23.07 ±0.71 11.81 ±0.55
VR-MCL
77.15 ±0.75 67.34 ±1.10 40.02 ±0.89 29.32 ±0.94 30.43 ±1.00 19.60 ±0.58
Table 18: Reduced ResNet-18 performance under different inner batch sizes.
Inner batch size
6
8
10
Acc
53.80 ±2.36
56.48 ±1.79
56.81 ±1.07
AAA
65.73 ±3.06
66.97 ±1.58
67.26 ±0.61
the inner loops. Steps 18-19 show the computation of the hypergradient for θb. In steps 20-25, we
perform the inner loop to obtain θ′
b(K) and compute the hypergradient for θb as gϵb
θb−1. During the
update of memory buffer M (step 28), we employ the reservoir sampling strategy to ensure that M
is updated in a way that the stored examples are uniformly sampled from the tasks during online
training.
E.1
VARYING TASK LENGTH ANALYSIS
To further demonstrate the effectiveness of the proposed VR-MCL across varying task lengths, we
adjust the split mechanism of Seq-CIFAR100 and Seq-TinyImageNet to increase their original task
numbers. Specifically, we restructure Seq-CIFAR100, originally composed of 10 tasks with 10
classes each, into 20 tasks, each with 5 classes. Similarly, we modify Seq-TinyImageNet, initially
consisting of 20 tasks with 10 classes each, into 40 tasks, each comprising 5 classes. For clarity, we
refer to these new division datasets as Seq-CIFAR100l and Seq-TinyImageNetl respectively. The
performance of various baselines on these two datasets is presented in Table 19.
E.2
EMPIRICAL RESULTS OF THE OUTER-LOOP LOSS
Figure 7: The outer-loop training loss for each task T j in Seq-CIFAR10 (j = 2, 3, 4, 5).
29
Published as a conference paper at ICLR 2024
Algorithm 1 The Algorithm of the proposed VR-MCL.
Require: Initial model weights θ, initial inner-loop learning rate β, outer-loop learning rate α,
number of inner-loop steps K, memory buffer M, and the task number in total N.
1: Sample training data T 1
(s) (s = 1, ..., K) from current task T 1.
2: θ(0) = θ, θ0 = θ
3: for i = 1 to K do
4:
obtain samples T 1
(i)
5:
θ(i) = θ(i−1) −β∇L1
(i)(θ)
6: end for
7: sample ϵ0 from M ∪T 1
8: ˆgϵ0
θ0 = ∇θL(θ(K), ϵ0)
9: θ1 = θ0 −αˆgϵ0
θ0
10: for j = 1 to N do
11:
for b = (j−1)×B + 1 to j×B do
12:
sample training data T j
(s) (s = 1, ..., K) from current task T j.
13:
θb(0) = θb
14:
for i = 1 to K do
15:
obtain samples T j
(i)
16:
θb(i) = θb(i−1) −β∇Lj
(i)(θb(i−1))
17:
end for
18:
sample ϵb from M ∪T j
19:
gϵb
θb = ∇L(θb(K), ϵb)
20:
θ
′
b(0) = θb−1
21:
for i = 1 to K do
22:
use the same sampled training data T j
(i)
23:
θ
′
b(i) = θ
′
b(i−1) −β∇Lj
(i)(θ
′
b(i−1))
24:
end for
25:
gϵb
θb−1 = ∇L(θ
′
b(K), ϵb)
26:
ˆgϵb
b = gϵb
b + r(ˆgϵb−1
b−1 −gϵb
θb−1)
27:
θb+1 = θb −αˆgϵb
b
28:
Update the memory buffer M with T j
(i) using reservoir sampling strategy.
29:
end for
30: end for
Table 19: Experimental results under varying task lengths with |M| = 1000 and a 95% confidence
interval. The Notation N denotes the task length.
Seq-CIFAR-100l (N=20) Seq-TinyImageNetl(N=40)
Method
AAA
Acc
AAA
Acc
SGD
9.23 ±0.26
3.34 ±0.13
4.95 ±0.16
1.19 ±0.27
A-GEM
10.84 ±0.23
3.56 ±0.17
6.10 ±0.14
1.67 ±0.07
GEM
16.04 ±2.58
7.01 ±1.95
7.92 ±0.15
2.93 ±0.38
ER
20.46 ±0.48
12.71 ±0.28
13.75 ±0.12
6.82 ±0.11
DER++
16.32 ±0.49
8.24 ±0.41
9.39 ±0.07
3.76 ±0.81
CLSER
22.03 ±0.96
15.39 ±2.36
14.93 ±0.36
7.74 ±0.91
ER-OBC
21.04 ±0.65
15.87 ±1.26
14.92 ±0.20
8.45 ±2.29
La-MAML 17.42 ±0.79
10.27 ±0.46
10.83 ±0.59
5.29 ±0.28
VR-MCL
24.29 ±1.07
17.44 ±0.97
18.28 ±0.14
10.54 ±0.30
Fig. 7 illustrates the value of L[1:j](θ(K)) during the training of each task T j in Seq-CIFAR10
(j=2,3,4,5). The loss L[1:j](θ(K)) exhibits a significant decrease within an average of 5 outer loop
update steps, suggesting a close proximity between θ(K) and θ∗as stated in Proposition 2.
30