Question:
This paper redefines gradient matching in domain generalization through "arithmetic meta-learning" (AML). Compared to methods like MAML, which optimize a "fast adaptable initial point (θ)," AML focuses on optimizing the update step itself (Δθ). Which of the following statements most fundamentally explains the core philosophical and methodological assumptions behind AML's strategy?
A. AML's philosophy is that in a multi-domain setting, the gradient of any single domain is only a biased estimate of the "generalization direction." Therefore, it transforms the optimization problem from finding an exact second-order derivative (as implied by MAML) to robustly estimating the density of multiple biased estimates in gradient space, taking their mode as the update direction.
B. AML's core assumption is that there exists a unique, optimal domain-invariant representation θ*, which is geometrically the arithmetic center of all optimal solutions θ_i* in the source domain. Therefore, the functional consequence of this method is that it reduces the complex DG problem to a linear interpolation process that greedily moves toward this fixed center point at each step.
C. AML's methodology is rooted in the idea that a truly generalizable update step Δθ should itself be considered a "transferable policy," rather than a simple parameter shift. Therefore, the meta-learning goal is to maximize the expected return of this "policy" across different domain distributions (meta-training set vs. meta-test set). This is essentially learning an "update policy" that is insensitive to domain shifts.
D. AML's fundamental motivation lies in the belief that the "curvature" of the loss landscape across domains is unreliable or noisy for generalization. Therefore, it abandons the use of curvature (i.e., second-order derivatives) and assumes that simply aligning the directions of first-order gradient vectors is sufficient to find a "common descent channel" that works across different loss landscapes.

Answer: C

Although gradient matching currently serves as the basic
theory in meta-learning for domain generalization, we be-
lieve it has not been fully explored. As shown in Fig. 1, each
red arrow represents a gradient updating process. While ex-
isting methods often select the direction of g1 + g2, there
are numerous other ways in the yellow region that can also
achieve gradient matching. Alternatively, the centroid of a
structure is known for its balanced nature. In Fig. 3 (d)(h),
updating direction towards the centroid of source domain
experts, marked by the yellow color, tend to lie closer to the
target domain’s loss basin [9, 24] with better generalization
ability than those in the g1 + g2 direction. This suggests
that focusing only on gradient matching without consider-
ing balanced positioning seems insufficient.
We propose an arithmetic meta-learning framework by
adjusting the weights of g1 and g2 in Fig. 1 to identify a
more balanced position between source domains, where the
weights of gradients are selected to form an arithmetic pro-
gression. Intuitively, since g2 is less correlated with the ini-
tial model than g1, it is reasonable to assign g2 with smaller
weights. We further demonstrate that a set of arithmetically
decreasing gradient weights not only follows the principle
of gradient matching, but also reflects model averaging [24]
that indirectly estimate the centroid of domain experts. This
centroid approximation is expected to achieve a more ac-
curate balance across source domains to enhance model’s
generalization stability in the unseen target scenarios.
2.3. Balanced Positioning
We adopt the analytical technique proposed in [40]. Let
Wi represent the optimal parameter set for domain i. Our
goal is to determine Θ such that the distance D(Θ, Wi) is
small and uniform among all source domains. We specify
that each step is sampled from a single source domain i,
then the step-wise optimization objective can be regarded
as minimizing the squared distance:
argmin
θi
1
2D(θi, Wi)2.
(7)
Given a non-pathological set Z ⊂Rd, for almost all points
ϕ ∈Rd the gradient of the squared distance D(ϕ, Z)2
equals 2(ϕ −PZ(ϕ)), where PZ(ϕ) is the projection (i.e.
closest point) of ϕ onto Z. Thus, the updating process at
step i can be represented as:
θi+1 = θi −ηi∇θi
1
2D(θi, Wi)2
(8)
= θi −ηi(θi −PWi(θi))
(9)
= (1 −ηi)θi + ηiPWi(θi).
(10)
Eq. (10) illustrates that step i can be interpreted as an inter-
polation between θi and its optimal projection onto the i-th
source domain, where ηi ∈(0, 1) is not a hyperparameter,
but a measure of their relative weights. Eq. (10) can also be
iteratively expanded as:
θi+1 =
iY
j=1
(1 −ηj) · θ1 +
i
X
j=1
iY
k=j+1
(1 −ηk) · ηjΦj. (11)
We denote PWi(θi) as Φi to emphasize its optimality for
domain i, regardless of which model θi it originates from.
Eq. (11) reveals that θi+1 is a weighted average of the ini-
tial model θ1 and the optimal parameters Φ. In the second
part of Eq. (11), terms with smaller j contain more (1 −η)
that are smaller than 1, indicating a progressive decay in
the weights of earlier steps. This confirms the tendency of
models to prioritize the most recent data they encounter.
Previous methods typically interpolate between θ1 and
θn+1 during the outer loop. By applying Eq. (11), we find
that θ1 excludes Φ, while θn+1 has an unbalanced distri-
bution of {Φ1, Φ2, ..., Φn}. As a result, the interpolation
between θ1 and θn+1 also fails to correct this biased dis-
tribution. In the early stages of training, this strategy may
be acceptable if the model is far from Φ, resulting in small
values for η so that (1 −η) is close to 1, which ensures
relatively balanced weights for each Φ. However, as the
model approaches convergence, oscillations around Φ cause
(1−η) to decrease, making it challenging to maintain equal
weights for each domain-optimal parameters.
We propose to address this imbalance by utilizing the in-
termediate models {θ2, θ3, ..., θn+1} in the inner loop. Each
θi+1 is expected to specifically tailored for its correspond-
ing source domain i. While accurately computing Φ is im-
practical, we can estimate it by performing multiple gradi-
ent updates on the respective source domain to minimize all
(1 −η) towards 0. Ideally, only the last term in Eq. (11),
which exclusively involves η without (1 −η) should be re-
tained:
θi+1 ≈Φi.
(12)
By substituting Eq. (12) into Eq. (5), we derive:
Θ ←
1
n + ϵ(ϵΘ +
n
X
i=1
Φi).
(13)
This term represents the initial model Θ adjusted by the av-
erage of domain-optimal parameters, with each Φ equally
weighted. Consequently, this strategy effectively achieves a
good balance across source domains.

Balanced Direction from Multifarious Choices:
Arithmetic Meta-Learning for Domain Generalization
Xiran Wang1
Jian Zhang1
Lei Qi2
Yinghuan Shi1,3,*
1Nanjing University
2Southeast University
3Suzhou Laboratory
{zzwdx, zhangjian7369}@smail.nju.edu.cn, qilei@seu.edu.cn, syh@nju.edu.cn
Abstract
Domain generalization is proposed to address distribu-
tion shift, arising from statistical disparities between train-
ing source and unseen target domains. The widely used
first-order meta-learning algorithms demonstrate strong
performance for domain generalization by leveraging the
gradient matching theory, which aims to establish balanced
parameters across source domains to reduce overfitting to
any particular domain. However, our analysis reveals that
there are actually numerous directions to achieve gradient
matching, with current methods representing just one possi-
ble path. These methods actually overlook another critical
factor that the balanced parameters should be close to the
centroid of optimal parameters of each source domain. To
address this, we propose a simple yet effective arithmetic
meta-learning with arithmetic-weighted gradients. This ap-
proach, while adhering to the principles of gradient match-
ing, promotes a more precise balance by estimating the cen-
troid between domain-specific optimal parameters. Experi-
mental results validate the effectiveness of our strategy. Our
code is available at https://github.com/zzwdx/ARITH.
1. Introduction
Deep neural networks [28] usually hinge on the premise
that both training and test data are independent and iden-
tically distributed (i.i.d.). However, this assumption often
fails in dynamic real-world contexts, resulting in perfor-
mance degradation when test data diverges from the distri-
bution encountered during training [35]. This shift in dis-
*Xiran Wang, Jian Zhang, and Yinghuan Shi are with State Key Lab-
oratory for Novel Software Technology, Nanjing University, China. This
work was supported by National Science and Technology Major Project
(2023ZD0120700), NSFC Project (62222604, 62206052, 624B2063),
China Postdoctoral Science Foundation (2024M750424), Fundamental
Research Funds for Central Universities (020214380120, 020214380128),
State Key Laboratory Fund (ZZKT2024A14), Postdoctoral Fellowship
Program of CPSF (GZC20240252), Jiangsu Funding Program for Excel-
lent Postdoctoral Talent (2024ZB242), and Jiangsu Science and Technol-
ogy Major Project (BG2024031). Corresponding: Yinghuan Shi.
Figure 1. In the ternary parameter space (x, y, z), gradient match-
ing can be deduced in any directions of the yellow surface. The
black arrow is the updating direction of existing methods, which
moves away from the optimal solution of domain 1.
tribution proves detrimental, as the model tends to overfit
specific representations that may be absent or inadequately
represented during the test phase [25, 43, 46].
In recent years, the domain generalization (DG) [54]
paradigm has been widely studied to deal with distribution
shift, which refers to leveraging multiple source domains to
develop a model with the generalization ability that can be
directly applied to arbitrary unseen target domains.
Some studies identify invariance as a critical factor for
domain generalization, which assume that certain stable el-
ements persist across source domains and can also apply to
unseen target domains. These methods often employ tech-
niques such as data augmentation [20, 36, 63] and adver-
sarial learning [13, 34] to capture invariant features, while
others impose more intricate forms of invariance at gradient
[47, 55] or predictor [3, 29] level. However, the presumed
invariance doesn’t always translate to improved generaliza-
tion due to the random nature of target domains. As illus-
trated in [19], only a few of them surpass vanilla empirical
risk minimization (ERM) [39] in terms of average accuracy
across multiple standard datasets.
Rather than strictly enforcing an invariance factor, bal-
ance-based methods [9, 16, 45] take a more flexible way by
ensuring that model parameters are balanced across vari-
ous domains. For example, meta-learning [32, 49, 62] aims
arXiv:2503.18987v1  [cs.LG]  23 Mar 2025
to achieve an optimal balance among source domains by
implicitly guiding their gradient directions called gradient
matching. The rationale is that large angles between gra-
dients indicate conflicting objectives, suggesting that up-
dating one domain may negatively impact the optimization
process of others. In contrast, smaller gradient angles imply
that optimizing one domain does not disrupt other domains,
allowing for a mutually beneficial outcome by optimizing
their combined gradient, thereby mitigating the risk of over-
fitting to specific domains [48, 56].
Although gradient matching currently serves as the basic
theory in meta-learning for domain generalization, we be-
lieve it has not been fully explored. As shown in Fig. 1, each
red arrow represents a gradient updating process. While ex-
isting methods often select the direction of g1 + g2, there
are numerous other ways in the yellow region that can also
achieve gradient matching. Alternatively, the centroid of a
structure is known for its balanced nature. In Fig. 3 (d)(h),
updating direction towards the centroid of source domain
experts, marked by the yellow color, tend to lie closer to the
target domain’s loss basin [9, 24] with better generalization
ability than those in the g1 + g2 direction. This suggests
that focusing only on gradient matching without consider-
ing balanced positioning seems insufficient.
We propose an arithmetic meta-learning framework by
adjusting the weights of g1 and g2 in Fig. 1 to identify a
more balanced position between source domains, where the
weights of gradients are selected to form an arithmetic pro-
gression. Intuitively, since g2 is less correlated with the ini-
tial model than g1, it is reasonable to assign g2 with smaller
weights. We further demonstrate that a set of arithmetically
decreasing gradient weights not only follows the principle
of gradient matching, but also reflects model averaging [24]
that indirectly estimate the centroid of domain experts. This
centroid approximation is expected to achieve a more ac-
curate balance across source domains to enhance model’s
generalization stability in the unseen target scenarios. Our
contribution can be summarized as follows:
• We prove that existing first-order meta-learning strategies
for domain generalization represent just one of many pos-
sible directions for gradient matching, and they overlook
the need to balance the model’s position from the optimal
parameters of each source domain.
• We integrate model averaging into meta-learning and pro-
pose an arithmetic gradient-based strategy to simulate this
process, which aims to estimate the centroid of domain-
specific optimal parameters while ensuring consistency in
gradient direction. Our method is simple to implement,
requiring adjustments of one line in Algorithm 1.
• Arithmetic meta-learning outperforms traditional meta-
learning strategies across multiple domain generalization
benchmarks, and shows synergistic potential when inte-
grated with global averaging techniques.
Algorithm 1 Arithmetic Meta-Learning for DG
Input: Domains {D1, D2, ..., DS}; model parametrized by
Θ; hyperparameters k and ϵ
1: for iterations = 1, 2, ... do
2:
Initialize parameters θ1 ←Θ;
3:
for Di ∈{D1, D2, ..., DS} do
4:
Perform k gradient updates from Di to obtain
θi+1 ←θi with gi = θi −θi+1
5:
end for
6:
Θ ←Θ −ϵ Pn
i=1 gi // for Fish
7:
Θ ←Θ−
1
n+ϵ
Pn
i=1(n+1−i)gi // for Arith
8: end for
2. Method
2.1. Preliminary
Problem setting. In domain generalization, we are pro-
vided with source domains S = {D1, D2, ..., DS} and un-
seen target domains T = {DS+1, DS+2, ..., DS+T }. The
s-th domain consisting of Ns samples is represented as
Ds = {(xs
i, ys
i )}Ns
i=1, where xs
i denotes the i-th sample and
ys
i refers to its corresponding label. Our goal is to lever-
age these source domains S to develop a model capable of
seamlessly generalizing to any unseen target domain T .
First-order meta-learning [40] segments the optimiza-
tion process into an inner loop and an outer loop. Each
iteration can be summarized as follows: a task comprises
a batch of data sampled from a specific data distribution,
and a step aggregates multiple tasks to form a larger batch.
In domain generalization, tasks are commonly partitioned
by domains. MLDG [32] evenly allocates tasks into two
steps, with one step for the meta-train set and the other for
the meta-test set. Fish [48] selects one task for each step,
which is demonstrated to achieve pairwise gradient match-
ing between all domains. During the inner loop, the model
is sequentially updated with steps to reach parameters ˆΘ.
Then in the outer loop, the original parameters Θ are up-
dated towards ˆΘ in previous methods.
Organization. We present our arithmetic meta-learning
in Algorithm 1. The parallel sections Sec. 2.2 and Sec. 2.3
compare between Arith and existing methods from two per-
spectives. Sec. 2.2 demonstrates we share the same prop-
erties of gradient matching as previous work. Sec. 2.3 ex-
plains how our method achieves a more balanced position-
ing across source domains compared to prior approaches.
2.2. Gradient Matching
We first introduce our proof process of gradient matching.
During an inner loop comprising n steps, as the model’s pa-
rameters transition from Θ to ˆΘ, we depict the trajectory of
parameter updates as {θ1, θ2, ..., θn+1}, where θ1 and θn+1
correspond to Θ and ˆΘ, respectively. At each step, the loss
Figure 2. Comparison of different learning strategies. Each step of the inner loop corresponds to a distinct domain, while in the outer loop,
the gradient is computed as the weighted average of those from the inner loop, with the values above representing their respective weights.
is denoted as {L1, L2, ..., Ln}, and the corresponding gra-
dients as {g1, g2, ..., gn}, where gi = α∇Li(θi), with α
indicating the learning rate of the inner loop.
First, let’s represent the outer loop of existing methods
Θ + ϵ(ˆΘ −Θ) in gradient form:
Θ ←Θ −ϵ
n
X
i=1
gi,
(1)
thus the optimization objective can be written as:
argmin
Θ
n
X
i=1
Li(θi).
(2)
Given that each gi in Eq. (1) shares the common coefficient
ϵ, so the weights of loss Li(θi) in Eq. (2) are also the same.
Li(θi) can be approximated as the loss on the original pa-
rameters θ1 subtracting a regularization term. Please refer
to our supplementary material for more details:
Li(θi) = Li(θ1)−α
i−1
X
j=1
∇Li(θ1)·∇Lj(θ1)+O(α2). (3)
This equation aims to maximize dot product between gradi-
ent i and those from the preceding i −1 steps, promoting
smaller angles between gradients and thus ensuring consis-
tency in the updating direction across domains.
The derivation above excludes the coefficient ϵ in Eq. (1)
since the weights of losses in Eq. (2) can be arbitrary. This
implies that as long as the update procedure is expressed as
adjusting the original parameters Θ by the inner-loop gra-
dients {g1, g2, ..., gn}, gradient matching can be deduced,
allowing for an infinite range of update directions to satisfy
gradient matching. The more general form is:
Θ ←Θ −
n
X
i=1
ϵigi,
(4)
where ϵi are arbitrary coefficients. For instance, if we con-
sider averaging all intermediate models {θ2, θ3, ..., θn+1}
during the inner loop, then the new outer loop is written as:
Θ ←
1
n + ϵ(ϵθ1 +
n
X
i=1
θi+1).
(5)
Eq. (5) represents the non-gradient form of our proposed
arithmetic meta-learning. The optimization process can be
obtained by substituting θi+1 = θ1 −Pi
j=1 gj into Eq. (5),
with gradient weights following an arithmetic progression
that decreases from
n
n+ϵ to
1
n+ϵ:
Θ ←Θ −
1
n + ϵ
n
X
i=1
(n + 1 −i)gi.
(6)
This form also adheres to Eq. (4), illustrating that Arith can
also achieve gradient matching as previous methods.
2.3. Balanced Positioning
We adopt the analytical technique proposed in [40]. Let
Wi represent the optimal parameter set for domain i. Our
goal is to determine Θ such that the distance D(Θ, Wi) is
small and uniform among all source domains. We specify
that each step is sampled from a single source domain i,
then the step-wise optimization objective can be regarded
as minimizing the squared distance:
argmin
θi
1
2D(θi, Wi)2.
(7)
Given a non-pathological set Z ⊂Rd, for almost all points
ϕ ∈Rd the gradient of the squared distance D(ϕ, Z)2
equals 2(ϕ −PZ(ϕ)), where PZ(ϕ) is the projection (i.e.
closest point) of ϕ onto Z. Thus, the updating process at
step i can be represented as:
θi+1 = θi −ηi∇θi
1
2D(θi, Wi)2
(8)
= θi −ηi(θi −PWi(θi))
(9)
= (1 −ηi)θi + ηiPWi(θi).
(10)
Eq. (10) illustrates that step i can be interpreted as an inter-
polation between θi and its optimal projection onto the i-th
source domain, where ηi ∈(0, 1) is not a hyperparameter,
but a measure of their relative weights. Eq. (10) can also be
iteratively expanded as:
θi+1 =
iY
j=1
(1 −ηj) · θ1 +
i
X
j=1
iY
k=j+1
(1 −ηk) · ηjΦj. (11)
Figure 3. Loss surface plots of various domains on the PACS dataset, where the deeper is better. The yellow triangle in (a)(b)(c)(e)(f)(g)
shows the estimated optimal parameters from the respective source domain, while the black triangle represents the estimated optimal
parameters for the other source domains. The red arrow in (d)(h) is the updating direction of previous methods, while the yellow arrow
towards the centroid marks the update direction of arithmetic meta-learning.
Figure 4. Adam optimizer’s gradient distribution over the first fif-
teen steps of inner loop. Three domains are alternately optimized
at each step. As momentum builds, the gradient contributions from
each domain converge to similar proportions in the later stages.
We denote PWi(θi) as Φi to emphasize its optimality for
domain i, regardless of which model θi it originates from.
Eq. (11) reveals that θi+1 is a weighted average of the ini-
tial model θ1 and the optimal parameters Φ. In the second
part of Eq. (11), terms with smaller j contain more (1 −η)
that are smaller than 1, indicating a progressive decay in
the weights of earlier steps. This confirms the tendency of
models to prioritize the most recent data they encounter.
Previous methods typically interpolate between θ1 and
θn+1 during the outer loop. By applying Eq. (11), we find
that θ1 excludes Φ, while θn+1 has an unbalanced distri-
bution of {Φ1, Φ2, ..., Φn}. As a result, the interpolation
between θ1 and θn+1 also fails to correct this biased dis-
tribution. In the early stages of training, this strategy may
be acceptable if the model is far from Φ, resulting in small
values for η so that (1 −η) is close to 1, which ensures
relatively balanced weights for each Φ. However, as the
model approaches convergence, oscillations around Φ cause
(1−η) to decrease, making it challenging to maintain equal
weights for each domain-optimal parameters.
We propose to address this imbalance by utilizing the in-
termediate models {θ2, θ3, ..., θn+1} in the inner loop. Each
θi+1 is expected to specifically tailored for its correspond-
ing source domain i. While accurately computing Φ is im-
practical, we can estimate it by performing multiple gradi-
ent updates on the respective source domain to minimize all
(1 −η) towards 0. Ideally, only the last term in Eq. (11),
which exclusively involves η without (1 −η) should be re-
tained:
θi+1 ≈Φi.
(12)
By substituting Eq. (12) into Eq. (5), we derive:
Θ ←
1
n + ϵ(ϵΘ +
n
X
i=1
Φi).
(13)
This term represents the initial model Θ adjusted by the av-
erage of domain-optimal parameters, with each Φ equally
weighted. Consequently, this strategy effectively achieves a
good balance across source domains.
2.4. Optimizer Selection
Previous research commonly (i) applies the Adam optimizer
in the inner loop and (ii) performs direct interpolation in the
outer loop. However, this method conflicts with the prin-
ciple of domain-wise gradient matching, as the Adam opti-
Table 1. Accuracy (%) on the DomainBed benchmark across five datasets. The best and second-best results per dataset are bolded and
underlined respectively. Detailed results for each domain are available in the supplementary material.
Method
PACS
VLCS
OfficeH
TerraInc
DomainNet
Avg
ERM [39]
85.5
77.5
66.5
46.1
40.9
63.3
IRM [3]
83.5
78.6
64.3
47.6
33.9
61.6
CausIRL [11]
83.6
76.5
68.1
47.4
41.8
63.5
VREx [29]
84.9
78.3
66.4
46.4
33.6
61.9
CORAL [51]
86.2
78.8
68.7
47.7
41.5
64.5
RSC [23]
85.2
77.1
65.5
46.6
38.9
62.7
ARM [62]
85.1
77.6
64.8
45.5
35.5
61.7
AND-mask [41]
84.4
78.1
65.6
44.6
37.2
62.0
SAND-mask [47]
84.6
77.4
65.8
42.9
32.1
60.6
MLDG [32]
84.9
77.2
66.8
47.8
41.2
63.6
Fish [48]
85.5
77.8
68.6
45.1
42.7
63.9
Fishr [44]
85.5
77.8
67.8
47.4
41.7
64.0
HGP [22]
84.7
77.6
68.2
43.6
41.1
63.0
Hutchinson [22]
83.9
76.8
68.2
46.6
41.6
63.4
Arith
86.5 ± 0.3
79.4 ± 0.3
69.4 ± 0.1
48.1 ± 1.2
41.5 ± 0.1
65.0
mizer contains momentum:
¯gi = β ¯
gi−1 + (1 −β)gi,
(14)
where β controls the weighting between momentum
¯
gi−1
and the current gradient gi, with a default value of 0.9. Con-
sequently, each gi in Eq. (1) is actually ¯gi in Eq. (14), which
becomes a blend of gradients from all domains. In Fig. 4,
we visualize the gradients of Adam during alternating op-
timization across three domains for the first fifteen steps,
illustrating how this gradient matching resembles uniform
sampling without domain-specific gradients.
To prevent the failure of domain-wise gradient matching,
we propose (i) using the Adam optimizer only in the outer
loop and (ii) employing stochastic gradient descent without
momentum in the inner loop to ensure each step accurately
reflects the true gradient of its corresponding domain.
2.5. Relationships with Weight Averaging
Previous meta-learning methods for domain generalization
closely resembles weight averaging [9, 24], as the interpo-
lation between two models can be treated as the weighted
average of them. For arithmetic meta-learning, we can also
equivalently substitute it with the average of all intermediate
models during the inner loop. However, arithmetic meta-
learning differs from averaging-based methods in several
key aspects: (i) Varied objectives: Model averaging aims
to find flat minima across domain-agnostic models, ensur-
ing robustness against shifts in the loss landscape between
training and test sets. In contrast, arithmetic meta-learning
prevents models from becoming overly biased towards spe-
cific domains by averaging between domain-specific mod-
els. (ii) Different scope: Model averaging operates on a
global scale, encompassing most of the models throughout
the entire training process, while arithmetic meta-learning
is more localized, averaging only a small subset of mod-
els during each iteration. (iii) Unique implementation: By
transforming weight averaging into gradient form, arith-
metic meta-learning enables the optimization of outer loop
to easily adapt to optimizers such as Adam.
Given that our strategy is based on the first-order meta-
learning framework, it faces common challenges associated
with such methods, including difficulties in converging to
flat minima, and the balance between source domains may
not translate to optimal parameters of the unseen target do-
main. However, the orthogonal nature of arithmetic meta-
learning and model averaging at local and global scales en-
ables effective synergy between these approaches.. Experi-
mental results in Sec. 3 further demonstrate that the combi-
nation of them leads to enhanced performance.
3. Experiment
3.1. Datasets
We experiment on ten standard DG datasets, five of which
are from the DomainBed [19] benchmark: (i) PACS [31]
contains 4 domains (photo, art-painting, cartoon, sketch)
with 7 classes and 9,991 images. (ii) Office-Home [53]
comprises 4 domains (art, clipart, product, real-world) with
65 classes and 15,588 images. (iii) VLCS [17] consists of 4
domains (pascal, labelme, caltech, sun) with 5 classes and
10,729 images. (iv) TerraIncognita [6] is composed of 4
domains (location38, location43, location46, location100)
with 100 classes and 24,788 images. (v) DomainNet [42]
includes 6 domains (clipart, infograph, painting, quick-
Table 2. Ablation studies (%) on PACS dataset. The best results
are bolded. The m denotes whether to enable momentum during
the inner loop, while ds signifies whether the data is partitioned
by domains for each step or uniformly sampled.
Method
m
ds
A
C
P
S
Avg
MAML
-
-
84.6
80.8
96.7
79.3
85.3
✓
✓
84.8
81.9
96.0
77.5
85.0
-
✓
85.5
78.5
97.2
76.4
84.4
Fish
-
-
85.0
81.1
95.2
80.0
85.3
✓
✓
85.6
82.0
95.7
78.5
85.4
-
✓
85.7
81.1
96.7
81.0
86.1
Arith
-
-
85.6
80.7
96.3
80.9
85.9
✓
✓
85.1
82.2
96.6
78.9
85.7
-
✓
85.9
81.3
97.1
81.8
86.5
draw, real, sketch) with 345 classes and 586,575 images.
The other five datasets AMAZON, CAMELYON17 [5],
CIVILCOMMENTS [8], IWILDCAM [7], and FMOW
[12] are conducted on the WILDS [27] benchmark which
contains multiple modalities. Due to space limitations, we
present results only on CAMELYON17 in the main text and
please refer to the supplementary material for others.
3.2. Implementation Details
We follow the protocol proposed in DomainBed [19] and
use ResNet50 [21] pretrained on ImageNet [14] as our back-
bone network. We apply stochastic gradient descent in the
inner loop and the Adam optimizer in the outer loop, with
each of the domains restricted to a single step over 5000 it-
erations. The domains are optimized in random order, with
a batch size of 32 and a learning rate of 5e −5. For datasets
other than DomainNet, which consists of three domains, the
weight of gradients are assigned as {1/2, 1/3, 1/6}. For Do-
mainNet, which comprises five domains, the correspond-
ing weights are set as {1/3, 4/15, 1/5, 2/15, 1/15}. We select
one target domain for test and use the remaining domains
for training and validation. We reserve 20% of the sam-
ples for validation from each source domain and choose the
model with maximized accuracy on the overall validation
set, which is the same as the training-domain validation set
in [19]. Each experiment is conducted on a single Nvidia
RTX 2080Ti GPU with Pytorch 1.10.1.
3.3. Main Results
As illustrated in Tab. 1, we first compare arithmetic meta-
learning with classic domain generalization methods on the
DomainBed [19] benchmark, where the strategies below
RSC [23] are gradient-based.
For each dataset, we per-
form experiments three times and report the average results
followed by the standard deviation. Our method demon-
strates superior performance on four datasets, surpassing
Table 3. Ablation studies (%) on OfficeHome and VLCS dataset.
The best results are bolded.
The scaled means increasing the
learning rate to 1.5 times its original value, while adam refers to
utilizing the Adam optimizer during the outer loop.
Method
scaled
adam
D1
D2
D3
D4
Avg
OfficeHome
Fish
✓
-
61.5
55.3
74.6
77.2
67.2
✓
✓
65.0
54.8
77.0
78.8
68.9
-
-
60.7
53.6
75.7
77.3
66.8
-
✓
64.3
55.3
77.2
79.0
69.0
Arith
✓
-
62.5
54.7
76.4
77.2
67.7
✓
✓
64.8
55.8
76.1
79.5
69.1
-
-
61.6
55.2
75.6
77.1
67.4
-
✓
64.7
56.3
77.5
79.2
69.4
VLCS
Fish
✓
-
97.1
63.7
72.3
77.9
77.8
✓
✓
98.6
64.3
76.7
76.4
79.0
-
-
97.5
63.8
73.6
78.1
78.3
-
✓
98.7
65.0
76.5
76.0
79.2
Arith
✓
-
97.3
64.8
73.2
77.8
78.3
✓
✓
98.7
64.1
76.3
78.2
79.3
-
-
98.5
64.7
76.0
77.3
79.1
-
✓
98.7
64.6
76.3
77.8
79.4
the second-best method by 0.3%, 0.6%, 0.7% and 0.3%
respectively. We attribute our improvement to two key as-
pects. First, our approach can estimate the centroid among
domain experts, which still offers opportunities for further
enhancement in this table. Since our method requires pre-
cise estimation of optimal parameters for each source do-
main, updating parameters only once may introduce bias.
To address this, we conduct multiple-step per domain ex-
periments in the following ablation studies. Second, we re-
fine the usage of optimizers from previous methods. By re-
moving momentum during the inner loop, we achieve more
precise gradient matching across domains, while using the
Adam optimizer in the outer loop helps ensure smoother
convergence for the final model.
3.4. Ablation & Analysis
Loss surface visualization. We visualize the loss surface
using the technique proposed in [24]. We select three inter-
mediate models from the inner loop and compute their lin-
early interpolated losses, where each model is sequentially
derived from 30 gradient updates on a single source domain.
We calculate the averaged parameters of these models (i.e.,
yellow stars), which represent the optimization direction of
arithmetic meta-learning in the outer loop. As shown in
Fig. 3, while the loss basins differ between domains, the
variations are not substantial. The relative positions of each
source domain’s loss basins closely align with those of their
corresponding intermediate models (i.e., yellow triangles),
Table 4. Ablation study (%) with global averaging of five datasets
on the DomainBed benchmark. The best results are bolded.
Method
swad
P
V
O
T
D
Avg
ERM
-
84.7
78.0
67.8
46.5
40.8
63.6
✓
87.5
78.1
70.4
49.4
44.1
65.9
MLDG
-
85.4
77.8
68.5
47.3
41.4
64.1
✓
88.3
78.9
69.8
50.5
44.6
66.4
Fish
-
86.1
78.4
68.3
47.0
41.5
64.3
✓
88.5
79.4
70.1
49.5
44.9
66.5
Arith
-
86.5
79.4
69.4
48.1
41.5
65.0
✓
88.7
79.8
70.2
49.9
44.9
66.7
suggesting their capacity to approximate domain-optimal
parameters, thus the averaged parameters can be regarded as
a good balance across source domains. However, although
the final averaged model happens to match with the basin of
the target domain in Fig. 3, it’s essential to acknowledge that
since the target domain is unseen and arbitrary, our ability
is limited to preventing overfitting to source domains, but
cannot guarantee optimal performance on the target domain
in every scenario.
Ablation study on steps per domain k. We investigate
how the performance correlates with the step count for each
domain. As illustrated in Fig. 5, the optimization process
of Fish [48] signify linear interpolations between the initial
and final parameters, while the arithmetic gradient weights
simulates the updating direction towards the average of all
intermediate models. As the number of steps increases, the
accuracy of arithmetic meta-learning consistently improves.
This indicates that performing multiple gradient updates re-
sults in a more precise estimation of optimal parameters for
each domain, achieving a better balance through averaging.
In contrast, the performance of Fish initially improves but
then declines due to its biased estimation of the optimal bal-
ance. As the displacement of parameters increases, this in-
accuracy may become more serious.
Ablation study on gradient weights / optimizer selec-
tion. The weights of gradients in the inner loop have two
key attributes: ratio and magnitude. The ratio determines
the final update direction and is fixed for each strategy. For
instance, a 1 : 1 : 1 ratio for Fish means updating in the
direction of the last model, while a 3 : 2 : 1 ratio for Arith
means shifting towards the average of domain experts. The
magnitude affects the step size of the outer loop, similar to
the learning rate. To match with inner loop, we set the to-
tal sum of weights around 1, such as {1/3, 1/3, 1/3} for Fish
and {1/2, 1/3, 1/6} for Arith. In DomainBed, Fish is imple-
mented with weights of {1/3, 1/3, 1/3}, so we additionally set
Arith’s weights to {3/4, 1/2, 1/4} when scaling is enabled,
ensuring that the sum of the weights is equal. As shown
in Tab. 3, Arith generally outperforms Fish in most scenar-
Figure 5. Accuracy (%) on PACS dataset with the varying number
of steps for each domain during the inner loop .
ios and achieves higher average accuracy. Unlike the inner
loop, the outer loop optimizer permits momentum. When
using the Adam optimizer, the impact of gradient weights
on model performance is minimal due to Adam’s adaptive
learning rate, which leads to better outcomes than models
that do not utilize this condition.
Ablation study on domain-specific steps / gradients.
Gradient-based meta-learning methods for domain general-
ization typically select samples from each domain to per-
form domain-specific gradient matching. We also conduct
experiments with uniform sampling and inner loop momen-
tum. As shown in Tab. 2, domain-specific sampling strate-
gies without momentum generally outperform other imple-
mentations, while the performance of uniform sampling and
momentum is comparable. It is noted in Sec. 2.4 that uni-
form sampling and inner loop momentum results in mixed
gradients, which fail to effectively match across source do-
mains.
In arithmetic meta-learning, parameters obtained
through domain-specific gradient matching are more likely
to estimate globally optimal parameters rather than being
tailored to individual ones, thus mitigating the risk of over-
fitting towards specific domains.
Combining with global averaging. Arithmetic meta-
learning simulates localized model averaging during each
iteration, aiming to find a balanced position among source
domains instead of pursuing a global flat minima. To further
enhance this approach, we propose to integrate it with the
global averaging method [9], which facilitates the achieve-
ment of well-generalized flat minima. As shown in Tab. 4,
our method yields additional improvements over this strong
baseline, underscoring the complementary strengths of lo-
cal and global averaging techniques. It is important to note
that some of the results differ from those in Tab. 1, as they
are based on our own implementation.
4. Related Work
4.1. Domain Generalization
One key aspect of domain generalization is to learn features
invariant across source domains, assuming they will gener-
Table 5. Accuracy (%) on CAMELYON17 dataset. The best results are bolded.
Method
20
21
22
23
24
25
26
27
28
29
Avg
ERM
49.2
30.2
73.6
74.8
64.4
60.8
57.0
37.8
89.6
77.3
73.1
Fish
52.4
36.0
72.3
77.5
69.0
65.1
59.3
43.6
90.0
77.6
74.8
Arith
54.4
33.8
83.6
75.2
72.5
69.5
64.0
40.7
90.1
79.9
76.6
alize to unseen target domains as well. Approaches such
as domain adversarial learning [13, 50] aim to train a fea-
ture extractor insensitive to domain-specific traits. Invariant
risk minimization [1, 3] enforces consistency of the opti-
mal classifier across all domains within the representation
space. Feature decoupling methods aim to [10, 38] distill
domain-invariant semantic information from original fea-
tures. To diversify training data and improve generalized
feature representations, data augmentation methods manip-
ulate statistical characteristics at either the image or fea-
ture level. Commonly employed techniques include mix-
ing [57, 63] and Fourier transformations [20, 59]. Rather
than strictly enforcing an invariance factor, balance-based
methods typically propose model-agnostic strategies to re-
duce overfitting to specific source domains. Meta-learning
[16, 61] aims to leverage prior knowledge to guide the learn-
ing of current tasks. Ensemble learning [37, 64] presumes
the target domain distribution as a blend of source domain
distributions, thereby incorporating statistics from domain
expert models. Weight averaging [4, 9] aggregates model
weights across training episodes to achieve flatter minima
that are less prone to overfitting.
4.2. Meta-Learning
Meta-learning [2, 52] is a well-established field aimed at en-
abling models to generalize across a diverse range of tasks.
Initially, meta-learning focused on discovering initial pa-
rameters that can adapt swiftly to new tasks within a few
gradient steps. For instance, model-agnostic meta-learning
(MAML) [18] and first-order meta-learning (Reptile) [40]
split the optimization process into inner and outer loops.
The inner loop handles task-specific adaptation, while the
outer loop seeks a globally optimal initialization. More re-
cently, meta-learning has been applied to domain general-
ization (MLDG) [32] to simulate domain shifts by synthe-
sizing meta-train and meta-test domains. S-MLDG [33] en-
hances MLDG with sequential and lifelong learning. To
address the computational cost of second-order derivatives,
Fish [48] use first-order algorithms for efficient and fine-
grained task sampling. Traditional meta-learning empha-
sizes intra-task gradient matching, whereas meta-learning
for domain generalization focuses on inter-domain gradi-
ent matching. Despite differing objectives, both approaches
have demonstrated strong performance in their respective
fields. We believe meta-learning for domain generalization
can inform traditional meta-learning, given the conceptual
parallel between domains and tasks. Supporting research
[30] also indicates that inter-task gradient matching con-
tributes to effective initialization for general tasks.
4.3. Weight Averaging
Weight averaging [24, 58] is a variant of ensemble learn-
ing [15, 60], in which the outputs of individual models are
combined to enhance overall performance. However, tradi-
tional ensemble methods incur high computational and stor-
age costs due to the need for training and inference with
multiple models. Weight averaging addresses this issue by
merging these models into a single entity. By averaging pa-
rameters from similar training trajectories, {θ1, θ2, ..., θn},
previous work has demonstrated that it effectively approxi-
mates the ensemble output [4, 24]:
f( 1
n
n
X
i=1
θi) ≈1
n
n
X
i=1
f(θi).
(15)
SWA [24] uses a specialized learning rate schedule and pe-
riodically aggregates model weights. SWAD [9] imposes
constraints on the sampling range and increases the sam-
pling frequency to address overfitting in domain generaliza-
tion tasks. Some methods focus on model selection [45, 58]
by using a greedy algorithm to iteratively select the most ef-
fective model during the averaging process. Methods incor-
porating multiple updating trajectories are also employed in
weight averaging [4, 26], such as aligning the directions or
leveraging the combined model outputs from different tra-
jectories to achieve improved performance.
5. Conclusion
In this paper, we propose a simple yet effective enhance-
ment to first-order meta-learning for domain generalization.
While previous methods rely on constant gradient weights
in the inner loop to ensure gradient matching across source
domains, they overlook the crucial positioning of the model
relative to the optimal parameters of each domain. We in-
troduce an arithmetic gradient-based meta-learning strategy
that approximates the direction toward the average of all
models within the inner loop. While preserving the prin-
ciple of gradient matching, our approach guides the model
towards the centroid of the each domain-optimal parame-
ters, achieving a more precise balance. Experimental results
demonstrate the superior performance of our method.
References
[1] Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-
Christophe
Gagnon-Audet,
Yoshua
Bengio,
Ioannis
Mitliagkas, and Irina Rish. Invariance principle meets in-
formation bottleneck for out-of-distribution generalization.
Advances in Neural Information Processing Systems, 34:
3438–3450, 2021. 8
[2] Maruan Al-Shedivat, Liam Li, Eric Xing, and Ameet Tal-
walkar. On data efficiency of meta-learning. In Interna-
tional Conference on Artificial Intelligence and Statistics,
pages 1369–1377. PMLR, 2021. 8
[3] Martin Arjovsky, L´eon Bottou, Ishaan Gulrajani, and David
Lopez-Paz.
Invariant risk minimization.
Stat, 1050:27,
2020. 1, 5, 8
[4] Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming
Xiong. Ensemble of averages: Improving model selection
and boosting performance in domain generalization.
Ad-
vances in Neural Information Processing Systems, 35:8265–
8277, 2022. 8
[5] Peter Bandi,
Oscar Geessink,
Quirine Manson,
Mar-
cory Van Dijk, Maschenka Balkenhol, Meyke Hermsen,
Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun
Paeng, Aoxiao Zhong, et al. From detection of individual
metastases to classification of lymph node status at the pa-
tient level: the camelyon17 challenge. IEEE Transactions
on Medical Imaging, 2018. 6
[6] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition
in terra incognita. In Proceedings of the European Confer-
ence on Computer Vision (ECCV), pages 456–473, 2018. 5
[7] Sara Beery, Elijah Cole, and Arvi Gjoka.
The iwildcam
2020 competition dataset. arXiv preprint arXiv:2004.10340,
2020. 6
[8] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum
Thain, and Lucy Vasserman. Nuanced metrics for measur-
ing unintended bias with real data for text classification. In
Companion Proceedings of The 2019 World Wide Web Con-
ference, 2019. 6
[9] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol
Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park.
Swad:
Domain generalization by seeking flat minima.
Advances in Neural Information Processing Systems, 34:
22405–22418, 2021. 1, 2, 5, 7, 8
[10] Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman.
Learning to balance specificity and invariance for in and out
of domain generalization. In European Conference on Com-
puter Vision, pages 301–318, 2020. 8
[11] Mathieu Chevalley, Charlotte Bunne, Andreas Krause, and
Stefan Bauer. Invariant causal mechanisms through distri-
bution matching. arXiv preprint arXiv:2206.11646, 2022.
5
[12] Gordon Christie, Neil Fendley, James Wilson, and Ryan
Mukherjee. Functional map of the world. In Proceedings
of the IEEE Conference on Computer Vision and Pattern
Recognition, 2018. 6
[13] Aveen Dayal, Vimal KB, Linga Reddy Cenkeramaddi, C
Mohan, Abhinav Kumar, and Vineeth N Balasubramanian.
Madg: Margin-based adversarial learning for domain gener-
alization. Advances in Neural Information Processing Sys-
tems, 36, 2024. 1, 8
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
and Li Fei-Fei. Imagenet: A large-scale hierarchical image
database. In 2009 IEEE Conference on Computer Vision and
Pattern Recognition, pages 248–255, 2009. 6
[15] Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and
Qianli Ma. A survey on ensemble learning. Frontiers of
Computer Science, 14:241–258, 2020. 8
[16] Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas,
and Ben Glocker. Domain generalization via model-agnostic
learning of semantic features. Advances in Neural Informa-
tion Processing Systems, 32, 2019. 1, 8
[17] Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased met-
ric learning: On the utilization of multiple datasets and web
images for softening bias. In Proceedings of the IEEE Inter-
national Conference on Computer Vision, pages 1657–1664,
2013. 5
[18] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-
agnostic meta-learning for fast adaptation of deep networks.
In International Conference on Machine Learning, pages
1126–1135. PMLR, 2017. 8
[19] Ishaan Gulrajani and David Lopez-Paz. In search of lost
domain generalization.
arXiv preprint arXiv:2007.01434,
2020. 1, 5, 6
[20] Jintao Guo, Na Wang, Lei Qi, and Yinghuan Shi. Aloft:
A lightweight mlp-like architecture with dynamic low-
frequency transform for domain generalization. In Proceed-
ings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 24132–24141, 2023. 1, 8
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proceed-
ings of the IEEE Conference on Computer Vision and Pat-
tern Recognition, pages 770–778, 2016. 6
[22] Sobhan Hemati, Guojun Zhang, Amir Estiri, and Xi Chen.
Understanding hessian alignment for domain generalization.
In Proceedings of the IEEE/CVF International Conference
on Computer Vision, pages 19004–19014, 2023. 5
[23] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang.
Self-challenging improves cross-domain generalization. In
European Conference on Computer Vision, pages 124–140,
2020. 5, 6
[24] Pavel Izmailov,
Dmitrii Podoprikhin,
Timur Garipov,
Dmitry Vetrov, and Andrew Gordon Wilson.
Averaging
weights leads to wider optima and better generalization.
arXiv preprint arXiv:1803.05407, 2018. 2, 5, 6, 8
[25] Pavel Izmailov, Polina Kirichenko, Nate Gruver, and An-
drew G Wilson. On feature learning in the presence of spu-
rious correlations. Advances in Neural Information Process-
ing Systems, 35:38516–38532, 2022. 1
[26] Alexia Jolicoeur-Martineau, Emy Gervais, Kilian Fatras,
Yan Zhang, and Simon Lacoste-Julien. Population param-
eter averaging (papa).
arXiv preprint arXiv:2304.03094,
2023. 8
[27] Pang
Wei
Koh,
Shiori
Sagawa,
Henrik
Marklund,
Sang Michael Xie, Marvin Zhang, Akshay Balsubramani,
Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips,
Irena Gao, et al. Wilds: A benchmark of in-the-wild dis-
tribution shifts.
In International conference on Machine
Learning, pages 5637–5664. PMLR, 2021. 6
[28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural net-
works. Communications of the ACM, 60(6):84–90, 2017. 1
[29] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen,
Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi
Le Priol, and Aaron Courville. Out-of-distribution general-
ization via risk extrapolation (rex). In International Confer-
ence on Machine Learning, pages 5815–5826. PMLR, 2021.
1, 5
[30] Seanie Lee, Hae Beom Lee, Juho Lee, and Sung Ju Hwang.
Sequential reptile: Inter-task gradient alignment for multi-
lingual learning. In International Conference on Learning
Representations, 2021. 8
[31] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M
Hospedales. Deeper, broader and artier domain generaliza-
tion. In Proceedings of the IEEE International Conference
on Computer Vision, pages 5542–5550, 2017. 5
[32] Da Li,
Yongxin Yang,
Yi-Zhe Song,
and Timothy
Hospedales. Learning to generalize: Meta-learning for do-
main generalization. In Proceedings of the AAAI Conference
on Artificial Intelligence, 2018. 1, 2, 5, 8
[33] Da Li,
Yongxin Yang,
Yi-Zhe Song,
and Timothy
Hospedales. Sequential learning for domain generalization.
In European Conference on Computer Vision, pages 603–
619. Springer, 2020. 8
[34] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot.
Domain generalization with adversarial feature learning. In
Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pages 5400–5409, 2018. 1
[35] Haidong Li, Jiongcheng Li, Xiaoming Guan, Binghao
Liang, Yuting Lai, and Xinglong Luo. Research on over-
fitting of deep learning. In 2019 15th International Con-
ference on Computational Intelligence and Security, pages
78–81, 2019. 1
[36] Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and
Timothy M Hospedales. A simple feature augmentation for
domain generalization. In Proceedings of the IEEE/CVF In-
ternational Conference on Computer Vision, pages 8886–
8895, 2021. 1
[37] Ziyue Li, Kan Ren, Xinyang Jiang, Bo Li, Haipeng
Zhang, and Dongsheng Li.
Domain generalization us-
ing pretrained models without fine-tuning. arXiv preprint
arXiv:2203.04600, 2022. 8
[38] Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold
Liu, Ziteng Wang, and Di Liu. Causality inspired represen-
tation learning for domain generalization. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 8046–8056, 2022. 8
[39] Vapnik Vladimir Naumovich and Vapnik Vlamimir. Statis-
tical learning theory, 1998. 1, 5
[40] Alex Nichol, Joshua Achiam, and John Schulman.
On
first-order meta-learning algorithms.
arXiv preprint
arXiv:1803.02999, 2018. 2, 3, 8
[41] Giambattista Parascandolo, Alexander Neitz, Antonio Orvi-
eto, Luigi Gresele, and Bernhard Sch¨olkopf. Learning ex-
planations that are hard to vary. In International Conference
on Learning Representations, 2020. 5
[42] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate
Saenko, and Bo Wang. Moment matching for multi-source
domain adaptation. In Proceedings of the IEEE/CVF Inter-
national Conference on Computer Vision, pages 1406–1415,
2019. 5
[43] Muyang Qiu, Jian Zhang, Lei Qi, Qian Yu, Yinghuan Shi,
and Yang Gao.
The devil is in the statistics: Mitigating
and exploiting statistics difference for generalizable semi-
supervised medical image segmentation. In European Con-
ference on Computer Vision, pages 74–91. Springer, 2024.
1
[44] Alexandre Rame, Corentin Dancette, and Matthieu Cord.
Fishr: Invariant gradient variances for out-of-distribution
generalization.
In International Conference on Machine
Learning, pages 18347–18377. PMLR, 2022. 5
[45] Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier,
Alain Rakotomamonjy, Patrick Gallinari, and Matthieu
Cord. Diverse weight averaging for out-of-distribution gen-
eralization. Advances in Neural Information Processing Sys-
tems, 35:10821–10836, 2022. 1, 8
[46] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek
Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias
in neural networks. Advances in Neural Information Pro-
cessing Systems, 33:9573–9585, 2020. 1
[47] Soroosh Shahtalebi, Jean-Christophe Gagnon-Audet, Touraj
Laleh, Mojtaba Faramarzi, Kartik Ahuja, and Irina Rish.
Sand-mask: An enhanced gradient masking strategy for the
discovery of invariances in domain generalization.
arXiv
preprint arXiv:2106.02266, 2021. 1, 5
[48] Yuge Shi, Jeffrey Seely, Philip Torr, N Siddharth, Awni
Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient
matching for domain generalization. In International Con-
ference on Learning Representations, 2021. 2, 5, 7, 8
[49] Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang,
and Mingsheng Long.
Open domain generalization with
domain-augmented meta-learning.
In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 9624–9633, 2021. 1
[50] Anthony Sicilia, Xingchen Zhao, and Seong Jae Hwang.
Domain adversarial neural networks for domain generaliza-
tion: When it works and how to improve. Machine Learning,
pages 1–37, 2023. 8
[51] Baochen Sun and Kate Saenko.
Deep coral: Correlation
alignment for deep domain adaptation. In European Confer-
ence on Computer Vision, pages 443–450, 2016. 5
[52] Sebastian Thrun and Lorien Pratt.
Learning to learn.
Springer Science & Business Media, 2012. 8
[53] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty,
and Sethuraman Panchanathan. Deep hashing network for
unsupervised domain adaptation.
In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recog-
nition, pages 5018–5027, 2017. 5
[54] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang,
Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip
Yu. Generalizing to unseen domains: A survey on domain
generalization. IEEE Transactions on Knowledge and Data
Engineering, 2022. 1
[55] Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang.
Sharpness-aware gradient matching for domain generaliza-
tion. In Proceedings of the IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition, pages 3769–3778,
2023. 1
[56] Xiran Wang, Jian Zhang, Lei Qi, and Yinghuan Shi. Gen-
eralizable decision boundaries: Dualistic meta-learning for
open set domain generalization.
In Proceedings of the
IEEE/CVF International Conference on Computer Vision,
pages 11564–11573, 2023. 2
[57] Yufei Wang, Haoliang Li, and Alex C Kot. Heterogeneous
domain generalization via domain mixup. In IEEE Interna-
tional Conference on Acoustics, Speech and Signal Process-
ing, pages 3622–3626, 2020. 8
[58] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Re-
becca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos,
Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon
Kornblith, et al. Model soups: averaging weights of mul-
tiple fine-tuned models improves accuracy without increas-
ing inference time. In International Conference on Machine
Learning, pages 23965–23998. PMLR, 2022. 8
[59] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and
Qi Tian. A fourier-based framework for domain generaliza-
tion. In Proceedings of the IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition, pages 14383–14392,
2021. 8
[60] Yongquan Yang, Haijun Lv, and Ning Chen. A survey on
ensemble learning under the era of deep learning. Artificial
Intelligence Review, 56(6):5545–5589, 2023. 8
[61] Jian Zhang, Lei Qi, Yinghuan Shi, and Yang Gao. Mvdg: A
unified multi-view framework for domain generalization. In
European Conference on Computer Vision, pages 161–177,
2022. 8
[62] Marvin Zhang, Henrik Marklund, Abhishek Gupta, Sergey
Levine, and Chelsea Finn.
Adaptive risk minimization:
A meta-learning approach for tackling group shift. arXiv
preprint arXiv:2007.02931, 8(9):4, 2020. 1, 5
[63] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Do-
main generalization with mixstyle. In International Confer-
ence on Learning Representations, 2020. 1, 8
[64] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Do-
main adaptive ensemble learning. IEEE Transactions on Im-
age Processing, 30:8008–8018, 2021. 8