Question:
Attrributed triple is an ideal knowledge representation form because:
A. Attributed triple is flexible and can be applied widely in many scenarios.
B. Attributed triple can mitigate ambiguity in structured knowledge representation for its enhancement in binary relationships.
C. Attributed triples can be obtained without human assistance with designed extraction model
D. Attributed triples can be integrated into a graph database easily.

Answer: AB

Current studies on complex knowledge extraction lie primarily within the biomedical domain, which involves distilling an overarching relation among specific categories of entities [4–7]. These methods necessitate the predefinition of entity types prior to extraction, as depicted in Fig. 1. However, their application in other industrial fields, such as Information and Communication Technology (ICT), can lead to two major challenges. 1) Beyond the overall relations among entities, individual relations between entities also matter. For instance, UE device sends Service Request message to AN device, and the message contains Service type, where UE and AN are undefined entities in ICT. Here, the relation between UE device and Service Request message is defined as send, while the relation between Service Request message and Service type is contain and the collective relation of the entities (UE device, Service Request message, AN device) is (send. . . to. . . ). Representing these entities using a single overarching relation can compromise information integrity. 2) The diversity and continual emergence of new entity types in textual data make the predefinition impractical.

Intuitively, the ideal representation would involve a more granular form of complex knowledge, capable of articulating both the pairwise and overall relations among multiple entities. Consequently, we distill complex text into primary components and modifiers, achieving a fine-grained representation of complex knowledge. Moreover, we observe that the range of relations in most extraction datasets is considerably narrower than the variety of entity types. An intuitive solution is to identify all relevant entity types within the text by recognizing relations, thus circumventing the need of entity-type predefinition.

In this paper, we introduce a novel representational form for complex knowledge: attributed triple, comprising a primary triple allied with auxiliary attribute-value pairs. The primary triple delineates the central subject-predicate-object structure of the text, while the auxiliary pairs provide additional contextual details. The instance mentioned above can be represented as (UE device, send:{to, AN device}, Service Request message:{contain, Service type}), where (UE device, send, Service Request message) constitutes the primary triple, and {to, AN device}, {contain, Service type} are auxiliary pairs. Based on this representation, we propose an attributed triple extraction model. This model employs multi-label classifiers to identify potential relations, followed by entity extraction through a sequence tagger based on the BIO pattern. This process yields the primary triple and auxiliary pairs without predefining entity types.

Attributed Triple Extraction
by Combination Under
Contrastive Learning
Runzhe Wang1,2, Gaofeng Xu1,2, Guilin Qi1,2(B), Yun Li3, Yongrui Chen1,2,
Songlin Zhai1,2, Yuxin Zhang1,2, Rihui Jin1,2, Nijun Li3, and Qianren Wang3
1 School of Computer Science and Engineering, Southeast University,
Nanjing 211189, China
{220232335,230229361,gqi,yrchen,songlin zhai,zzyx cs,220212061}@seu.edu.cn
2 Key Laboratory of New Generation Artiﬁcial Intelligence Technology and its
Interdisplinary Applications (Southeast University), Ministry of Education,
Beijing, China
3 Advanced Cognitive AI Lab, Huawei Technologies, Shenzhen, China
{liyun5,linijun,wangqianran}@huawei.com
Abstract. Complex knowledge extraction is a signiﬁcant topic of
research. Previous methodologies were limited to extracting only the
overall relations among multiple entities and they necessitated the prede-
ﬁned entity types, rendering them inapplicable in industrial domains such
as Information and Communication Technology (ICT), where numerous
entities are undeﬁned. To address these challenges, this paper introduces
a novel representation for complex knowledge: attributed triple, which
consists of a primary triple coupled with auxiliary attribute-value pairs,
enabling a ﬁne-grained depiction of both the overall inter-entity relations
and the pairwise relations between entities. Based on this representation,
we propose an attributed triple extraction model. This model employs
multi-label classiﬁers to identify all potential relations before extract-
ing relevant entities with sequence taggers based on the BIO pattern,
where B represents Begin, I representsInside and O represents Outside.
This process makes the extraction of primary triples and auxiliary pairs
without predeﬁning entity types. Subsequently, a linear combination dis-
criminator is designed to assess the semantic feasibility of candidate com-
binations formed by concatenating primary triplets and auxiliary pairs.
Furthermore, contrastive learning is adopted to enhance the model’s abil-
ity of representation with insuﬃcient training data. To better evaluate
the model’s performance over extracting attributed triple, we construct a
high-quality dataset based on data from ICT corpus. Our model demon-
strates substantial and consistent superiority over baselines across vari-
ous metrics. Our code is publicly available. https://github.com/sid0527/
Attributed-Triple-Extraction
Keywords: Attributed Triple · Information Extraction · Contrastive
Learning
The author contributed equally to this work.
c
⃝The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024
M. Onizuka et al. (Eds.): DASFAA 2024, LNCS 14856, pp. 402–414, 2024.
https://doi.org/10.1007/978-981-97-5575-2_30
Attributed Triple Extraction by Combination Under Contrastive Learning
403
Fig. 1. The diﬀerences in the various representations of complex data. In the ﬁgure, the
red font indicates entities, green indicates entity types and yellow indicates relations.
(Color ﬁgure online)
1
Introduction
Information extraction, an essential process in the construction and comple-
tion of knowledge graphs [1,2], facilitates the automated retrieval of structured
information from voluminous text. Previous works mainly focused on binary
relational information extraction, which is also called triple extraction [3]. How-
ever, in numerous industrial scenarios, the text encapsulates complex knowledge
characterized by interactions among multiple entities. The extraction of such
intricate knowledge is signiﬁcant as it can enrich knowledge graphs with more
comprehensive and detailed entity-relation information.
Current studies on complex knowledge extraction lie primarily within the
biomedical domain, which involves distilling an overarching relation among spe-
ciﬁc categories of entities [4–7]. These methods necessitate the predeﬁnition of
entity types prior to extraction, as depicted in Fig. 1. However, their application
in other industrial ﬁelds, such as Information and Communication Technology
(ICT), can lead to two major challenges. 1) Beyond the overall relations among
entities, individual relations between entities also matter. For instance, UE
device sends Service Request message to AN device, and the message
contains Service type, where UE and AN are undeﬁned entities in ICT. Here,
the relation between UE device and Service Request message is deﬁned
as send, while the relation between Service Request message and Service
type is contain and the collective relation of the entities (UE device, Service
Request message, AN device) is (send. . . to. . . ). Representing these entities
using a single overarching relation can compromise information integrity. 2) The
diversity and continual emergence of new entity types in textual data make the
provision of entity-type predeﬁnition impractical.
Intuitively, the ideal representation would involve a more granular form of
complex knowledge, capable of articulating both the pairwise and overall rela-
tions among multiple entities. Consequently, we distill complex text into primary
components and modiﬁers, achieving a ﬁne-grained representation of complex
knowledge. Moreover, we observe that the range of relations in most extraction
404
R. Wang et al.
datasets is considerably narrower than the variety of entity types. An intuitive
solution is to identify all relevant entity types within the text by recognizing
relations, thus circumventing the need of entity-type predeﬁnition.
In this paper, we introduce a novel representational form for complex knowl-
edge: attributed triple, comprising a primary triple allied with auxiliary
attribute-value pairs. The primary triple delineates the central subject-predicate-
object structure of the text, while the auxiliary pairs provide additional con-
textual details. The instance mentioned above can be represented as (UE
device, send:{to, AN device}, Service Request message:{contain, Ser-
vice type}), where (UE device, send, Service Request message) consti-
tutes the primary triple, and {to, AN device}, {contain, Service type} are
auxiliary pairs. Based on this representation, we propose an attributed triple
extraction model. This model employs multi-label classiﬁers to identify poten-
tial relations, followed by entity extraction through a sequence tagger based on
the BIO pattern. This process yields the primary triple and auxiliary pairs with-
out predeﬁning entity types. A linear combination discriminator is then used to
evaluate the semantic feasibility of candidate combinations, formed by concate-
nating primary triplets and auxiliary pairs. To reduce the eﬀect of industrial
data scarcity on model’s ability of representation, contrastive learning is inte-
grated during the training phase. Moreover, to assess the model’s eﬀectiveness in
extracting attributed triples, we have developed a high-quality dataset derived
from the ICT domain. This dataset, featuring a variety of primary and auxiliary
relations with diverse combinations, more accurately mirrors the complexities
of real-world industrial scenarios, in contrast to existing data. Extensive exper-
imental results demonstrate that our model can eﬀectively extract high-quality
attributed triples, proving to be of signiﬁcant value in practical applications. In
summary, the contributions of this paper are threefold:
1. We introduce a new representational form for multi-entity information:
attributed triple, which is capable of depicting both the pairwise and over-
all relations among entities, thereby enabling ﬁne-grained representation of
information among multiple entities.
2. We have developed an attributed triple extraction model that prioritizes rela-
tion dentiﬁcation over entity recognition, negating the need for predeﬁning
entity type. This model integrates contrastive learning and employs a shared
loss function during training, enhancing the accuracy of the model outputs
and substantially improving its practicality.
3. To thoroughly evaluate the model’s capacity for extracting attributed triples,
we semi-automatically curated a high-quality dataset based on the ICT
domain. This dataset comprises a diverse array of primary and auxiliary rela-
tions, better mirroring actual industry scenarios compared to open datasets.
Extensive experiments on both open and industrial datasets demonstrate that
our model achieves excellent performance across a range of metrics.
Attributed Triple Extraction by Combination Under Contrastive Learning
405
Fig. 2. An overview of Combination under Contrastive Learning. Both relation iden-
tiﬁcation and related entity recognition module are designed to be composed of two
similar components, which beneﬁts the separate extraction of primary triples and aux-
iliary relation-entity pairs. To address insuﬃciency of training data, parameter sharing
mechanism and contrastive learning are introduced.
2
Preliminaries
2.1
Attributed Triple
An attributed triple F is a primary subject-relation-object triple (s, r, o) cou-
pled with m auxiliary pairs {(ai, vi)}m
i=1, where r, a1, . . . , am ∈R and s, o, v1,
. . . , vm
∈
E, with R and E
being the sets of relations and entities,
respectively. We denote the attributed triple as F
=
(s : {(ai, vi)}ms
i=1 ,
r : {(ai, vi)}mr
i=1 , o : {(ai, vi)}mo
i=1), where ms + mr + m0 = m and x : y denotes
y attribute to x. We slightly abuse terminology here by referring to the primary
relation and auxiliary attribute as relations, and referring to the subject, object,
and values as entities unless otherwise speciﬁed.
2.2
Problem Formulation
Our goal is to extract attributed triples from text. Given the input text T =
{t1, t2, . . . , tn}, we aim to extract G(T ) →F, where G denotes the attributed
triples extraction model.
3
Combination Under Contrastive Learning
This section presents CCL, our model to extract attributed triples by combi-
nation under contrastive learning. There are two key modules in our model:
attributed triple combination and contrastive learning. The former combines
attributed triples by Relation Identiﬁcation, Related Entity Recognition
and Combination, and the latter designs a loss to address data scarcity. An
overview illustration of our model is shown in Fig. 2.
406
R. Wang et al.
3.1
Attributed Triple Combination
A pre-trained BERT is employed as encoder to transform the input text S into
the d-dimensional contextual representation:
[h1, h2, . . . , hL] = Encoder(S),
(1)
where hi corresponds to the i-th input token and L is the length of S.
Relation Identiﬁcation. Diﬀerent from works [8,9] which extract all candi-
date entities before permutating them to form triples, we ﬁrst obtain subsets of
potential primary relation types and auxiliary relation types respectively, and
only perform entity extraction for these potential relations. Since much more
possible entity combinations is taken into consideration in one attributed triple,
this part will greatly improve the eﬃciency of extraction.
With the output of the encoder Henc = [h1, h2, . . . , hL], probability of one
relation type being potential in text S can be predicted by:
Prel = Sigmoid(Wrelhavg
enc + brel),
(2)
where havg
enc is the average vector among all hi in Henc. Wrel and brel represent
a trainable weight and a trainable bias, respectively. The subset of potential
relations can be obtained as Rpot = {ri|Pri > λ}, where λ is a preset threshold.
According to cross-entropy loss, loss function of this part can be deﬁned as:
lpri
rel = −1
npri
rel
npri
rel

i=1

ypri
i
log(Prel) + (1 −ypri
i
) log(1 −Prel)

,
(3)
laux
rel = −
1
naux
rel
naux
rel

i=1
(yaux
i
log(Prel) + (1 −yaux
i
) log(1 −Prel)) ,
(4)
where npri
rel and naux
rel denote the size of primary and auxiliary relation set.
Related Entity Recognition. Potential primary relation types and auxiliary
relation types obtained will be the input of this part in the form of embedding,
together with that of each token in text S [10]. We can update the representation
of i-th token in the context of potential relation type rj to:
hi,j = hi + vrj,
(5)
where vj ∈Rd×1 is a trainable embedding vector for the j-th relation. Related
entity recognition is based on the new representation through sequence tagging
in the BIO pattern as is shown in Fig. 2.
When it comes to the network in this component, we dismiss the LSTM-CRF
[11], which proves to be eﬀective but may be too complicated to be trained well
in consideration of the insuﬃciency of training data in some industrial domains.
Attributed Triple Extraction by Combination Under Contrastive Learning
407
Instead, FCNN is adopted to simplify the network. For each potential primary
relation type, there are three entities to be recognized: subject, object, and rela-
tion. Hence, three rounds of sequence tagging will be performed independently:
P sub
i,j
= Softmax(Wsubhi,j + bsub),
(6)
P obj
i,j = Softmax(Wobjhi,j + bobj),
(7)
P re
i,j = Softmax(Wrehi,j + bre),
(8)
where W(·) ∈R3×d is a trainable weight and b(·) ∈R3×1 is a trainable bias.
For each potential auxiliary relation type, we just take advantage of Wobj and
bobj to get the object in the context of this relation.
Loss function of three taggers can be calculated by:
lsub
seq = −
npri
pre

j=1
L

i=1
ysub
i,j log(P sub
i,j )
L × npri
pre
,
(9)
lmen
seq
= −
npri
pre

j=1
L

i=1
ypre
i,j log(P pre
i,j )
L × npri
pre
,
(10)
lobj
seq = −
1
2 × L[
npri
pre

j=1
L

i=1
yobj
i,j log(P obj
i,j )
npri
pre
−
naux
pre

j=1
L

i=1
yobj
i,j log(P obj
i,j )
naux
pre
],
(11)
where L is the length of input text. npri
pre and naux
pre represent the size of potential
primary relation type set and that of auxiliary relation type set.
Combination.
Each auxiliary pair will be concatenated with all primary
triplets respectively to generate candidate combinations. To determine whether
a combination is semantically valid, a linear classiﬁer is introduced in this com-
ponent.
It is worth mentioning that each part of the primary triple is an entity com-
posed of multiple tokens, which may hinder parallel computation. Therefore, we
represent entities with an average vector to keep the dimension. The represen-
tation of every combination can be obtained as follows:
hcom
i,j
= [hei; vrj; hej],
(12)
where ei is an entity in one primary triple and ej is the object corresponding to
auxiliary relation type rj. Symbol ; represents the operation of concatenation.
Probability of such combination being valid can be measured by:
P com
i,j
= Sigmoid(Wcomhcom
i,j
+ bcom),
(13)
where Wcom is a trainable weight and bcom is a trainable bias. If P com
i,j
is the
greatest one among all candidate combinations containing the auxiliary pair
(rj, ej), (rj, ej) will be combined with ei into an attributed triple.
408
R. Wang et al.
The loss of combination is similar to that of relation identiﬁcation:
lcom = −
npri

i=1
naux

j=1
yi,j log(P com
i,j ) + (1 −yi,j) log(1 −P com
i,j )
npri × naux

,
(14)
where npri refers to the number of entities in the primary triples and naux refers to the
number of auxiliary pairs in input text.
3.2
Contrastive Learning
Contrastive learning is introduced to enable our model to learn subtle diﬀerences
between samples, which improves its capability of representation when training data is
insuﬃcient. The contrastive loss is calculated based on all candidate combinations in
one batch, where two combinations with same relation type will be taken as positive.
Following previous works [12,13], we deﬁne the loss for a positive pair as:
l(hcomi, hcomj) = −log

e(hcomi ·hcom
j
/τ)
ncom
k=0 1i̸=k · e(hcomi ·hcomk /τ)

,
(15)
where comi is representation of one combination, τ is a temperature parameter, and
ncom is number of candidate combinations in the batch. 1i̸=k is the indicator function.
The ﬁnal loss is computed from all positive pairs in the batch.
3.3
Objective Function
We deﬁne the objective function as a total loss composed of loss from each module.
With Eqs. 3 and 4, loss for relation identiﬁcation is denoted as:
lrel = 1
2(lpri
rel + laux
rel ).
(16)
According to Eqs. 9, 10 and 11, the ﬁnal loss for entity recognition is deﬁned as an
average of three loss values:
lseq = 1
3(lsub
seq + lobj
seq + lmen
seq ).
(17)
The ﬁnal objective function is the total of mentioned loss functions:
l = lrel + lseq + α × lcom + (1 −α) × lcon,
(18)
where 0 ≤α ≤1 is a weight to be tuned during training.
4
Experiment
4.1
Datasets
To evaluate the performance of our model on open data, we choose DuIE, a well-known
dataset for information extraction, as the source of one dataset because some triples
in DuIE contain complex objects. Each of these complex objects provides not only a
object value but also some other slots that can be taken as auxiliary relations. In this
way, we obtain a dataset with 2070 attributed triples in the newly proposed form.
Together with 3 experts, we also manually build a dataset based on information
from ICT corpus. This dataset consists of 171 attributed triples, and it enables us to
explore how our model behaves when applied to real industrial scenario. Details of the
proposed datasets are shown in Table 1.
Attributed Triple Extraction by Combination Under Contrastive Learning
409
Table 1. Details of the proposed datasets.
Dataset Composition
Relation type
DuIE
Primary triple Dub
Play
Award Box Oﬃce -
461
1302
469
118
-
Auxiliary pair inWork onDate inArea -
-
1978
340
118
-
-
ICT
Primary triple Support Set
Send
Provide
-
39
52
44
36
-
Auxiliary pair Be
TO
For
Contain
Through
27
44
60
20
48
4.2
Experiment Settings
Baseline. In the absence of existing work for extracting our proposed complex rela-
tions, we employ a simpliﬁed version of our model as a baseline for comparison. This
simpliﬁed model omits the contrastive learning and combination modules, using a
straightforward nearest-neighbor principle for combination, eﬀective in simpler con-
texts. Additionally, it alters the entity recognition module to train object recognizers
independently for primary and auxiliary relations, without parameter sharing.
Given the proven excellence of large language models in various natural language
processing tasks, we also utilize GPT-3.5-turbo as an additional baseline to assess our
model’s performance.
Metrics. Our evaluation metrics for model performance include Precision, Recall,
and F1 score, encompassing three aspects: primary triples, auxiliary pairs, and
attributed triples. For primary triples and auxiliary pairs, we apply strict match-
ing, deeming results correct only if they exactly match the annotations. In evaluating
attributed triples, we combine strict and partial matching to more accurately gauge
the model’s eﬀectiveness. Equation 19 details the evaluation criteria for each attributed
triple.
Nprecision = 1primary triple · ncorrect
ntotal , Nrecall = 1primary triple ·
ncorrect
nannotation ,
(19)
where 1primary triple equals to 1 only when the extracted primary triple is correct
and ncorrect is the number of auxiliary pairs extracted correctly. ntotal and nannotation
represent the total number of auxiliary pairs extracted and the number of auxiliary
pairs annotated respectively. The ﬁnal Precision and Recall will be the average of
all Nprecision and Nrecall.
Details. Our experiment, conducted on the PyTorch framework, utilized an NVIDIA
RTX 3080Ti GPU and CUDA 11.3. We employed Hugging Face’s bert-chinese-base
as the pre-trained BERT model, featuring a 768-dimension hidden text representation.
The maximum sentence sequence length was set at 100. Batch sizes were 5 for training,
24 for validation, and 64 for testing, with a maximum of 40 training epochs. Hyper-
parameters were ﬁne-tuned on the validation set, setting BERT’s learning rate at 1e-4
with a 0.01 weight decay, and 1e-3 for other model components.
410
R. Wang et al.
Table 2. Precision(%), Recall (%) and F1-score (%) of CCL and baselines.
Dataset Model
Composition Prec. Rec.
F1
DuIE
GPT-3.5-turbo Primary
0.577 0.536 0.556
Auxiliary
0.324 0.302 0.314
Attributed
0.308 0.286 0.297
Simpliﬁed CCL Primary
0.468 0.552 0.507
Auxiliary
0.213 0.235 0.223
Attributed
0.172 0.201 0.185
CCL
Primary
0.708 0.687 0.697
Auxiliary
0.729 0.642 0.683
Attributed
0.646 0.627 0.636
ICT
GPT-3.5-turbo Primary
0.682 0.625 0.652
Auxiliary
0.321 0.290 0.305
Attributed
0.347 0.309 0.327
Simpliﬁed CCL Primary
0.469 0.567 0.514
Auxiliary
0.287 0.292 0.289
Attributed
0.233 0.235 0.234
CCL
Primary
0.917 0.917 0.917
Auxiliary
0.855 0.824 0.839
Attributed
0.810 0.805 0.807
4.3
Results
We evaluate CCL and baselines on two datasets through ten-fold cross-validation and
the overall results are shown in Table 2. CCL performs best in all evaluation metrics
and even outperforms GPT-3.5-turbo by 33.9% and 48.0% in F1-score on two datasets.
We also ﬁnd that all extraction is better performed on ICT dataset. One important
reason is that open dataset is built by means of some ﬁxed mapping rules without
considering semantic correctness, which not only has bad eﬀects on model training but
may also cause wrong judgment on outputs due to incorrect annotations. In addition,
entities in open data is more complicated while most entities in the ICT dataset are
merely names of devices, which reduces diﬃculty in relation extraction.
It’s noteworthy that GPT’s suboptimal performance may stem from the inherent
traits of generative models. We observe GPT often produces semantically valid relations
that fall outside predeﬁned ranges. While this is minor for general reading, it poses
challenges in data structuring and normalization, requiring additional alignment eﬀorts.
Consequently, CCL proves more adept for structuring text data in industrial contexts.
4.4
Ablation Study
To assess the impact of individual modules on extraction performance, we per-
formed ablation studies on the contrastive learning module, combination module, and
parameter-sharing mechanism. Table 3 depicts the performance variations in attributed
Attributed Triple Extraction by Combination Under Contrastive Learning
411
triple extraction on the ICT dataset upon sequential removal of each component.
Results indicate that all these modules positively inﬂuence attributed triple extrac-
tion. Model4 is actually the simpliﬁed CCL.
Table 3. Results of ablation study.
Name
Detail
Prec. Rec.
F1
Model1 CCL
0.810 0.805 0.807
Model2 Model1-Contrastive Learning 0.652 0.583 0.616
Model3 Model2-Combination
0.429 0.402 0.415
Model4 Model3-Parameter Sharing
0.233 0.235 0.234
Eﬀect of Contrastive Learning. The results in the ﬁrst two rows of Table 3
indicate that the contrastive learning module does enhance the model’s capability
of representation with limited training data, leading to an improvement of 19.1% in
F1-score when extracting attributed triples. Extra experiments demonstrate that this
module also enhances CCL’s performance in extracting primary and auxiliary relational
information by 7.8% and 13.1% in F1-score respectively, which is shown in Table 4.
Table 4. Detailed results of ablation on contrastive learning.
Model
Composition
Prec. Rec.
F1
CCL
Primary triple 0.917 0.917 0.917
Auxiliary pair 0.855 0.824 0.839
CCL-Contrastive Learning Primary triple 0.832 0.814 0.823
Auxiliary pair 0.705 0.679 0.692
Eﬀect of Combination. Results in Table 3 show that replacing combination mod-
ule by simple rules like nearest-neighbor criterion brings a decline of 20.1% in F1-score,
which means that the combination module is eﬀective. In fact, we ﬁnd that combination
can be replaced by more complicated rules that contain common combination patterns.
But it takes too much time to manually summarize such rules and rules may fail when
encountering unseen combination patterns. Hence, combination module is eﬃcient in
industrial scenarios, especially when the semantics are complicated.
412
R. Wang et al.
Table 5. Results of entity recognition w/o parameter sharing.
Model
Prec. Rec.
F1
CCL
0.931 0.923 0.927
CCL-Parameter Sharing 0.543 0.521 0.532
Eﬀect of Parameter Sharing. Parameter sharing mechanism is designed to
increase the training of entity recognition module when in lack of training data. Accord-
ing to ablation results, parameter sharing mechanism improves F1-score by 18.1% on
ICT dataset. To further verify its eﬀect on entity recognition, we compare the per-
formance of entity recognition of our model on ICT dataset when with and without
parameter sharing. The result in Table 5 proves the eﬀectiveness of this mechanism.
Fig. 3. Eﬀect of contrastive learning on combination. CL is short for contrastive learn-
ing in legend. We use candidates shown in combination module in Fig. 2.
4.5
Case Study
In order to illustrate the advantages of combination under contrastive learning more
intuitively, we conduct a case study by comparing results of combination module w/o
contrastive learning. Figure 3 shows the visualized eﬀect of contrastive learning on the
probability of each candidate combination being valid. It can be observed that valid
combinations are more likely to be accepted and invalid combinations are more likely
to be denied with contrastive learning. This reﬂects that our model learns a more
reasonable representation space with same training data, which is of great use when
the semantics are puzzling.
Attributed Triple Extraction by Combination Under Contrastive Learning
413
5
Conclusion
In this paper, we presented a novel representation of complex knowledge, termed
attributed triple, and develop a corresponding extraction model. The attributed
triple is designed to intricately detail both the collective and speciﬁc relations among
multiple entities. This extraction model identiﬁes relations and then leverages these
identiﬁed relations to extract relevant entities, thereby generating the primary triple
and auxiliary pairs. It eﬀectively overcomes the limitations imposed by entity-type pre-
deﬁnition. Additionally, a linear combination discriminator is utilized to ascertain the
semantic feasibility of the combinations formed by these primary triples and auxiliary
pairs. To robustly evaluate the model’s proﬁciency in extracting attributed triple data,
we constructed a high-quality dataset derived from ICT corpus. Our model signiﬁcantly
outperforms established baselines across a range of metrics.1
References
1. Al-Moslmi, T., Oca˜na, M., Opdahl, A., et al.: Named entity extraction for knowl-
edge graphs: a literature overview. IEEE Access 8, 32862–32881 (2020)
2. Ma, X.: Knowledge graph construction and application in geosciences: a review.
Comput. Geosci. 161, 105082 (2022)
3. Zheng, H., Wen, R., Chen, X., et al.: PRGC: Potential relation and global cor-
respondence based joint relational triple extraction. In: Proceedings of the 59th
Annual Meeting of the Association for Computational Linguistics (Volume: Long
Papers)1, 6225–6235 (2021)
4. Peng, N., Poon, H., Quirk, C., et al.: Cross-sentence n-ary relation extraction with
graph LSTMs. Trans. Assoc. Comput. Linguist. 5, 101–115 (2017)
5. Song, L., Zhang, Y., Wang, Z., et al.: N-ary relation extraction using graph-state
LSTM. In: Proceedings of the 2018 Conference on Empirical Methods in Natural
Language Processing, pp. 2226–2235 (2018)
6. Guo, Z., Zhang, Y., Lu.W.: Attention guided graph convolutional networks for
relation extraction. In: Proceedings of the 57th Conference of the Association for
Computational Linguistics, Vol.1, Long Papers, pp. 241–251 (2019)
7. Zeng, D., Zhao, C., Jiang, C., et al.: Document-level relation extraction with con-
text guided mention integration and inter-pair reasoning. IEEE ACM Trans. Audio
Speech Lang. Process 31, 3659–3666 (2023)
8. Shang, Y., Huang, H., Sun, X., et al.: Relational triple extraction: one step is
enough. In: Luc De Raedt, editor, In: Proceedings of the Thirty-First International
Joint Conference on Artiﬁcial Intelligence, pp. 4360–4366 (2022)
9. Shang, Y., Huang, H., Mao, X.:. Onerel: joint entity and relation extraction with
one module in one step. In: Thirty-Sixth AAAI Conference on Artiﬁcial Intelli-
gence, pp. 11285–11293 (2022)
10. Wei, Z., Su, J., Wang, Y., et al.: A novel cascade binary tagging framework for
relational triple extraction. In: Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics, pp. 1476–1488 (2020)
1 This work is partially supported by National Nature Science Foundation of China
under No. U21A20488. We thank the Big Data Computing Center of Southeast
University for providing the facility support on the numerical calculations in this
paper.
414
R. Wang et al.
11. Panchendrarajan, R., Amaresan, A.: Bidirectional LSTM-CRF for named entity
recognition. In: Proceedings of the 32nd Paciﬁc Asia Conference on Language,
Information and Computation (2018)
12. Ye, H., Zhang, N., Deng, S., et al.: Contrastive triple extraction with generative
transformer. In: Proceedings of the AAAI Conference on Artiﬁcial Intelligence, vol.
35, 14257–14265 (2021)
13. Chen, T., Kornblith, S., Norouzi, M. et al.: A simple framework for contrastive
learning of visual representations. In: Proceedings of the 37th International Confer-
ence on Machine Learning, vol. 119 of Proceedings of Machine Learning Research,
pp. 1597–1607 (2020)


Question:
Why contrastive learning module is employed in the extraction model?
A. To mitigate bad effects caused by the shortage of domain data.
B. To improve the model performance on extracting entities and relations.
C. To enhance the model's capability of representation.
D. To accelerate the model training by learn subtle differences between samples.

Answer: AC

Contrastive learning is introduced to enable our model to learn subtle diﬀerences between samples, which improves its capability of representation when training data is insuﬃcient. The contrastive loss is calculated based on all candidate combinations in one batch, where two combinations with same relation type will be taken as positive. Following previous works [12,13], we deﬁne the loss for a positive pair as:
l(hcomi, hcomj) = −log

e(hcomi ·hcom
j
/τ)
ncom
k=0 1i̸=k · e(hcomi ·hcomk /τ)

,
where comi is representation of one combination, τ is a temperature parameter, and ncom is number of candidate combinations in the batch. 1i̸=k is the indicator function. The ﬁnal loss is computed from all positive pairs in the batch.

Attributed Triple Extraction
by Combination Under
Contrastive Learning
Runzhe Wang1,2, Gaofeng Xu1,2, Guilin Qi1,2(B), Yun Li3, Yongrui Chen1,2,
Songlin Zhai1,2, Yuxin Zhang1,2, Rihui Jin1,2, Nijun Li3, and Qianren Wang3
1 School of Computer Science and Engineering, Southeast University,
Nanjing 211189, China
{220232335,230229361,gqi,yrchen,songlin zhai,zzyx cs,220212061}@seu.edu.cn
2 Key Laboratory of New Generation Artiﬁcial Intelligence Technology and its
Interdisplinary Applications (Southeast University), Ministry of Education,
Beijing, China
3 Advanced Cognitive AI Lab, Huawei Technologies, Shenzhen, China
{liyun5,linijun,wangqianran}@huawei.com
Abstract. Complex knowledge extraction is a signiﬁcant topic of
research. Previous methodologies were limited to extracting only the
overall relations among multiple entities and they necessitated the prede-
ﬁned entity types, rendering them inapplicable in industrial domains such
as Information and Communication Technology (ICT), where numerous
entities are undeﬁned. To address these challenges, this paper introduces
a novel representation for complex knowledge: attributed triple, which
consists of a primary triple coupled with auxiliary attribute-value pairs,
enabling a ﬁne-grained depiction of both the overall inter-entity relations
and the pairwise relations between entities. Based on this representation,
we propose an attributed triple extraction model. This model employs
multi-label classiﬁers to identify all potential relations before extract-
ing relevant entities with sequence taggers based on the BIO pattern,
where B represents Begin, I representsInside and O represents Outside.
This process makes the extraction of primary triples and auxiliary pairs
without predeﬁning entity types. Subsequently, a linear combination dis-
criminator is designed to assess the semantic feasibility of candidate com-
binations formed by concatenating primary triplets and auxiliary pairs.
Furthermore, contrastive learning is adopted to enhance the model’s abil-
ity of representation with insuﬃcient training data. To better evaluate
the model’s performance over extracting attributed triple, we construct a
high-quality dataset based on data from ICT corpus. Our model demon-
strates substantial and consistent superiority over baselines across vari-
ous metrics. Our code is publicly available. https://github.com/sid0527/
Attributed-Triple-Extraction
Keywords: Attributed Triple · Information Extraction · Contrastive
Learning
The author contributed equally to this work.
c
⃝The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024
M. Onizuka et al. (Eds.): DASFAA 2024, LNCS 14856, pp. 402–414, 2024.
https://doi.org/10.1007/978-981-97-5575-2_30
Attributed Triple Extraction by Combination Under Contrastive Learning
403
Fig. 1. The diﬀerences in the various representations of complex data. In the ﬁgure, the
red font indicates entities, green indicates entity types and yellow indicates relations.
(Color ﬁgure online)
1
Introduction
Information extraction, an essential process in the construction and comple-
tion of knowledge graphs [1,2], facilitates the automated retrieval of structured
information from voluminous text. Previous works mainly focused on binary
relational information extraction, which is also called triple extraction [3]. How-
ever, in numerous industrial scenarios, the text encapsulates complex knowledge
characterized by interactions among multiple entities. The extraction of such
intricate knowledge is signiﬁcant as it can enrich knowledge graphs with more
comprehensive and detailed entity-relation information.
Current studies on complex knowledge extraction lie primarily within the
biomedical domain, which involves distilling an overarching relation among spe-
ciﬁc categories of entities [4–7]. These methods necessitate the predeﬁnition of
entity types prior to extraction, as depicted in Fig. 1. However, their application
in other industrial ﬁelds, such as Information and Communication Technology
(ICT), can lead to two major challenges. 1) Beyond the overall relations among
entities, individual relations between entities also matter. For instance, UE
device sends Service Request message to AN device, and the message
contains Service type, where UE and AN are undeﬁned entities in ICT. Here,
the relation between UE device and Service Request message is deﬁned
as send, while the relation between Service Request message and Service
type is contain and the collective relation of the entities (UE device, Service
Request message, AN device) is (send. . . to. . . ). Representing these entities
using a single overarching relation can compromise information integrity. 2) The
diversity and continual emergence of new entity types in textual data make the
provision of entity-type predeﬁnition impractical.
Intuitively, the ideal representation would involve a more granular form of
complex knowledge, capable of articulating both the pairwise and overall rela-
tions among multiple entities. Consequently, we distill complex text into primary
components and modiﬁers, achieving a ﬁne-grained representation of complex
knowledge. Moreover, we observe that the range of relations in most extraction
404
R. Wang et al.
datasets is considerably narrower than the variety of entity types. An intuitive
solution is to identify all relevant entity types within the text by recognizing
relations, thus circumventing the need of entity-type predeﬁnition.
In this paper, we introduce a novel representational form for complex knowl-
edge: attributed triple, comprising a primary triple allied with auxiliary
attribute-value pairs. The primary triple delineates the central subject-predicate-
object structure of the text, while the auxiliary pairs provide additional con-
textual details. The instance mentioned above can be represented as (UE
device, send:{to, AN device}, Service Request message:{contain, Ser-
vice type}), where (UE device, send, Service Request message) consti-
tutes the primary triple, and {to, AN device}, {contain, Service type} are
auxiliary pairs. Based on this representation, we propose an attributed triple
extraction model. This model employs multi-label classiﬁers to identify poten-
tial relations, followed by entity extraction through a sequence tagger based on
the BIO pattern. This process yields the primary triple and auxiliary pairs with-
out predeﬁning entity types. A linear combination discriminator is then used to
evaluate the semantic feasibility of candidate combinations, formed by concate-
nating primary triplets and auxiliary pairs. To reduce the eﬀect of industrial
data scarcity on model’s ability of representation, contrastive learning is inte-
grated during the training phase. Moreover, to assess the model’s eﬀectiveness in
extracting attributed triples, we have developed a high-quality dataset derived
from the ICT domain. This dataset, featuring a variety of primary and auxiliary
relations with diverse combinations, more accurately mirrors the complexities
of real-world industrial scenarios, in contrast to existing data. Extensive exper-
imental results demonstrate that our model can eﬀectively extract high-quality
attributed triples, proving to be of signiﬁcant value in practical applications. In
summary, the contributions of this paper are threefold:
1. We introduce a new representational form for multi-entity information:
attributed triple, which is capable of depicting both the pairwise and over-
all relations among entities, thereby enabling ﬁne-grained representation of
information among multiple entities.
2. We have developed an attributed triple extraction model that prioritizes rela-
tion dentiﬁcation over entity recognition, negating the need for predeﬁning
entity type. This model integrates contrastive learning and employs a shared
loss function during training, enhancing the accuracy of the model outputs
and substantially improving its practicality.
3. To thoroughly evaluate the model’s capacity for extracting attributed triples,
we semi-automatically curated a high-quality dataset based on the ICT
domain. This dataset comprises a diverse array of primary and auxiliary rela-
tions, better mirroring actual industry scenarios compared to open datasets.
Extensive experiments on both open and industrial datasets demonstrate that
our model achieves excellent performance across a range of metrics.
Attributed Triple Extraction by Combination Under Contrastive Learning
405
Fig. 2. An overview of Combination under Contrastive Learning. Both relation iden-
tiﬁcation and related entity recognition module are designed to be composed of two
similar components, which beneﬁts the separate extraction of primary triples and aux-
iliary relation-entity pairs. To address insuﬃciency of training data, parameter sharing
mechanism and contrastive learning are introduced.
2
Preliminaries
2.1
Attributed Triple
An attributed triple F is a primary subject-relation-object triple (s, r, o) cou-
pled with m auxiliary pairs {(ai, vi)}m
i=1, where r, a1, . . . , am ∈R and s, o, v1,
. . . , vm
∈
E, with R and E
being the sets of relations and entities,
respectively. We denote the attributed triple as F
=
(s : {(ai, vi)}ms
i=1 ,
r : {(ai, vi)}mr
i=1 , o : {(ai, vi)}mo
i=1), where ms + mr + m0 = m and x : y denotes
y attribute to x. We slightly abuse terminology here by referring to the primary
relation and auxiliary attribute as relations, and referring to the subject, object,
and values as entities unless otherwise speciﬁed.
2.2
Problem Formulation
Our goal is to extract attributed triples from text. Given the input text T =
{t1, t2, . . . , tn}, we aim to extract G(T ) →F, where G denotes the attributed
triples extraction model.
3
Combination Under Contrastive Learning
This section presents CCL, our model to extract attributed triples by combi-
nation under contrastive learning. There are two key modules in our model:
attributed triple combination and contrastive learning. The former combines
attributed triples by Relation Identiﬁcation, Related Entity Recognition
and Combination, and the latter designs a loss to address data scarcity. An
overview illustration of our model is shown in Fig. 2.
406
R. Wang et al.
3.1
Attributed Triple Combination
A pre-trained BERT is employed as encoder to transform the input text S into
the d-dimensional contextual representation:
[h1, h2, . . . , hL] = Encoder(S),
(1)
where hi corresponds to the i-th input token and L is the length of S.
Relation Identiﬁcation. Diﬀerent from works [8,9] which extract all candi-
date entities before permutating them to form triples, we ﬁrst obtain subsets of
potential primary relation types and auxiliary relation types respectively, and
only perform entity extraction for these potential relations. Since much more
possible entity combinations is taken into consideration in one attributed triple,
this part will greatly improve the eﬃciency of extraction.
With the output of the encoder Henc = [h1, h2, . . . , hL], probability of one
relation type being potential in text S can be predicted by:
Prel = Sigmoid(Wrelhavg
enc + brel),
(2)
where havg
enc is the average vector among all hi in Henc. Wrel and brel represent
a trainable weight and a trainable bias, respectively. The subset of potential
relations can be obtained as Rpot = {ri|Pri > λ}, where λ is a preset threshold.
According to cross-entropy loss, loss function of this part can be deﬁned as:
lpri
rel = −1
npri
rel
npri
rel

i=1

ypri
i
log(Prel) + (1 −ypri
i
) log(1 −Prel)

,
(3)
laux
rel = −
1
naux
rel
naux
rel

i=1
(yaux
i
log(Prel) + (1 −yaux
i
) log(1 −Prel)) ,
(4)
where npri
rel and naux
rel denote the size of primary and auxiliary relation set.
Related Entity Recognition. Potential primary relation types and auxiliary
relation types obtained will be the input of this part in the form of embedding,
together with that of each token in text S [10]. We can update the representation
of i-th token in the context of potential relation type rj to:
hi,j = hi + vrj,
(5)
where vj ∈Rd×1 is a trainable embedding vector for the j-th relation. Related
entity recognition is based on the new representation through sequence tagging
in the BIO pattern as is shown in Fig. 2.
When it comes to the network in this component, we dismiss the LSTM-CRF
[11], which proves to be eﬀective but may be too complicated to be trained well
in consideration of the insuﬃciency of training data in some industrial domains.
Attributed Triple Extraction by Combination Under Contrastive Learning
407
Instead, FCNN is adopted to simplify the network. For each potential primary
relation type, there are three entities to be recognized: subject, object, and rela-
tion. Hence, three rounds of sequence tagging will be performed independently:
P sub
i,j
= Softmax(Wsubhi,j + bsub),
(6)
P obj
i,j = Softmax(Wobjhi,j + bobj),
(7)
P re
i,j = Softmax(Wrehi,j + bre),
(8)
where W(·) ∈R3×d is a trainable weight and b(·) ∈R3×1 is a trainable bias.
For each potential auxiliary relation type, we just take advantage of Wobj and
bobj to get the object in the context of this relation.
Loss function of three taggers can be calculated by:
lsub
seq = −
npri
pre

j=1
L

i=1
ysub
i,j log(P sub
i,j )
L × npri
pre
,
(9)
lmen
seq
= −
npri
pre

j=1
L

i=1
ypre
i,j log(P pre
i,j )
L × npri
pre
,
(10)
lobj
seq = −
1
2 × L[
npri
pre

j=1
L

i=1
yobj
i,j log(P obj
i,j )
npri
pre
−
naux
pre

j=1
L

i=1
yobj
i,j log(P obj
i,j )
naux
pre
],
(11)
where L is the length of input text. npri
pre and naux
pre represent the size of potential
primary relation type set and that of auxiliary relation type set.
Combination.
Each auxiliary pair will be concatenated with all primary
triplets respectively to generate candidate combinations. To determine whether
a combination is semantically valid, a linear classiﬁer is introduced in this com-
ponent.
It is worth mentioning that each part of the primary triple is an entity com-
posed of multiple tokens, which may hinder parallel computation. Therefore, we
represent entities with an average vector to keep the dimension. The represen-
tation of every combination can be obtained as follows:
hcom
i,j
= [hei; vrj; hej],
(12)
where ei is an entity in one primary triple and ej is the object corresponding to
auxiliary relation type rj. Symbol ; represents the operation of concatenation.
Probability of such combination being valid can be measured by:
P com
i,j
= Sigmoid(Wcomhcom
i,j
+ bcom),
(13)
where Wcom is a trainable weight and bcom is a trainable bias. If P com
i,j
is the
greatest one among all candidate combinations containing the auxiliary pair
(rj, ej), (rj, ej) will be combined with ei into an attributed triple.
408
R. Wang et al.
The loss of combination is similar to that of relation identiﬁcation:
lcom = −
npri

i=1
naux

j=1
yi,j log(P com
i,j ) + (1 −yi,j) log(1 −P com
i,j )
npri × naux

,
(14)
where npri refers to the number of entities in the primary triples and naux refers to the
number of auxiliary pairs in input text.
3.2
Contrastive Learning
Contrastive learning is introduced to enable our model to learn subtle diﬀerences
between samples, which improves its capability of representation when training data is
insuﬃcient. The contrastive loss is calculated based on all candidate combinations in
one batch, where two combinations with same relation type will be taken as positive.
Following previous works [12,13], we deﬁne the loss for a positive pair as:
l(hcomi, hcomj) = −log

e(hcomi ·hcom
j
/τ)
ncom
k=0 1i̸=k · e(hcomi ·hcomk /τ)

,
(15)
where comi is representation of one combination, τ is a temperature parameter, and
ncom is number of candidate combinations in the batch. 1i̸=k is the indicator function.
The ﬁnal loss is computed from all positive pairs in the batch.
3.3
Objective Function
We deﬁne the objective function as a total loss composed of loss from each module.
With Eqs. 3 and 4, loss for relation identiﬁcation is denoted as:
lrel = 1
2(lpri
rel + laux
rel ).
(16)
According to Eqs. 9, 10 and 11, the ﬁnal loss for entity recognition is deﬁned as an
average of three loss values:
lseq = 1
3(lsub
seq + lobj
seq + lmen
seq ).
(17)
The ﬁnal objective function is the total of mentioned loss functions:
l = lrel + lseq + α × lcom + (1 −α) × lcon,
(18)
where 0 ≤α ≤1 is a weight to be tuned during training.
4
Experiment
4.1
Datasets
To evaluate the performance of our model on open data, we choose DuIE, a well-known
dataset for information extraction, as the source of one dataset because some triples
in DuIE contain complex objects. Each of these complex objects provides not only a
object value but also some other slots that can be taken as auxiliary relations. In this
way, we obtain a dataset with 2070 attributed triples in the newly proposed form.
Together with 3 experts, we also manually build a dataset based on information
from ICT corpus. This dataset consists of 171 attributed triples, and it enables us to
explore how our model behaves when applied to real industrial scenario. Details of the
proposed datasets are shown in Table 1.
Attributed Triple Extraction by Combination Under Contrastive Learning
409
Table 1. Details of the proposed datasets.
Dataset Composition
Relation type
DuIE
Primary triple Dub
Play
Award Box Oﬃce -
461
1302
469
118
-
Auxiliary pair inWork onDate inArea -
-
1978
340
118
-
-
ICT
Primary triple Support Set
Send
Provide
-
39
52
44
36
-
Auxiliary pair Be
TO
For
Contain
Through
27
44
60
20
48
4.2
Experiment Settings
Baseline. In the absence of existing work for extracting our proposed complex rela-
tions, we employ a simpliﬁed version of our model as a baseline for comparison. This
simpliﬁed model omits the contrastive learning and combination modules, using a
straightforward nearest-neighbor principle for combination, eﬀective in simpler con-
texts. Additionally, it alters the entity recognition module to train object recognizers
independently for primary and auxiliary relations, without parameter sharing.
Given the proven excellence of large language models in various natural language
processing tasks, we also utilize GPT-3.5-turbo as an additional baseline to assess our
model’s performance.
Metrics. Our evaluation metrics for model performance include Precision, Recall,
and F1 score, encompassing three aspects: primary triples, auxiliary pairs, and
attributed triples. For primary triples and auxiliary pairs, we apply strict match-
ing, deeming results correct only if they exactly match the annotations. In evaluating
attributed triples, we combine strict and partial matching to more accurately gauge
the model’s eﬀectiveness. Equation 19 details the evaluation criteria for each attributed
triple.
Nprecision = 1primary triple · ncorrect
ntotal , Nrecall = 1primary triple ·
ncorrect
nannotation ,
(19)
where 1primary triple equals to 1 only when the extracted primary triple is correct
and ncorrect is the number of auxiliary pairs extracted correctly. ntotal and nannotation
represent the total number of auxiliary pairs extracted and the number of auxiliary
pairs annotated respectively. The ﬁnal Precision and Recall will be the average of
all Nprecision and Nrecall.
Details. Our experiment, conducted on the PyTorch framework, utilized an NVIDIA
RTX 3080Ti GPU and CUDA 11.3. We employed Hugging Face’s bert-chinese-base
as the pre-trained BERT model, featuring a 768-dimension hidden text representation.
The maximum sentence sequence length was set at 100. Batch sizes were 5 for training,
24 for validation, and 64 for testing, with a maximum of 40 training epochs. Hyper-
parameters were ﬁne-tuned on the validation set, setting BERT’s learning rate at 1e-4
with a 0.01 weight decay, and 1e-3 for other model components.
410
R. Wang et al.
Table 2. Precision(%), Recall (%) and F1-score (%) of CCL and baselines.
Dataset Model
Composition Prec. Rec.
F1
DuIE
GPT-3.5-turbo Primary
0.577 0.536 0.556
Auxiliary
0.324 0.302 0.314
Attributed
0.308 0.286 0.297
Simpliﬁed CCL Primary
0.468 0.552 0.507
Auxiliary
0.213 0.235 0.223
Attributed
0.172 0.201 0.185
CCL
Primary
0.708 0.687 0.697
Auxiliary
0.729 0.642 0.683
Attributed
0.646 0.627 0.636
ICT
GPT-3.5-turbo Primary
0.682 0.625 0.652
Auxiliary
0.321 0.290 0.305
Attributed
0.347 0.309 0.327
Simpliﬁed CCL Primary
0.469 0.567 0.514
Auxiliary
0.287 0.292 0.289
Attributed
0.233 0.235 0.234
CCL
Primary
0.917 0.917 0.917
Auxiliary
0.855 0.824 0.839
Attributed
0.810 0.805 0.807
4.3
Results
We evaluate CCL and baselines on two datasets through ten-fold cross-validation and
the overall results are shown in Table 2. CCL performs best in all evaluation metrics
and even outperforms GPT-3.5-turbo by 33.9% and 48.0% in F1-score on two datasets.
We also ﬁnd that all extraction is better performed on ICT dataset. One important
reason is that open dataset is built by means of some ﬁxed mapping rules without
considering semantic correctness, which not only has bad eﬀects on model training but
may also cause wrong judgment on outputs due to incorrect annotations. In addition,
entities in open data is more complicated while most entities in the ICT dataset are
merely names of devices, which reduces diﬃculty in relation extraction.
It’s noteworthy that GPT’s suboptimal performance may stem from the inherent
traits of generative models. We observe GPT often produces semantically valid relations
that fall outside predeﬁned ranges. While this is minor for general reading, it poses
challenges in data structuring and normalization, requiring additional alignment eﬀorts.
Consequently, CCL proves more adept for structuring text data in industrial contexts.
4.4
Ablation Study
To assess the impact of individual modules on extraction performance, we per-
formed ablation studies on the contrastive learning module, combination module, and
parameter-sharing mechanism. Table 3 depicts the performance variations in attributed
Attributed Triple Extraction by Combination Under Contrastive Learning
411
triple extraction on the ICT dataset upon sequential removal of each component.
Results indicate that all these modules positively inﬂuence attributed triple extrac-
tion. Model4 is actually the simpliﬁed CCL.
Table 3. Results of ablation study.
Name
Detail
Prec. Rec.
F1
Model1 CCL
0.810 0.805 0.807
Model2 Model1-Contrastive Learning 0.652 0.583 0.616
Model3 Model2-Combination
0.429 0.402 0.415
Model4 Model3-Parameter Sharing
0.233 0.235 0.234
Eﬀect of Contrastive Learning. The results in the ﬁrst two rows of Table 3
indicate that the contrastive learning module does enhance the model’s capability
of representation with limited training data, leading to an improvement of 19.1% in
F1-score when extracting attributed triples. Extra experiments demonstrate that this
module also enhances CCL’s performance in extracting primary and auxiliary relational
information by 7.8% and 13.1% in F1-score respectively, which is shown in Table 4.
Table 4. Detailed results of ablation on contrastive learning.
Model
Composition
Prec. Rec.
F1
CCL
Primary triple 0.917 0.917 0.917
Auxiliary pair 0.855 0.824 0.839
CCL-Contrastive Learning Primary triple 0.832 0.814 0.823
Auxiliary pair 0.705 0.679 0.692
Eﬀect of Combination. Results in Table 3 show that replacing combination mod-
ule by simple rules like nearest-neighbor criterion brings a decline of 20.1% in F1-score,
which means that the combination module is eﬀective. In fact, we ﬁnd that combination
can be replaced by more complicated rules that contain common combination patterns.
But it takes too much time to manually summarize such rules and rules may fail when
encountering unseen combination patterns. Hence, combination module is eﬃcient in
industrial scenarios, especially when the semantics are complicated.
412
R. Wang et al.
Table 5. Results of entity recognition w/o parameter sharing.
Model
Prec. Rec.
F1
CCL
0.931 0.923 0.927
CCL-Parameter Sharing 0.543 0.521 0.532
Eﬀect of Parameter Sharing. Parameter sharing mechanism is designed to
increase the training of entity recognition module when in lack of training data. Accord-
ing to ablation results, parameter sharing mechanism improves F1-score by 18.1% on
ICT dataset. To further verify its eﬀect on entity recognition, we compare the per-
formance of entity recognition of our model on ICT dataset when with and without
parameter sharing. The result in Table 5 proves the eﬀectiveness of this mechanism.
Fig. 3. Eﬀect of contrastive learning on combination. CL is short for contrastive learn-
ing in legend. We use candidates shown in combination module in Fig. 2.
4.5
Case Study
In order to illustrate the advantages of combination under contrastive learning more
intuitively, we conduct a case study by comparing results of combination module w/o
contrastive learning. Figure 3 shows the visualized eﬀect of contrastive learning on the
probability of each candidate combination being valid. It can be observed that valid
combinations are more likely to be accepted and invalid combinations are more likely
to be denied with contrastive learning. This reﬂects that our model learns a more
reasonable representation space with same training data, which is of great use when
the semantics are puzzling.
Attributed Triple Extraction by Combination Under Contrastive Learning
413
5
Conclusion
In this paper, we presented a novel representation of complex knowledge, termed
attributed triple, and develop a corresponding extraction model. The attributed
triple is designed to intricately detail both the collective and speciﬁc relations among
multiple entities. This extraction model identiﬁes relations and then leverages these
identiﬁed relations to extract relevant entities, thereby generating the primary triple
and auxiliary pairs. It eﬀectively overcomes the limitations imposed by entity-type pre-
deﬁnition. Additionally, a linear combination discriminator is utilized to ascertain the
semantic feasibility of the combinations formed by these primary triples and auxiliary
pairs. To robustly evaluate the model’s proﬁciency in extracting attributed triple data,
we constructed a high-quality dataset derived from ICT corpus. Our model signiﬁcantly
outperforms established baselines across a range of metrics.1
References
1. Al-Moslmi, T., Oca˜na, M., Opdahl, A., et al.: Named entity extraction for knowl-
edge graphs: a literature overview. IEEE Access 8, 32862–32881 (2020)
2. Ma, X.: Knowledge graph construction and application in geosciences: a review.
Comput. Geosci. 161, 105082 (2022)
3. Zheng, H., Wen, R., Chen, X., et al.: PRGC: Potential relation and global cor-
respondence based joint relational triple extraction. In: Proceedings of the 59th
Annual Meeting of the Association for Computational Linguistics (Volume: Long
Papers)1, 6225–6235 (2021)
4. Peng, N., Poon, H., Quirk, C., et al.: Cross-sentence n-ary relation extraction with
graph LSTMs. Trans. Assoc. Comput. Linguist. 5, 101–115 (2017)
5. Song, L., Zhang, Y., Wang, Z., et al.: N-ary relation extraction using graph-state
LSTM. In: Proceedings of the 2018 Conference on Empirical Methods in Natural
Language Processing, pp. 2226–2235 (2018)
6. Guo, Z., Zhang, Y., Lu.W.: Attention guided graph convolutional networks for
relation extraction. In: Proceedings of the 57th Conference of the Association for
Computational Linguistics, Vol.1, Long Papers, pp. 241–251 (2019)
7. Zeng, D., Zhao, C., Jiang, C., et al.: Document-level relation extraction with con-
text guided mention integration and inter-pair reasoning. IEEE ACM Trans. Audio
Speech Lang. Process 31, 3659–3666 (2023)
8. Shang, Y., Huang, H., Sun, X., et al.: Relational triple extraction: one step is
enough. In: Luc De Raedt, editor, In: Proceedings of the Thirty-First International
Joint Conference on Artiﬁcial Intelligence, pp. 4360–4366 (2022)
9. Shang, Y., Huang, H., Mao, X.:. Onerel: joint entity and relation extraction with
one module in one step. In: Thirty-Sixth AAAI Conference on Artiﬁcial Intelli-
gence, pp. 11285–11293 (2022)
10. Wei, Z., Su, J., Wang, Y., et al.: A novel cascade binary tagging framework for
relational triple extraction. In: Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics, pp. 1476–1488 (2020)
1 This work is partially supported by National Nature Science Foundation of China
under No. U21A20488. We thank the Big Data Computing Center of Southeast
University for providing the facility support on the numerical calculations in this
paper.
414
R. Wang et al.
11. Panchendrarajan, R., Amaresan, A.: Bidirectional LSTM-CRF for named entity
recognition. In: Proceedings of the 32nd Paciﬁc Asia Conference on Language,
Information and Computation (2018)
12. Ye, H., Zhang, N., Deng, S., et al.: Contrastive triple extraction with generative
transformer. In: Proceedings of the AAAI Conference on Artiﬁcial Intelligence, vol.
35, 14257–14265 (2021)
13. Chen, T., Kornblith, S., Norouzi, M. et al.: A simple framework for contrastive
learning of visual representations. In: Proceedings of the 37th International Confer-
ence on Machine Learning, vol. 119 of Proceedings of Machine Learning Research,
pp. 1597–1607 (2020)


Question:
The extraction model for attributed triples can be improved from:
A. Accuracy. With a more complicated architecture, the model can provide better attributed triples.
B. Recall. Lower the threshold in relation identification can boost more discovery of potential relations.
C. Efficiency. Design a new module to decrease the number of permutation when searching for candidate combinations.
D. Efficiency. Sample the candidate combinations generated to reduce expenditure on computation.

Answer: BC

Relation Identiﬁcation. Diﬀerent from works [8,9] which extract all candi-
date entities before permutating them to form triples, we ﬁrst obtain subsets of
potential primary relation types and auxiliary relation types respectively, and
only perform entity extraction for these potential relations. Since much more
possible entity combinations is taken into consideration in one attributed triple,
this part will greatly improve the eﬃciency of extraction.

With the output of the encoder Henc = [h1, h2, . . . , hL], probability of one
relation type being potential in text S can be predicted by:
Prel = Sigmoid(Wrelhavg
enc + brel),
(2)
where havg
enc is the average vector among all hi in Henc. Wrel and brel represent
a trainable weight and a trainable bias, respectively. The subset of potential
relations can be obtained as Rpot = {ri|Pri > λ}, where λ is a preset threshold.
According to cross-entropy loss, loss function of this part can be deﬁned as:
lpri
rel = −1
npri
rel
npri
rel

i=1

ypri
i
log(Prel) + (1 −ypri
i
) log(1 −Prel)

,
(3)
laux
rel = −
1
naux
rel
naux
rel

i=1
(yaux
i
log(Prel) + (1 −yaux
i
) log(1 −Prel)) ,
(4)
where npri
rel and naux
rel denote the size of primary and auxiliary relation set.
Related Entity Recognition. Potential primary relation types and auxiliary
relation types obtained will be the input of this part in the form of embedding,
together with that of each token in text S [10]. We can update the representation
of i-th token in the context of potential relation type rj to:
hi,j = hi + vrj,
(5)
where vj ∈Rd×1 is a trainable embedding vector for the j-th relation. Related
entity recognition is based on the new representation through sequence tagging
in the BIO pattern as is shown in Fig. 2.
When it comes to the network in this component, we dismiss the LSTM-CRF
[11], which proves to be eﬀective but may be too complicated to be trained well
in consideration of the insuﬃciency of training data in some industrial domains.
Instead, FCNN is adopted to simplify the network. For each potential primary
relation type, there are three entities to be recognized: subject, object, and rela-
tion. Hence, three rounds of sequence tagging will be performed independently:
P sub
i,j
= Softmax(Wsubhi,j + bsub),
(6)
P obj
i,j = Softmax(Wobjhi,j + bobj),
(7)
P re
i,j = Softmax(Wrehi,j + bre),
(8)
where W(·) ∈R3×d is a trainable weight and b(·) ∈R3×1 is a trainable bias.
For each potential auxiliary relation type, we just take advantage of Wobj and
bobj to get the object in the context of this relation.
Combination.
Each auxiliary pair will be concatenated with all primary
triplets respectively to generate candidate combinations. To determine whether
a combination is semantically valid, a linear classiﬁer is introduced in this com-
ponent.

It is worth mentioning that each part of the primary triple is an entity com-
posed of multiple tokens, which may hinder parallel computation. Therefore, we
represent entities with an average vector to keep the dimension. The represen-
tation of every combination can be obtained as follows:
hcom
i,j
= [hei; vrj; hej],
(12)
where ei is an entity in one primary triple and ej is the object corresponding to
auxiliary relation type rj. Symbol ; represents the operation of concatenation.
Probability of such combination being valid can be measured by:
P com
i,j
= Sigmoid(Wcomhcom
i,j
+ bcom),
(13)
where Wcom is a trainable weight and bcom is a trainable bias. If P com
i,j
is the
greatest one among all candidate combinations containing the auxiliary pair
(rj, ej), (rj, ej) will be combined with ei into an attributed triple.

The loss of combination is similar to that of relation identiﬁcation:
lcom = −
npri

i=1
naux

j=1
yi,j log(P com
i,j ) + (1 −yi,j) log(1 −P com
i,j )
npri × naux

,
(14)
where npri refers to the number of entities in the primary triples and naux refers to the
number of auxiliary pairs in input text.

Attributed Triple Extraction
by Combination Under
Contrastive Learning
Runzhe Wang1,2, Gaofeng Xu1,2, Guilin Qi1,2(B), Yun Li3, Yongrui Chen1,2,
Songlin Zhai1,2, Yuxin Zhang1,2, Rihui Jin1,2, Nijun Li3, and Qianren Wang3
1 School of Computer Science and Engineering, Southeast University,
Nanjing 211189, China
{220232335,230229361,gqi,yrchen,songlin zhai,zzyx cs,220212061}@seu.edu.cn
2 Key Laboratory of New Generation Artiﬁcial Intelligence Technology and its
Interdisplinary Applications (Southeast University), Ministry of Education,
Beijing, China
3 Advanced Cognitive AI Lab, Huawei Technologies, Shenzhen, China
{liyun5,linijun,wangqianran}@huawei.com
Abstract. Complex knowledge extraction is a signiﬁcant topic of
research. Previous methodologies were limited to extracting only the
overall relations among multiple entities and they necessitated the prede-
ﬁned entity types, rendering them inapplicable in industrial domains such
as Information and Communication Technology (ICT), where numerous
entities are undeﬁned. To address these challenges, this paper introduces
a novel representation for complex knowledge: attributed triple, which
consists of a primary triple coupled with auxiliary attribute-value pairs,
enabling a ﬁne-grained depiction of both the overall inter-entity relations
and the pairwise relations between entities. Based on this representation,
we propose an attributed triple extraction model. This model employs
multi-label classiﬁers to identify all potential relations before extract-
ing relevant entities with sequence taggers based on the BIO pattern,
where B represents Begin, I representsInside and O represents Outside.
This process makes the extraction of primary triples and auxiliary pairs
without predeﬁning entity types. Subsequently, a linear combination dis-
criminator is designed to assess the semantic feasibility of candidate com-
binations formed by concatenating primary triplets and auxiliary pairs.
Furthermore, contrastive learning is adopted to enhance the model’s abil-
ity of representation with insuﬃcient training data. To better evaluate
the model’s performance over extracting attributed triple, we construct a
high-quality dataset based on data from ICT corpus. Our model demon-
strates substantial and consistent superiority over baselines across vari-
ous metrics. Our code is publicly available. https://github.com/sid0527/
Attributed-Triple-Extraction
Keywords: Attributed Triple · Information Extraction · Contrastive
Learning
The author contributed equally to this work.
c
⃝The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024
M. Onizuka et al. (Eds.): DASFAA 2024, LNCS 14856, pp. 402–414, 2024.
https://doi.org/10.1007/978-981-97-5575-2_30
Attributed Triple Extraction by Combination Under Contrastive Learning
403
Fig. 1. The diﬀerences in the various representations of complex data. In the ﬁgure, the
red font indicates entities, green indicates entity types and yellow indicates relations.
(Color ﬁgure online)
1
Introduction
Information extraction, an essential process in the construction and comple-
tion of knowledge graphs [1,2], facilitates the automated retrieval of structured
information from voluminous text. Previous works mainly focused on binary
relational information extraction, which is also called triple extraction [3]. How-
ever, in numerous industrial scenarios, the text encapsulates complex knowledge
characterized by interactions among multiple entities. The extraction of such
intricate knowledge is signiﬁcant as it can enrich knowledge graphs with more
comprehensive and detailed entity-relation information.
Current studies on complex knowledge extraction lie primarily within the
biomedical domain, which involves distilling an overarching relation among spe-
ciﬁc categories of entities [4–7]. These methods necessitate the predeﬁnition of
entity types prior to extraction, as depicted in Fig. 1. However, their application
in other industrial ﬁelds, such as Information and Communication Technology
(ICT), can lead to two major challenges. 1) Beyond the overall relations among
entities, individual relations between entities also matter. For instance, UE
device sends Service Request message to AN device, and the message
contains Service type, where UE and AN are undeﬁned entities in ICT. Here,
the relation between UE device and Service Request message is deﬁned
as send, while the relation between Service Request message and Service
type is contain and the collective relation of the entities (UE device, Service
Request message, AN device) is (send. . . to. . . ). Representing these entities
using a single overarching relation can compromise information integrity. 2) The
diversity and continual emergence of new entity types in textual data make the
provision of entity-type predeﬁnition impractical.
Intuitively, the ideal representation would involve a more granular form of
complex knowledge, capable of articulating both the pairwise and overall rela-
tions among multiple entities. Consequently, we distill complex text into primary
components and modiﬁers, achieving a ﬁne-grained representation of complex
knowledge. Moreover, we observe that the range of relations in most extraction
404
R. Wang et al.
datasets is considerably narrower than the variety of entity types. An intuitive
solution is to identify all relevant entity types within the text by recognizing
relations, thus circumventing the need of entity-type predeﬁnition.
In this paper, we introduce a novel representational form for complex knowl-
edge: attributed triple, comprising a primary triple allied with auxiliary
attribute-value pairs. The primary triple delineates the central subject-predicate-
object structure of the text, while the auxiliary pairs provide additional con-
textual details. The instance mentioned above can be represented as (UE
device, send:{to, AN device}, Service Request message:{contain, Ser-
vice type}), where (UE device, send, Service Request message) consti-
tutes the primary triple, and {to, AN device}, {contain, Service type} are
auxiliary pairs. Based on this representation, we propose an attributed triple
extraction model. This model employs multi-label classiﬁers to identify poten-
tial relations, followed by entity extraction through a sequence tagger based on
the BIO pattern. This process yields the primary triple and auxiliary pairs with-
out predeﬁning entity types. A linear combination discriminator is then used to
evaluate the semantic feasibility of candidate combinations, formed by concate-
nating primary triplets and auxiliary pairs. To reduce the eﬀect of industrial
data scarcity on model’s ability of representation, contrastive learning is inte-
grated during the training phase. Moreover, to assess the model’s eﬀectiveness in
extracting attributed triples, we have developed a high-quality dataset derived
from the ICT domain. This dataset, featuring a variety of primary and auxiliary
relations with diverse combinations, more accurately mirrors the complexities
of real-world industrial scenarios, in contrast to existing data. Extensive exper-
imental results demonstrate that our model can eﬀectively extract high-quality
attributed triples, proving to be of signiﬁcant value in practical applications. In
summary, the contributions of this paper are threefold:
1. We introduce a new representational form for multi-entity information:
attributed triple, which is capable of depicting both the pairwise and over-
all relations among entities, thereby enabling ﬁne-grained representation of
information among multiple entities.
2. We have developed an attributed triple extraction model that prioritizes rela-
tion dentiﬁcation over entity recognition, negating the need for predeﬁning
entity type. This model integrates contrastive learning and employs a shared
loss function during training, enhancing the accuracy of the model outputs
and substantially improving its practicality.
3. To thoroughly evaluate the model’s capacity for extracting attributed triples,
we semi-automatically curated a high-quality dataset based on the ICT
domain. This dataset comprises a diverse array of primary and auxiliary rela-
tions, better mirroring actual industry scenarios compared to open datasets.
Extensive experiments on both open and industrial datasets demonstrate that
our model achieves excellent performance across a range of metrics.
Attributed Triple Extraction by Combination Under Contrastive Learning
405
Fig. 2. An overview of Combination under Contrastive Learning. Both relation iden-
tiﬁcation and related entity recognition module are designed to be composed of two
similar components, which beneﬁts the separate extraction of primary triples and aux-
iliary relation-entity pairs. To address insuﬃciency of training data, parameter sharing
mechanism and contrastive learning are introduced.
2
Preliminaries
2.1
Attributed Triple
An attributed triple F is a primary subject-relation-object triple (s, r, o) cou-
pled with m auxiliary pairs {(ai, vi)}m
i=1, where r, a1, . . . , am ∈R and s, o, v1,
. . . , vm
∈
E, with R and E
being the sets of relations and entities,
respectively. We denote the attributed triple as F
=
(s : {(ai, vi)}ms
i=1 ,
r : {(ai, vi)}mr
i=1 , o : {(ai, vi)}mo
i=1), where ms + mr + m0 = m and x : y denotes
y attribute to x. We slightly abuse terminology here by referring to the primary
relation and auxiliary attribute as relations, and referring to the subject, object,
and values as entities unless otherwise speciﬁed.
2.2
Problem Formulation
Our goal is to extract attributed triples from text. Given the input text T =
{t1, t2, . . . , tn}, we aim to extract G(T ) →F, where G denotes the attributed
triples extraction model.
3
Combination Under Contrastive Learning
This section presents CCL, our model to extract attributed triples by combi-
nation under contrastive learning. There are two key modules in our model:
attributed triple combination and contrastive learning. The former combines
attributed triples by Relation Identiﬁcation, Related Entity Recognition
and Combination, and the latter designs a loss to address data scarcity. An
overview illustration of our model is shown in Fig. 2.
406
R. Wang et al.
3.1
Attributed Triple Combination
A pre-trained BERT is employed as encoder to transform the input text S into
the d-dimensional contextual representation:
[h1, h2, . . . , hL] = Encoder(S),
(1)
where hi corresponds to the i-th input token and L is the length of S.
Relation Identiﬁcation. Diﬀerent from works [8,9] which extract all candi-
date entities before permutating them to form triples, we ﬁrst obtain subsets of
potential primary relation types and auxiliary relation types respectively, and
only perform entity extraction for these potential relations. Since much more
possible entity combinations is taken into consideration in one attributed triple,
this part will greatly improve the eﬃciency of extraction.
With the output of the encoder Henc = [h1, h2, . . . , hL], probability of one
relation type being potential in text S can be predicted by:
Prel = Sigmoid(Wrelhavg
enc + brel),
(2)
where havg
enc is the average vector among all hi in Henc. Wrel and brel represent
a trainable weight and a trainable bias, respectively. The subset of potential
relations can be obtained as Rpot = {ri|Pri > λ}, where λ is a preset threshold.
According to cross-entropy loss, loss function of this part can be deﬁned as:
lpri
rel = −1
npri
rel
npri
rel

i=1

ypri
i
log(Prel) + (1 −ypri
i
) log(1 −Prel)

,
(3)
laux
rel = −
1
naux
rel
naux
rel

i=1
(yaux
i
log(Prel) + (1 −yaux
i
) log(1 −Prel)) ,
(4)
where npri
rel and naux
rel denote the size of primary and auxiliary relation set.
Related Entity Recognition. Potential primary relation types and auxiliary
relation types obtained will be the input of this part in the form of embedding,
together with that of each token in text S [10]. We can update the representation
of i-th token in the context of potential relation type rj to:
hi,j = hi + vrj,
(5)
where vj ∈Rd×1 is a trainable embedding vector for the j-th relation. Related
entity recognition is based on the new representation through sequence tagging
in the BIO pattern as is shown in Fig. 2.
When it comes to the network in this component, we dismiss the LSTM-CRF
[11], which proves to be eﬀective but may be too complicated to be trained well
in consideration of the insuﬃciency of training data in some industrial domains.
Attributed Triple Extraction by Combination Under Contrastive Learning
407
Instead, FCNN is adopted to simplify the network. For each potential primary
relation type, there are three entities to be recognized: subject, object, and rela-
tion. Hence, three rounds of sequence tagging will be performed independently:
P sub
i,j
= Softmax(Wsubhi,j + bsub),
(6)
P obj
i,j = Softmax(Wobjhi,j + bobj),
(7)
P re
i,j = Softmax(Wrehi,j + bre),
(8)
where W(·) ∈R3×d is a trainable weight and b(·) ∈R3×1 is a trainable bias.
For each potential auxiliary relation type, we just take advantage of Wobj and
bobj to get the object in the context of this relation.
Loss function of three taggers can be calculated by:
lsub
seq = −
npri
pre

j=1
L

i=1
ysub
i,j log(P sub
i,j )
L × npri
pre
,
(9)
lmen
seq
= −
npri
pre

j=1
L

i=1
ypre
i,j log(P pre
i,j )
L × npri
pre
,
(10)
lobj
seq = −
1
2 × L[
npri
pre

j=1
L

i=1
yobj
i,j log(P obj
i,j )
npri
pre
−
naux
pre

j=1
L

i=1
yobj
i,j log(P obj
i,j )
naux
pre
],
(11)
where L is the length of input text. npri
pre and naux
pre represent the size of potential
primary relation type set and that of auxiliary relation type set.
Combination.
Each auxiliary pair will be concatenated with all primary
triplets respectively to generate candidate combinations. To determine whether
a combination is semantically valid, a linear classiﬁer is introduced in this com-
ponent.
It is worth mentioning that each part of the primary triple is an entity com-
posed of multiple tokens, which may hinder parallel computation. Therefore, we
represent entities with an average vector to keep the dimension. The represen-
tation of every combination can be obtained as follows:
hcom
i,j
= [hei; vrj; hej],
(12)
where ei is an entity in one primary triple and ej is the object corresponding to
auxiliary relation type rj. Symbol ; represents the operation of concatenation.
Probability of such combination being valid can be measured by:
P com
i,j
= Sigmoid(Wcomhcom
i,j
+ bcom),
(13)
where Wcom is a trainable weight and bcom is a trainable bias. If P com
i,j
is the
greatest one among all candidate combinations containing the auxiliary pair
(rj, ej), (rj, ej) will be combined with ei into an attributed triple.
408
R. Wang et al.
The loss of combination is similar to that of relation identiﬁcation:
lcom = −
npri

i=1
naux

j=1
yi,j log(P com
i,j ) + (1 −yi,j) log(1 −P com
i,j )
npri × naux

,
(14)
where npri refers to the number of entities in the primary triples and naux refers to the
number of auxiliary pairs in input text.
3.2
Contrastive Learning
Contrastive learning is introduced to enable our model to learn subtle diﬀerences
between samples, which improves its capability of representation when training data is
insuﬃcient. The contrastive loss is calculated based on all candidate combinations in
one batch, where two combinations with same relation type will be taken as positive.
Following previous works [12,13], we deﬁne the loss for a positive pair as:
l(hcomi, hcomj) = −log

e(hcomi ·hcom
j
/τ)
ncom
k=0 1i̸=k · e(hcomi ·hcomk /τ)

,
(15)
where comi is representation of one combination, τ is a temperature parameter, and
ncom is number of candidate combinations in the batch. 1i̸=k is the indicator function.
The ﬁnal loss is computed from all positive pairs in the batch.
3.3
Objective Function
We deﬁne the objective function as a total loss composed of loss from each module.
With Eqs. 3 and 4, loss for relation identiﬁcation is denoted as:
lrel = 1
2(lpri
rel + laux
rel ).
(16)
According to Eqs. 9, 10 and 11, the ﬁnal loss for entity recognition is deﬁned as an
average of three loss values:
lseq = 1
3(lsub
seq + lobj
seq + lmen
seq ).
(17)
The ﬁnal objective function is the total of mentioned loss functions:
l = lrel + lseq + α × lcom + (1 −α) × lcon,
(18)
where 0 ≤α ≤1 is a weight to be tuned during training.
4
Experiment
4.1
Datasets
To evaluate the performance of our model on open data, we choose DuIE, a well-known
dataset for information extraction, as the source of one dataset because some triples
in DuIE contain complex objects. Each of these complex objects provides not only a
object value but also some other slots that can be taken as auxiliary relations. In this
way, we obtain a dataset with 2070 attributed triples in the newly proposed form.
Together with 3 experts, we also manually build a dataset based on information
from ICT corpus. This dataset consists of 171 attributed triples, and it enables us to
explore how our model behaves when applied to real industrial scenario. Details of the
proposed datasets are shown in Table 1.
Attributed Triple Extraction by Combination Under Contrastive Learning
409
Table 1. Details of the proposed datasets.
Dataset Composition
Relation type
DuIE
Primary triple Dub
Play
Award Box Oﬃce -
461
1302
469
118
-
Auxiliary pair inWork onDate inArea -
-
1978
340
118
-
-
ICT
Primary triple Support Set
Send
Provide
-
39
52
44
36
-
Auxiliary pair Be
TO
For
Contain
Through
27
44
60
20
48
4.2
Experiment Settings
Baseline. In the absence of existing work for extracting our proposed complex rela-
tions, we employ a simpliﬁed version of our model as a baseline for comparison. This
simpliﬁed model omits the contrastive learning and combination modules, using a
straightforward nearest-neighbor principle for combination, eﬀective in simpler con-
texts. Additionally, it alters the entity recognition module to train object recognizers
independently for primary and auxiliary relations, without parameter sharing.
Given the proven excellence of large language models in various natural language
processing tasks, we also utilize GPT-3.5-turbo as an additional baseline to assess our
model’s performance.
Metrics. Our evaluation metrics for model performance include Precision, Recall,
and F1 score, encompassing three aspects: primary triples, auxiliary pairs, and
attributed triples. For primary triples and auxiliary pairs, we apply strict match-
ing, deeming results correct only if they exactly match the annotations. In evaluating
attributed triples, we combine strict and partial matching to more accurately gauge
the model’s eﬀectiveness. Equation 19 details the evaluation criteria for each attributed
triple.
Nprecision = 1primary triple · ncorrect
ntotal , Nrecall = 1primary triple ·
ncorrect
nannotation ,
(19)
where 1primary triple equals to 1 only when the extracted primary triple is correct
and ncorrect is the number of auxiliary pairs extracted correctly. ntotal and nannotation
represent the total number of auxiliary pairs extracted and the number of auxiliary
pairs annotated respectively. The ﬁnal Precision and Recall will be the average of
all Nprecision and Nrecall.
Details. Our experiment, conducted on the PyTorch framework, utilized an NVIDIA
RTX 3080Ti GPU and CUDA 11.3. We employed Hugging Face’s bert-chinese-base
as the pre-trained BERT model, featuring a 768-dimension hidden text representation.
The maximum sentence sequence length was set at 100. Batch sizes were 5 for training,
24 for validation, and 64 for testing, with a maximum of 40 training epochs. Hyper-
parameters were ﬁne-tuned on the validation set, setting BERT’s learning rate at 1e-4
with a 0.01 weight decay, and 1e-3 for other model components.
410
R. Wang et al.
Table 2. Precision(%), Recall (%) and F1-score (%) of CCL and baselines.
Dataset Model
Composition Prec. Rec.
F1
DuIE
GPT-3.5-turbo Primary
0.577 0.536 0.556
Auxiliary
0.324 0.302 0.314
Attributed
0.308 0.286 0.297
Simpliﬁed CCL Primary
0.468 0.552 0.507
Auxiliary
0.213 0.235 0.223
Attributed
0.172 0.201 0.185
CCL
Primary
0.708 0.687 0.697
Auxiliary
0.729 0.642 0.683
Attributed
0.646 0.627 0.636
ICT
GPT-3.5-turbo Primary
0.682 0.625 0.652
Auxiliary
0.321 0.290 0.305
Attributed
0.347 0.309 0.327
Simpliﬁed CCL Primary
0.469 0.567 0.514
Auxiliary
0.287 0.292 0.289
Attributed
0.233 0.235 0.234
CCL
Primary
0.917 0.917 0.917
Auxiliary
0.855 0.824 0.839
Attributed
0.810 0.805 0.807
4.3
Results
We evaluate CCL and baselines on two datasets through ten-fold cross-validation and
the overall results are shown in Table 2. CCL performs best in all evaluation metrics
and even outperforms GPT-3.5-turbo by 33.9% and 48.0% in F1-score on two datasets.
We also ﬁnd that all extraction is better performed on ICT dataset. One important
reason is that open dataset is built by means of some ﬁxed mapping rules without
considering semantic correctness, which not only has bad eﬀects on model training but
may also cause wrong judgment on outputs due to incorrect annotations. In addition,
entities in open data is more complicated while most entities in the ICT dataset are
merely names of devices, which reduces diﬃculty in relation extraction.
It’s noteworthy that GPT’s suboptimal performance may stem from the inherent
traits of generative models. We observe GPT often produces semantically valid relations
that fall outside predeﬁned ranges. While this is minor for general reading, it poses
challenges in data structuring and normalization, requiring additional alignment eﬀorts.
Consequently, CCL proves more adept for structuring text data in industrial contexts.
4.4
Ablation Study
To assess the impact of individual modules on extraction performance, we per-
formed ablation studies on the contrastive learning module, combination module, and
parameter-sharing mechanism. Table 3 depicts the performance variations in attributed
Attributed Triple Extraction by Combination Under Contrastive Learning
411
triple extraction on the ICT dataset upon sequential removal of each component.
Results indicate that all these modules positively inﬂuence attributed triple extrac-
tion. Model4 is actually the simpliﬁed CCL.
Table 3. Results of ablation study.
Name
Detail
Prec. Rec.
F1
Model1 CCL
0.810 0.805 0.807
Model2 Model1-Contrastive Learning 0.652 0.583 0.616
Model3 Model2-Combination
0.429 0.402 0.415
Model4 Model3-Parameter Sharing
0.233 0.235 0.234
Eﬀect of Contrastive Learning. The results in the ﬁrst two rows of Table 3
indicate that the contrastive learning module does enhance the model’s capability
of representation with limited training data, leading to an improvement of 19.1% in
F1-score when extracting attributed triples. Extra experiments demonstrate that this
module also enhances CCL’s performance in extracting primary and auxiliary relational
information by 7.8% and 13.1% in F1-score respectively, which is shown in Table 4.
Table 4. Detailed results of ablation on contrastive learning.
Model
Composition
Prec. Rec.
F1
CCL
Primary triple 0.917 0.917 0.917
Auxiliary pair 0.855 0.824 0.839
CCL-Contrastive Learning Primary triple 0.832 0.814 0.823
Auxiliary pair 0.705 0.679 0.692
Eﬀect of Combination. Results in Table 3 show that replacing combination mod-
ule by simple rules like nearest-neighbor criterion brings a decline of 20.1% in F1-score,
which means that the combination module is eﬀective. In fact, we ﬁnd that combination
can be replaced by more complicated rules that contain common combination patterns.
But it takes too much time to manually summarize such rules and rules may fail when
encountering unseen combination patterns. Hence, combination module is eﬃcient in
industrial scenarios, especially when the semantics are complicated.
412
R. Wang et al.
Table 5. Results of entity recognition w/o parameter sharing.
Model
Prec. Rec.
F1
CCL
0.931 0.923 0.927
CCL-Parameter Sharing 0.543 0.521 0.532
Eﬀect of Parameter Sharing. Parameter sharing mechanism is designed to
increase the training of entity recognition module when in lack of training data. Accord-
ing to ablation results, parameter sharing mechanism improves F1-score by 18.1% on
ICT dataset. To further verify its eﬀect on entity recognition, we compare the per-
formance of entity recognition of our model on ICT dataset when with and without
parameter sharing. The result in Table 5 proves the eﬀectiveness of this mechanism.
Fig. 3. Eﬀect of contrastive learning on combination. CL is short for contrastive learn-
ing in legend. We use candidates shown in combination module in Fig. 2.
4.5
Case Study
In order to illustrate the advantages of combination under contrastive learning more
intuitively, we conduct a case study by comparing results of combination module w/o
contrastive learning. Figure 3 shows the visualized eﬀect of contrastive learning on the
probability of each candidate combination being valid. It can be observed that valid
combinations are more likely to be accepted and invalid combinations are more likely
to be denied with contrastive learning. This reﬂects that our model learns a more
reasonable representation space with same training data, which is of great use when
the semantics are puzzling.
Attributed Triple Extraction by Combination Under Contrastive Learning
413
5
Conclusion
In this paper, we presented a novel representation of complex knowledge, termed
attributed triple, and develop a corresponding extraction model. The attributed
triple is designed to intricately detail both the collective and speciﬁc relations among
multiple entities. This extraction model identiﬁes relations and then leverages these
identiﬁed relations to extract relevant entities, thereby generating the primary triple
and auxiliary pairs. It eﬀectively overcomes the limitations imposed by entity-type pre-
deﬁnition. Additionally, a linear combination discriminator is utilized to ascertain the
semantic feasibility of the combinations formed by these primary triples and auxiliary
pairs. To robustly evaluate the model’s proﬁciency in extracting attributed triple data,
we constructed a high-quality dataset derived from ICT corpus. Our model signiﬁcantly
outperforms established baselines across a range of metrics.1
References
1. Al-Moslmi, T., Oca˜na, M., Opdahl, A., et al.: Named entity extraction for knowl-
edge graphs: a literature overview. IEEE Access 8, 32862–32881 (2020)
2. Ma, X.: Knowledge graph construction and application in geosciences: a review.
Comput. Geosci. 161, 105082 (2022)
3. Zheng, H., Wen, R., Chen, X., et al.: PRGC: Potential relation and global cor-
respondence based joint relational triple extraction. In: Proceedings of the 59th
Annual Meeting of the Association for Computational Linguistics (Volume: Long
Papers)1, 6225–6235 (2021)
4. Peng, N., Poon, H., Quirk, C., et al.: Cross-sentence n-ary relation extraction with
graph LSTMs. Trans. Assoc. Comput. Linguist. 5, 101–115 (2017)
5. Song, L., Zhang, Y., Wang, Z., et al.: N-ary relation extraction using graph-state
LSTM. In: Proceedings of the 2018 Conference on Empirical Methods in Natural
Language Processing, pp. 2226–2235 (2018)
6. Guo, Z., Zhang, Y., Lu.W.: Attention guided graph convolutional networks for
relation extraction. In: Proceedings of the 57th Conference of the Association for
Computational Linguistics, Vol.1, Long Papers, pp. 241–251 (2019)
7. Zeng, D., Zhao, C., Jiang, C., et al.: Document-level relation extraction with con-
text guided mention integration and inter-pair reasoning. IEEE ACM Trans. Audio
Speech Lang. Process 31, 3659–3666 (2023)
8. Shang, Y., Huang, H., Sun, X., et al.: Relational triple extraction: one step is
enough. In: Luc De Raedt, editor, In: Proceedings of the Thirty-First International
Joint Conference on Artiﬁcial Intelligence, pp. 4360–4366 (2022)
9. Shang, Y., Huang, H., Mao, X.:. Onerel: joint entity and relation extraction with
one module in one step. In: Thirty-Sixth AAAI Conference on Artiﬁcial Intelli-
gence, pp. 11285–11293 (2022)
10. Wei, Z., Su, J., Wang, Y., et al.: A novel cascade binary tagging framework for
relational triple extraction. In: Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics, pp. 1476–1488 (2020)
1 This work is partially supported by National Nature Science Foundation of China
under No. U21A20488. We thank the Big Data Computing Center of Southeast
University for providing the facility support on the numerical calculations in this
paper.
414
R. Wang et al.
11. Panchendrarajan, R., Amaresan, A.: Bidirectional LSTM-CRF for named entity
recognition. In: Proceedings of the 32nd Paciﬁc Asia Conference on Language,
Information and Computation (2018)
12. Ye, H., Zhang, N., Deng, S., et al.: Contrastive triple extraction with generative
transformer. In: Proceedings of the AAAI Conference on Artiﬁcial Intelligence, vol.
35, 14257–14265 (2021)
13. Chen, T., Kornblith, S., Norouzi, M. et al.: A simple framework for contrastive
learning of visual representations. In: Proceedings of the 37th International Confer-
ence on Machine Learning, vol. 119 of Proceedings of Machine Learning Research,
pp. 1597–1607 (2020)