Question:
Which components of the proposed joint model contribute to its ability to capture word dependencies and extract multiple relations for an entity?
A. Integration of self-attention mechanism to learn intra-sentence dependencies without relying on distance between words.
B. Utilization of BiLSTM layers to encode contextual information from both past and future token sequences.
C. Application of multi-head selection in the sigmoid layer to simultaneously identify all possible semantic relations for a specific entity.
D. Incorporation of label embedding to convey entity type information from the NER task to the relation extraction task.

Answer: AC

In this section, we explain our model, as shown in Figure 1, in detail. We develop a joint model
based on BiLSTM to (1) identify the boundary and the type of the medical entity and (2) extract
semantic relations among entities. The model is divided into five layers from bottom to top: (1) an
embedding layer, (2) a self-attention layer, (3) a BiLSTM layer, (4) a NER layer, and (5) a sigmoid
layer.
The input to the model is a sequence of tokens, which are then converted into the correspond-
ing embedding vectors. The self-attention and BiLSTM layers are used to extract deep contextual
information. Afterward, the NER and sigmoid layers perform the NER and RE tasks, respectively.
The output for each input token has two parts: (1) a recognized entity label, e.g., I-drug, which
denotes the token inside a named entity of drug, and (2) a set of tuples comprising the head tokens
of the entity and the types of relations between them, e.g., {( (bacteria), sensitive)}. An entity
often contains more than one token, and we only consider the last token of the entity for multi-
head selection. For example, there is a sensitive relation between the entities “ (penicillin)”
and “ (bacteria).” Instead of connecting all tokens of the entities, we only link “” with “”.
Moreover, if there is no semantic relation between the two entities, then we assign the label “N”
and regard the token itself as the head.

4.2
Self-attention Layer
Self-attention is a specific case of the attention mechanism that only requires a single sequence
to compute its representation. Following Vaswani et al. [4], we use multi-head attention based on
self-attention in this article. Figure 2 depicts the computation graph of the multi-head attention
mechanism.
Given a matrix of n query vectors Q ∈Rn∗d, keys K ∈Rn∗d, and values V ∈Rn∗d, the scaled
dot-product attention computes the attention scores with the following formula (1):
Attention (Q,K,V ) = softmax
QKT
√
d

V,
(1)
where Q = K = V in the self-attention layer.
For multi-head attention, X is converted to (Q,K,V ) through different linear elements. Then, h
parallel heads are employed to focus on a different part of channels of value vectors. We denote the
learned linear maps as W Q
i
∈Rn∗d/h, W K
i
∈Rn∗d/h,W V
i
∈Rn∗d/h, which correspond to queries,
keys, and values, respectively. For the ith head, the scaled dot-product attention is calculated, and
the mathematical formulation is shown as follows:
Mi = Attention

QW Q
i ,KW K
i ,VW V
i

.
(2)
Finally, all vectors produced by parallel heads are concatenated together to form a single vector.
A linear map is used to mix different channels from different heads:
M = Concat (M1, M2, . . . , Mh) ,
(3)
Y = MW ,
(4)
where M ∈Rn∗d,W ∈Rd∗d, andY. is the output of self-attention layer and also the input of BiLSTM
layer.

4.3
BiLSTM Layer
LSTM has the ability to well capture long-term dependencies and its extension BiLSTM can make
use of the contextual information in both past and future. Therefore, BiLSTM models have achieved
excellent results in many NLP tasks. In this work, we adopt a BiLSTM to encode the text of medicine
Chinese instruction. We empirically set the number of LSTM layers to 3. For each vector yi of Y,
we concatenate the forward output (−→
hi) and the backward output (←−
hi) at the timestep i. The entire
output of BiLSTM at the timestep i is then as follows:
hi =
−→
hi

,
←−
hi

;i = 0, 1, . . .n..
(5)

4.4
NER Layer
Similarly to Miwa and Bansal [27], Zheng et al. [32], and Katiyar and Cardie [29], we formulate
the NER task as a sequence labeling problem. In sequence labeling, an entity usually contains
more than one token and one tag is assigned to each token in the entity. We adopt the BIO tagging
scheme, i.e., using B, I, and O to denote the beginning, inside, and outside of the entity, respectively.
We annotate the first character of an entity as B-type, the other tokens of the entity as I-type, and
those non-entity tokens as O. With the BIO tagging scheme, we can identify the boundary and the
type of entity simultaneously. In Figure 1, the Chinese token “” has been tagged as B-drug, “”
and “” are tagged as I-drug, and “” is tagged as O. The scores of the tags are calculated by the
following formula:
s(e)(hi) = V (e) f (W (e)hi + b(e))V ∈Rl∗n (entity),W ∈Rl∗2d,b ∈Rl,
(6)
where e is the symbol denoting NER, f (·)is the activation function such as relu or tanh, d is the
number of LSTM hidden units, n(entity) is the number of NER tags, and l is the length of input w.
In the NER task, the BIO labeling scheme faces several restrictions. For example, B-drug can-
not be followed by I-disease. In the softmax, the tags assigned to the tokens according to their
probabilities are independent of each other, although BiLSTM is able to learn the contextual in-
formation. CRF has a transition characteristic that takes the order among tags into consideration.
Therefore, we use a linear-chain CRF (Lample et al. [34]) for the NER task. For input token wi, the
score sequence can be expressed as follows:
S

y1(e), . . . , yn (e)
=
n
i=0 s(e)
i,yi (e) +
n−1
i=1 Ti,yi (e),
(7)
where s(e)
i,yi (e) represents the tag score when the tag of wi is yi, T denotes the transition score from
tagyi toyi+1, andT ∈R(p+2,p+2). Then, the probability of a given sequence of tags over all possible
tag sequences for the input sentence w can be defined by the following formula (7):
Pr

y(e)
1 , . . . y(e)
n |w

=
es (yi (e), ...,y(e)
n
)

˜y(e)
1
... ˜y(e)
n es
y( ˜e)
l , . . . , ˜y(e)
n
 ,
(8)
where we adopt Viterbi to find the tag sequence ˜y(e)
i , which has the highest score.
Entity tags are then fed into the RE task as the corresponding label embedding, considering that
the types of entities involved benefit relation prediction. Our model takes the label embedding
as a parameter layer and learns it in the training phase. The input of the sigmoid layer is the
concatenation of the BiLSTM state hi and the label embedding дi of xi,
zi = (hi,дi).
(9)

4.5
Sigmoid Layer
Inspired by the work of Bekoulis et al. [2], we regard the RE task as a multi-head selection problem,
which can successfully solve the problem with most existing joint models that cannot recognize
all semantic relations for a specific entity in a sentence. For token wi, it may hold any type of
semantic relation rk with other tokenwj. In this article, we make the assumption that the semantic
relations between one entity and others are independent.
Given a sequence of tokens w and a set of relation labels R, the goal of our model is to identify the
vector of the most probable heads wj and the vector of the most probable corresponding relation
label rk for each token wi. We calculate the score between tokens wi and wj given a label rk as
follows:
s(r )(wi,wj,rk) = V r f

W (r )
1 zi +W (r )
2 zj + b(r )
,V ∈Rl∗n(relation),W ∈Rl∗2d+e,b ∈Rl,
(10)
where r, V r, d, e, and l represent the symbol of RE, the weight matrix, the number of LSTM hid-
den units, the dimension of label embedding, and the length of input w, respectively. f (·) and
n(relation) denote the activation function, relu or tanh, and the number of RE tags. The probabil-
ity of the token wj to be selected as the head of the token wi with the relation label rk between
them is defined as
Pr(head = wj,label = rk |wi) σ

s(r )
wi,wj,rk

,
(11)
where σ is the sigmoid function. Unlike softmax, sigmoid assumes that all relations are indepen-
dent of each other, and it does not add up all probabilities of relations to 1. If the value of Pr is
more than 0.5, then sigmoid will determine whether there is a semantic relation between the two
entities; otherwise, no relation exists, and it marks “N.” The goal of sigmoid is to minimize the
cross-entropy loss Lre. during the training phase,
Lre =
n

i
m

j
−log Pr(head = yi,j,re = ri,j |wi).
(12)
For the joint entity and relation extraction task, we calculate the final objective L = Lre + Lner.

59
Joint Model of Entity Recognition and Relation Extraction
with Self-attention Mechanism
MAOFU LIU and YUKUN ZHANG, Wuhan University of Science and Technology
WENJIE LI, The Hong Kong Polytechnic University
DONGHONG JI, Wuhan University
In recent years, the joint model of entity recognition (ER) and relation extraction (RE) has attracted more and
more attention in the healthcare and medical domains. However, there are some problems with the prior work.
The joint model cannot extract all the relations for a specific entity, and the majority of joint models heavily
rely on complex artificial features or professional natural language processing (NLP) tools. In this article, we
construct a novel joint model that can simultaneously extract all medical entities and relations from medicine
Chinese instructions. Moreover, the self-attention mechanism is introduced to the joint model to learn word
intra-sentence dependencies. The proposed model is evaluated using a medicine Chinese instruction dataset
that we collect and an open dataset provided in CoNLL-2004. Experimental results show that the model with
self-attention achieves the state-of-the-art performance.
CCS Concepts: • Computing methodologies →Artificial intelligence; Natural language processing;
Additional Key Words and Phrases: Joint model, entity recognition, relation extraction, self-attention,
medicine chinese instruction
ACM Reference format:
Maofu Liu, Yukun Zhang, Wenjie Li, and Donghong Ji. 2020. Joint Model of Entity Recognition and Relation
Extraction with Self-attention Mechanism. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 19, 4, Article 59
(May 2020), 19 pages.
https://dx.doi.org/10.1145/3387634
1
INTRODUCTION
The past decade has witnessed that more and more attention has been paid to the processing
of healthcare and medical texts, such as electronic medical records, electronic health records,
biomedical literatures from PubMed,1 medicine instructions, and question answering pairs from
the medical community. Unlike the texts from social media online platforms, the medicine
1https://pubmed.ncbi.nlm.nih.gov/.
The work presented in this article is supported by the Major Projects of National Social Science Foundation of China under
Grant No. 11&ZD189 and the National Natural Science Foundation of China under Grant No. 61672445.
Author’s addresses: M. Liu and Y. Zhang, School of Computer Science and Technology, Hubei Province Key Laboratory of
Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan
430065, China; emails: liumaofu@wust.edu.cn, 978523050@qq.com; W. Li, PQ707, Department of Computing, The Hong
Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong; email: cswjli@comp.polyu.edu.hk; D. Ji, School of Cyber
Science and Engineering, Wuhan University, Wuhan 430072, China; email: dhji@whu.edu.cn.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and
the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee. Request permissions from permissions@acm.org.
© 2020 Association for Computing Machinery.
2375-4699/2020/05-ART59 $15.00
https://dx.doi.org/10.1145/3387634
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
59:2
M. Liu et al.
instructions are usually well written, authoritative, comprehensive, and formally organized. More-
over, the medicine instructions contain the rich semantic data expressed with plenty of medical
entities and relations among them. Therefore, entity classification/recognition and relation extrac-
tion contribute to the semantic understanding of the medicine instructions.
Entity classification/recognition (EC/NER) and relation extraction (RE) from medical texts re-
fer to identifying biomedical entities, such as medicine, disease, gene, protein, and bacteria, from
the unstructured texts in the healthcare and medical domain, and extracting semantic relations
among these entities, e.g., gene and disease, protein and protein interaction, and so on. They are
the basic and critical tasks in medical event extraction, medical information retrieval, and clini-
cal medication guideline and can be beneficial for medical knowledge graph construction, medical
question answering, medicine recommendation, and many other applications related to healthcare
and medicine.
In the previous work, most methods have adopted pipeline models to tackle these tasks. Given
the medical texts, the EC/NER task is usually carried out by conditional random fields (CRF) and
then paired up to commit the RE task by support vector machines (SVM). There are two drawbacks
with these pipeline models: (1) error propagation, that is, errors in the EC/NER task are transmitted
to the RE task, and (2) the ignorance of the inherent association or interaction between EC/NER
and RE.
Recently, due to the popularity of deep learning, more and more researchers have turned to
explore joint models based on neural networks, performing both EC/NER and RE tasks simultane-
ously, to solve the aforementioned problems. They have achieved state-of-the-art performance on
many datasets, such as adverse drug effect (ADE) [1] and Dutch real estate classifieds (DREC) [2].
However, there still exist some problems: (1) Most models heavily rely on hand-crafted or artificial
features and NLP tools, and (2) they cannot identify all semantic relations of a specific entity in a
sentence. For instance, in the following Example 1, “ (urinary tract infection)” could be
involved in not only the cure relationship with “ (norfloxacin glutamate
injection)” but also the causality relationship with “(sensitive bacteria)” simultaneously.
Example 1: :

Norfloxacin glutamate injection: This product is suitable for respiratory tract, urinary tract in-
fection, gonorrhea, prostatitis, intestinal infection, typhoid caused by sensitive bacteria, and other
infections by salmonella.
Meanwhile, most studies have focused on English datasets, and so far, there is no public dataset
available in Chinese. Therefore, we construct our own dataset of medicine Chinese instructions
and propose a novel joint model to solve the two problems mentioned above, with the deep features
automatically learned by BiLSTM and extracting all the relations among entities by self-attention
mechanism and multi-head selection. In our constructed dataset, there are also some discontinuous
entities. As shown in Example 1, “ (respiratory tract)” actually represents “
(respiratory tract infection)” according to its successive “ (urinary tract infection),” and
the “ (respiratory tract infection)” consists of two parts, i.e., “ (respiratory
tract)” and “ (infection)”.
The self-attention mechanism is usually used to learn word dependency in a sentence and
capture the internal structure of the sentence. Compared with the convolutional neural network
(CNN) and the recurrent neural network (RNN), the self-attention ignores the distance between
words and directly calculates word dependency, which is very effective for both long-distance and
local dependencies. More important, it has much faster computation speed and fewer parame-
ters than RNN. At present, the self-attention mechanism has been successfully applied in many
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
Joint Model with Self-attention Mechanism
59:3
NLP tasks, including reading comprehension [3], text summarization [4], machine translation [4],
language inference [7, 8], relation extraction [9], and semantic role labeling (SRL) [10]. Different
from the previous studies, to the best of our knowledge, our work is the first to apply the self-
attention mechanism to the joint model for entity recognition and relation extraction.
In this article, we propose a joint neural network model with the self-attention mechanism for
medical entity classification/recognition and relation extraction from medicine Chinese instruc-
tions. The contributions of our work are summarized in three parts.
(1) We manually annotate 400 pieces of medicine Chinese instructions according to carefully
formulated guideline and scheme and construct a corpus containing 5,505 medical entities
and 5,512 semantic relations.
(2) We label the discontinuous entity taking the Chinese characteristic of medical instruction
into consideration and propose a joint model to process the discontinuous entity auto-
matically.
(3) The self-attention mechanism is integrated into our joint model to learn word dependency,
especially between two medical entities.
The remainder of this article is organized as follows. Section 2 reviews related work. Section 3 de-
scribes the dataset constructed. Section 4 discusses the proposed joint model with the self-attention
mechanism. Section 5 then presents experiments and discussions. Finally, Section 6 concludes the
article.
2
RELATE WORK
The work related to EC/NER and RE, joint learning models, and self-attention in healthcare and
medical domain will be described in detail in this section.
In the early stage, medical EC/NER and RE were mainly based on rules or dictionaries [11, 12].
At present, the mainstream methods are based on machine learning with artificial features. Frunza
et al. [13] investigated on three types of disease-treatment relations, i.e., cure, prevention, and side
effects, in the Medline abstract dataset and support vector machines– (SVM) based classification
model with the artificial semantic features via the external medical knowledge resource. Liu et al.
[14] adopted a k-nearest neighbor– (KNN) based model to extract relations from medicine Chinese
instructions, and the clue words about the corresponding relations had been introduced to the
model. These models heavily rely on complex artificial features and NLP tools, which have poor
portability and increase the complexity and training difficulty.
With the rise of deep learning, some researchers have turned to deep neural networks for
EC/NER and RE. Jagannatha and Yu [15] put forward a BiLSTM with a CRF model to identify
clinical events. Lin et al. [16] completed the relation extraction task based on CNN using the at-
tention mechanism at the sentence level. Quan et al. [17] constructed a multi-channel CNN model
for RE by introducing into the convolution channel the word vectors pre-trained from PubMed,
PMC, Medline, and Wikipedia.
All the above studies have regarded EC/NER and RE as two independent tasks in a pipeline
model. The joint models have recently attracted more and more attention, because they can solve
the problems mentioned in the Introduction well. Yang and Claire [18] presented a joint model that
sought a globally optimal solution from the optimal results of subtasks. Singh et al. [19] proposed
a single joint graphical model to represent the various dependencies between subtasks. Zheng
et al. [20] used the shared BiLSTM to encode the sentence and adopted an LSTM for NER and
a CNN for RE. Gupta et al. [21] proposed a method that relied on RNNs but used a lot of hand-
crafted features and additional NLP tools to extract features like POS tags. Adel and Schütze [22]
solved the EC and RE tasks using an approximation of a global normalization objective, i.e., CRF,
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
59:4
M. Liu et al.
and they replicated the context of the sentence including the left and right parts of the entities
to feed one entity pair once a time to a CNN for relation extraction. Pershina et al. [23] proved
that the relation extractors trained with distant supervision can benefit significantly from a small
number of manually labeled examples. Fu et al. [24] learned unified representation of relations
via multi-task learning between multiple relation datasets and obtained significant improvement
[25]. They also adopted a domain adversarial neural network to learn cross-domain features and
obtained improvement on all three test domains at ACE-2005. Chan et al. [26] presented a system
for rapidly customizing event extraction capability to find new event types and their arguments.
The aforementioned works have not simultaneously inferred other potential entities and relations
within the same sentence and only completed the EC and RE tasks.
Miwa and Bansal [27] introduced the dependency information into the RE task through parame-
ter sharing. They classified the relations according to their shortest paths in the dependency trees.
Li et al. [28] also proposed a model with tree-LSTMs to learn the dependency information for en-
tity and relation extraction from the biomedical texts. Katiyar and Cardie [29] and Bekoulis et al.
[30] investigated attention-based RNNs for extracting relations between entity mentions without
using any dependency parse tree feature. Zhang et al. [31] established an end-to-end relationship
extraction model based on global optimization, and used new LSTM features to better represent
context. Zheng et al. [32] employed a new annotation strategy to transform the joint task into
a sequence labeling problem. However, all of them assumed that the relations are mutually ex-
clusive. There are limitations with these approaches. First, an entity can only participate in one
relation. Second, the time complexity of entity recognition is increased compared to the standard
approaches that are of linear complexity. In our constructed dataset, we find that the entity of-
ten corresponds to multiple relationships. We solve this problem by casting relation extraction as
multi-head prediction.
Vaswani et al. [4] introduced self-attention to neural machine translation (NMT) and achieved
state-of-the-art performance. They also proposed multi-head attention based on self-attention.
Paulus et al. [5] exploited the distance between relative positions or sequence elements to expand
self-attention, which effectively improved the translation quality and efficiency. Tan et al. [10]
used self-attention in the SRL task and achieved state-of-the-art performance on the CoNLL-2005
and CoNLL-2012 datasets. Verga et al. [9] combined self-attention with the biomedical relation
extraction. They proposed a document-level model and obtained the significant experimental re-
sults in the dataset of chemical disease relations (CDR). In this article, we put forward the joint
model with the self-attention mechanism to commit medical EC/NER and RE simultaneously from
the medicine Chinese instructions. Lin et al. [7] used self-attention to learn the embedding of a
sentence, expressed as a two-dimensional matrix instead of a vector where each line in the matrix
represented different parts of the sentence. Cheng et al. [3] applied self-attention to reading com-
prehension, solving the limitations of traditional RNN for processing the inherently structured
input. Shaw et al. [6] introduced self-attention mechanism to the relative position representation
between elements, and the relative position representation of self-attention achieved a significant
improvement over absolute position encoding of self-attention in two machine translation tasks.
Shen et al. [8] adopted self-attention to natural language reasoning and made the state-of-the-art
performance on multiple datasets.
3
DATASET CONSTRUCTION
3.1
Medicine Instruction
In the healthcare and medical domain, there are already some datasets in English, e.g., ADE and
DREC, and many studies have achieved good performances on these datasets. However, they are
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
Joint Model with Self-attention Mechanism
59:5
not publicly available, and there are no available domain-specific Chinese datasets. In this work,
we construct a new linguistic corpus with medicine Chinese instructions.
The medicine instruction is a document that contains the important information about the
medicine. It is an important guiding document for clinicians and patients to use medicine safely
and effectively, which is of great significance in diagnosis and treatment of medicine. The descrip-
tion of the antibacterial medicine usually contains dense diseases, bacteria, and medicines with the
clear and identifiable boundaries, and the relationship between entities is much easier to judge, so
we select the antibacterial medicine instructions as the language material in our dataset. A total
of 1,053 antibacterial medicine Chinese instructions have been crawled from Chinese websites,2
and 400 medicine instructions with rich semantic information and clear structure are chosen to
construct the medicine Chinese instruction corpus.
3.2
Annotation Guideline
The annotation of medical entity and relationship between entities usually involves semantic un-
derstanding expertise. Under the guideline put forward by medical experts and iterations via an-
notation practices, we have formulated the complete medical entity and relationship annotation
guideline and system.
Referring to the medical glossary [33] and the suggestions from medical experts, we develop
the following annotation guideline.
(1) Our corpus contains seven types of medical entities, i.e., disease, symptom, medicine,
body-region, patient, bacteria, and treatment. There are nine kinds of medical entity re-
lations in our corpus, i.e., cure, hyponymy, causality, anaphora, meronymy, prevention,
inhibition, sensitivity, and drug-resistance.
(2) The longest principle is adopted to ensure the semantic integrity of a medical entity. If
there exists more than one type of a medical entity in a text span, then the longest one is
retained according to the longest principle and the other shorter are discarded.
(3) A medical entity may consist of two discontinuous parts. We refer to this kind of the
medical entity as the discontinuous entity and annotate two parts as a single medical
entity with the same entity type.
(4) After entity annotation, the semantic relationship between specific medical entities is rep-
resented by a structured triple (former-entity, relation-type, later-entity). For the entity
combination rule, the structured triple complies with the order of linguistic reading of
semantic relationship.
Example 2: :
,

Mezlocillin sodium injection: This product is suitable for respiratory, urinary, digestive, gynecolog-
ical, reproductive, skin, soft tissue, eye, ear, nose or throat infections caused by the sensitive strains of
gram-negative bacilli, such as Escherichia coli, enterobacter, and proteus, and these infections include
sepsis, purulent meningitis, peritonitis, and osteomyelitis.
In Example 1, the string “ (norfloxacin glutamate injection)” is labeled
as a medicine entity altogether according to our annotation guideline (2). Only when there exists
the string “ (norfloxacin glutamate)” in the medicine instruction, we will mark
“” as a medicine entity. In Example 2, there exists an abstract or concrete disease
2http://top.chinaz.com/site_www.yiwang.cn.html.
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
59:6
M. Liu et al.
Table 1. Medical Entities in Example 2
Entity type
Entity
medicine
mezlocillin sodium injection
specific disease
sepsis; purulent meningitis; peritonitis; osteomyelitis
abstract disease
(infection)
respiratory; urinary; digestive; gynecological; reproductive;
skin; soft tissue; eye; ear; nose; throat
bacteria
sensitive strains; gram-negative bacilli; E. coli; enterobacter;
proteus
about body-region, e.g., “ (throat infection),” we only annotate the disease “
(throat infection)” according to our annotation guideline (2) and the medical entity of body-region
“ (throat)” is discarded. In Example 2, “” in Chinese actually means “skin
and soft tissue infection” in English. So, the two diseases “ (skin infection)” and “
 (soft tissue infection)” are labeled according to our annotation guideline (3), and they share
the common linguistic component, i.e., “ (infection).” Afterward, the semantic relationship
is represented by the structured triple, e.g., (Escherichia coli, causality, digestive infection). The
former entity indicates the reason, and the latter one indicates the result in this causality relation,
according to our annotation guideline (4).
In Example 2, as shown in Table 1, there are 20 medical entities, including one medicine, four
specific diseases, 11 abstract body-region infections, and five bacteria in a single Chinese sentence.
Among these entities, we can find a number of relationships, e.g., the cure relationships among
medicine mezlocillin sodium infection and all diseases, the causality relationships among all bacte-
ria and diseases, the hyponymy relationships among gram-negative bacilli and E. coli, enterobacter,
proteus, the hyponymy relationships among sensitive strains and E. coli, enterobacter, and proteus.
3.3
Dataset
For each medicine Chinese instruction, five annotators manually annotate the entity, the entity
type, the semantic relation, and the relation type independently according to the annotation guide-
lines. We adopt the majority principle, i.e., the minority subordinating to the majority, to settle the
different opinions among annotators at the entity or the relation level. When the number of anno-
tators holding two different opinions is close on one instance, i.e., 3 to 2, we invite another medical
expert as the arbitrator to confirm reliability and consistency of our annotation.
The annotation results for 400 medicine instructions are shown in Table 2. In total, we have
annotated 5,505 entities and 5,512 relationships. The dataset is available via the github3 link.
4
MODEL
In this section, we explain our model, as shown in Figure 1, in detail. We develop a joint model
based on BiLSTM to (1) identify the boundary and the type of the medical entity and (2) extract
semantic relations among entities. The model is divided into five layers from bottom to top: (1) an
embedding layer, (2) a self-attention layer, (3) a BiLSTM layer, (4) a NER layer, and (5) a sigmoid
layer.
The input to the model is a sequence of tokens, which are then converted into the correspond-
ing embedding vectors. The self-attention and BiLSTM layers are used to extract deep contextual
information. Afterward, the NER and sigmoid layers perform the NER and RE tasks, respectively.
3https://github.com/zhangyukun230/medical-dataset.
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
Joint Model with Self-attention Mechanism
59:7
Table 2. Statistics of Medical Entities and Relations in Medicine Chinese Instruction Corpus
Entity
Amount
Relation
Amount
medicine
1,046
cure
1,734
disease
2,914
hyponymy
1,273
bacteria
1,280
causality
1,742
symptom
32
anaphora
254
patient
141
meronymy
7
body-region
37
prevention
29
treatment
55
inhibition
148
all
5,505
sensitivity
293
drug-resistance
28
all
5,512
Fig. 1. Overview of our joint model. The model consists of five layers, i.e., embedding, multi-head attention,
BiLSTM, CRF for NER, and sigmoid.
The output for each input token has two parts: (1) a recognized entity label, e.g., I-drug, which
denotes the token inside a named entity of drug, and (2) a set of tuples comprising the head tokens
of the entity and the types of relations between them, e.g., {( (bacteria), sensitive)}. An entity
often contains more than one token, and we only consider the last token of the entity for multi-
head selection. For example, there is a sensitive relation between the entities “ (penicillin)”
and “ (bacteria).” Instead of connecting all tokens of the entities, we only link “” with “”.
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
59:8
M. Liu et al.
Fig. 2. The computation graph of multi-head attention mechanism.
Moreover, if there is no semantic relation between the two entities, then we assign the label “N”
and regard the token itself as the head.
4.1
Embedding Layer
Given a sentence X = (x1,x2, . . . ,xn) as a sequence of tokens, the embedding layer transforms
these tokens into the vector matrix by pre-trained embedding so that they can be calculated in a
neural network model.
4.2
Self-attention Layer
Self-attention is a specific case of the attention mechanism that only requires a single sequence
to compute its representation. Following Vaswani et al. [4], we use multi-head attention based on
self-attention in this article. Figure 2 depicts the computation graph of the multi-head attention
mechanism.
Given a matrix of n query vectors Q ∈Rn∗d, keys K ∈Rn∗d, and values V ∈Rn∗d, the scaled
dot-product attention computes the attention scores with the following formula (1):
Attention (Q,K,V ) = softmax
QKT
√
d

V,
(1)
where Q = K = V in the self-attention layer.
For multi-head attention, X is converted to (Q,K,V ) through different linear elements. Then, h
parallel heads are employed to focus on a different part of channels of value vectors. We denote the
learned linear maps as W Q
i
∈Rn∗d/h, W K
i
∈Rn∗d/h,W V
i
∈Rn∗d/h, which correspond to queries,
keys, and values, respectively. For the ith head, the scaled dot-product attention is calculated, and
the mathematical formulation is shown as follows:
Mi = Attention

QW Q
i ,KW K
i ,VW V
i

.
(2)
Finally, all vectors produced by parallel heads are concatenated together to form a single vector.
A linear map is used to mix different channels from different heads:
M = Concat (M1, M2, . . . , Mh) ,
(3)
Y = MW ,
(4)
where M ∈Rn∗d,W ∈Rd∗d, andY. is the output of self-attention layer and also the input of BiLSTM
layer.
4.3
BiLSTM Layer
LSTM has the ability to well capture long-term dependencies and its extension BiLSTM can make
use of the contextual information in both past and future. Therefore, BiLSTM models have achieved
excellent results in many NLP tasks. In this work, we adopt a BiLSTM to encode the text of medicine
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
Joint Model with Self-attention Mechanism
59:9
Chinese instruction. We empirically set the number of LSTM layers to 3. For each vector yi of Y,
we concatenate the forward output (−→
hi) and the backward output (←−
hi) at the timestep i. The entire
output of BiLSTM at the timestep i is then as follows:
hi =
−→
hi

,
←−
hi

;i = 0, 1, . . .n..
(5)
4.4
NER Layer
Similarly to Miwa and Bansal [27], Zheng et al. [32], and Katiyar and Cardie [29], we formulate
the NER task as a sequence labeling problem. In sequence labeling, an entity usually contains
more than one token and one tag is assigned to each token in the entity. We adopt the BIO tagging
scheme, i.e., using B, I, and O to denote the beginning, inside, and outside of the entity, respectively.
We annotate the first character of an entity as B-type, the other tokens of the entity as I-type, and
those non-entity tokens as O. With the BIO tagging scheme, we can identify the boundary and the
type of entity simultaneously. In Figure 1, the Chinese token “” has been tagged as B-drug, “”
and “” are tagged as I-drug, and “” is tagged as O. The scores of the tags are calculated by the
following formula:
s(e)(hi) = V (e) f (W (e)hi + b(e))V ∈Rl∗n (entity),W ∈Rl∗2d,b ∈Rl,
(6)
where e is the symbol denoting NER, f (·)is the activation function such as relu or tanh, d is the
number of LSTM hidden units, n(entity) is the number of NER tags, and l is the length of input w.
In the NER task, the BIO labeling scheme faces several restrictions. For example, B-drug can-
not be followed by I-disease. In the softmax, the tags assigned to the tokens according to their
probabilities are independent of each other, although BiLSTM is able to learn the contextual in-
formation. CRF has a transition characteristic that takes the order among tags into consideration.
Therefore, we use a linear-chain CRF (Lample et al. [34]) for the NER task. For input token wi, the
score sequence can be expressed as follows:
S

y1(e), . . . , yn (e)
=
n
i=0 s(e)
i,yi (e) +
n−1
i=1 Ti,yi (e),
(7)
where s(e)
i,yi (e) represents the tag score when the tag of wi is yi, T denotes the transition score from
tagyi toyi+1, andT ∈R(p+2,p+2). Then, the probability of a given sequence of tags over all possible
tag sequences for the input sentence w can be defined by the following formula (7):
Pr

y(e)
1 , . . . y(e)
n |w

=
es (yi (e), ...,y(e)
n
)

˜y(e)
1
... ˜y(e)
n es
y( ˜e)
l , . . . , ˜y(e)
n
 ,
(8)
where we adopt Viterbi to find the tag sequence ˜y(e)
i , which has the highest score.
Entity tags are then fed into the RE task as the corresponding label embedding, considering that
the types of entities involved benefit relation prediction. Our model takes the label embedding
as a parameter layer and learns it in the training phase. The input of the sigmoid layer is the
concatenation of the BiLSTM state hi and the label embedding дi of xi,
zi = (hi,дi).
(9)
4.5
Sigmoid Layer
Inspired by the work of Bekoulis et al. [2], we regard the RE task as a multi-head selection problem,
which can successfully solve the problem with most existing joint models that cannot recognize
all semantic relations for a specific entity in a sentence. For token wi, it may hold any type of
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
59:10
M. Liu et al.
semantic relation rk with other tokenwj. In this article, we make the assumption that the semantic
relations between one entity and others are independent.
Given a sequence of tokens w and a set of relation labels R, the goal of our model is to identify the
vector of the most probable heads wj and the vector of the most probable corresponding relation
label rk for each token wi. We calculate the score between tokens wi and wj given a label rk as
follows:
s(r )(wi,wj,rk) = V r f

W (r )
1 zi +W (r )
2 zj + b(r )
,V ∈Rl∗n(relation),W ∈Rl∗2d+e,b ∈Rl,
(10)
where r, V r, d, e, and l represent the symbol of RE, the weight matrix, the number of LSTM hid-
den units, the dimension of label embedding, and the length of input w, respectively. f (·) and
n(relation) denote the activation function, relu or tanh, and the number of RE tags. The probabil-
ity of the token wj to be selected as the head of the token wi with the relation label rk between
them is defined as
Pr(head = wj,label = rk |wi) σ

s(r )
wi,wj,rk

,
(11)
where σ is the sigmoid function. Unlike softmax, sigmoid assumes that all relations are indepen-
dent of each other, and it does not add up all probabilities of relations to 1. If the value of Pr is
more than 0.5, then sigmoid will determine whether there is a semantic relation between the two
entities; otherwise, no relation exists, and it marks “N.” The goal of sigmoid is to minimize the
cross-entropy loss Lre. during the training phase,
Lre =
n

i
m

j
−log Pr(head = yi,j,re = ri,j |wi).
(12)
For the joint entity and relation extraction task, we calculate the final objective L = Lre + Lner.
We set the weights of Lre and Lner as the hyper-parameters in our experiments and find that the
performance is optimal when both the weights of them are 1.
5
EXPERIMENTS
5.1
Datasets and Settings
From Table 2, we can find that the medical entities and relationships in the medicine instructions
are distributed unevenly. In this work, we focus on the entities of medicine, disease, bacteria, pa-
tient, and treatment, which hold the almost even distribution in the dataset. Meanwhile, this article
only investigates the semantic relations of cure, hyponymy, causality, inhibition, and sensitivity.
Thirty-two pieces of medical instructions, which do not contain any of the five types of semantic
relations mentioned above, are removed from the data collection.
There are 839 discontinuous entities in our constructed dataset, accounting for approximately
17.90% of all annotated medical entities. In Example 3, we use “ (urinary tract),” “
(respiratory tract),” and “(skin)” to represent the discontinuous entities “ (urinary
tract infection),” “ (respiratory tract infection),” and “ (skin infection).” “ (uri-
nary tract),” “ (respiratory tract),” and “ (skin)” are labelled as diseases rather than
body-regions. In our model, “ (amoxicillin and clavulanate potassium
granules)” and “ (urinary tract)” form a cure relationship, with the underlying contextual in-
formation guidance. Our joint model can recognize “ (urinary tract)” as a disease. In addition,
the entity type information will be converted to label embedding and conveyed to the RE task.
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
Joint Model with Self-attention Mechanism
59:11
Table 3. Statistics of Medical Entities
Entity
Disease
Bacteria
Medicine
Patient
Treatment
All
training
2,118
784
723
63
33
3,721
test
515
239
176
26
11
967
all
2,633
1,023
899
89
44
4,688
Table 4. Statistics of Relations among Medical Entities
Relation
Hyponymy
Cure
Causality
Inhibition
Sensitivity
All
training
908
1,301
1,262
108
216
3,795
test
285
258
321
34
61
959
all
1,193
1,559
1,583
142
277
4,754
Example 3: :
;
Amoxicillin and clavulanate potassium granules: This product is suitable for respiratory tract uri-
nary tract and skin soft tissue infection caused by Klebsiella.
In the end, a total of 368 medicine Chinese instructions have been selected. Eighty percent of
them are used for training and the remaining 20% for testing. The statistics of the entities and
relations in the training set and the test set are presented in Tables 3 and 4, respectively. Besides
our constructed dataset, we also use the CoNLL-2004 dataset [35] to evaluate the generalization
of our model.
For comparison, we build a baseline model, which consists of four layers, i.e., embedding, BiL-
STM, CRF, and sigmoid. Compared to our model, the baseline model removes the self-attention
layer. In this way, we can not only explore the advantages of the joint model in relation to the
other works but also understand how self-attention improves the experimental results.
In this article, for Precision, Recall, and F1-score, we also use two different evaluation standards,
i.e., strict and relaxed, to be able to compare our experimental results with previous studies.
(1) Strict: An entity is considered correct if both the boundaries and the type of the entity are
correct.
(2) Relaxed: We regard a multi-token entity correct if at least one of its composite token types
is correct.
For CoNLL-2004, we use the embedding vectors in the previous work [21, 22]. We adopt the
same splits as in Gupta et al. [21] to do the EC/RE task and evaluate our model on the NER/RE
task similar to Miwa and Sasaki [36], Bekoulis et al. [2] on the same dataset using 10-fold cross
validation. We also perform the same two tasks mentioned above on our own constructed dataset.
We use pre-trained embedding vectors by word2vec based on Wikipedia. The dimension of the
vectors is 50.
The hyper-parameters of the model are shown in Table 5. The parameters of baseline and our
model are the same except for h-head. We use Adam algorithm to optimize the parameters, and the
number of parallel headsh is 2. We commit dropout after every layer and fix the hyper-parameters,
i.e. dropout values, best epoch, and learning rate, on the validation and test datasets.
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
59:12
M. Liu et al.
Table 5. The Hyper-parameter Settings
Hyper-
parameter
Optimizer Activation
Learning
rate
Dropout
LSTM
unit
LSTM
layers
Label
embedding
h-head
our
dataset
Adam
tanh
0.003
0.9
64
3
20
2
CoNLL-
2004
Adam
tanh
0.003
0.8
64
3
25
2
Table 6. The Experimental Results
Dataset
Model
Method
F
Standard
Entity
Relation
Overall
EC/RE
CoNLL-2004
Kate and Mooney (2010)
P

Relaxed
91.36
66.28
78.82
Kate and Mooney (2010)
J

Relaxed
91.70
66.36
79.03
Gupta et al. (2016)
P

Relaxed
91.00
97.1
79.05
Gupta et al. (2016)
J

Relaxed
92.40
69.90
81.15
Gupta et al. (2016)
J
×
R elaxed
88.80
58.30
73.60
Adel and Schütze (2017)
J
×
R elaxed
82.10
62.50
72.30
Bekoulis et al. (2018)
J
×
R elaxed
93.26
67.01
80.14
Our model
J
×
R elaxed
94.65
69.39
82.03
our dataset
Baseline
J
×
R elaxed
97.39
73.93
85.65
Our model
J
×
R elaxed
97.87
75.88
86.87
Our model -LE
J
×
R elaxed
96.13
74.98
85.56
NER/RE
CoNLL-2004
Miwa and Sasaki (2014)
J

Strict
80.70
61.00
70.85
Bekoulis et al. (2018)
J
×
S trict
83.04
61.04
72.04
Our model
J
×
S trict
85.26
62.59
73.93
our dataset
Baseline
J
×
S trict
91.80
72.43
82.12
Our model
J
×
S trict
93.25
73.57
83.41
Our model -LE
J
×
S trict
92.45
71.79
82.12
The models are as follows: (ⅰ) baseline for NER/RE, (ii) baseline for EC (predicting only entity classes)/RE, (iii) our model
for NER/RE, and (ⅳ) our model for EC/RE. The  and × symbols indicate whether the models rely on external NLP tools.
We include different evaluation standards (Strict and Relaxed). P means pipeline method and J means joint model. “-LE”
indicates that label embedding is removed from our model.
5.2
Experimental Results
The experimental results are shown in Table 6. On the two datasets we use both strict and re-
laxed standards. In the relaxed setting, we perform an EC task instead of NER, assuming that the
boundaries of entities are given.
For the EC task on the CoNLL-2004 dataset, due to a large number of experimental results,
we present the overall F1-score of different experiments in the form of histogram. We can see
the details in Figure 3, where the symbols “” and “ × ” mean whether the models combine the
artificial features.
Compared with the experiments of Kate and Mooney [37] and Gupta et al. [21], we can find that
the joint model performs better than the pipeline model under the same conditions. This verifies
the advantages of the joint model. The 93.25% F1-score of NER task on our dataset verifies that our
annotation strategy and joint model are very effective.
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
Joint Model with Self-attention Mechanism
59:13
Fig. 3. The overall F1-score of EC experiments.
Since the pipeline model cannot learn the semantic relevance between the entity and relation
extraction tasks, it brings in the entity extraction errors and transfers this type of errors to the re-
lation extraction. For Example 4, the pipeline model of the NER task detects medical entities with
the bacteria entity type like “ (mycoplasma pneumoniae),” “ (chlamy-
dia trachomatis),” and “ (mycoplasma humanity).” In subsequent relation extraction,
the causality relations between them are naturally misidentified. With our joint model, all these
entities and relations are extracted correctly.
Example 4: : 1.

2.

Azithromycin injection: This product is suitable for the following infections caused by sensi-
tive pathogenic strains. 1. Community acquired pneumonia caused by Chlamydia pneumoniae,
Haemophilus influenzae, Legionella pneumophila, Moraxella catarrhalis, Mycoplasma pneumoniae,
Staphylococcus aureus, or Streptococcus pneumoniae, which needs to be treated by intravenous drip
first. 2. Pelvic inflammation caused by Chlamydia trachomatis, Neisseria gonorrhoeae, and My-
coplasma hominis, which needs to be treated by intravenous drip first.
Entity:
medicine:  (azithromycin for injection);  (this product).
treatment:  (intravenous drip);  (intravenous drip).
disease:  (infections);  (community acquired pneumonia); 	 (pelvic
inflammation).
Pipeline model:
bacteria:  (Haemophilus influenzae);  (Legionella pneumophila);
 (Moraxella catarrhalis);  (Staphylococcus aureus);  (Strepto-
coccus pneumoniae);  (Neisseria gonorrhoeae).
Our joint model:
bacteria:      (Chlamydia pneumoniae);       (Haemophilus influen-
zae);  (Legionella pneumophila);  (Moraxella catarrhalis);
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
59:14
M. Liu et al.
Table 7. The Comparative Experimental Results with Our Model and
Bekoulis et al. (2018)
Dataset
Model
Entity
Relation
CoNLL-2004
Bekoulis et al. (2018)
896
257
CoNLL-2004
our model
920
264
our dataset
Bekoulis et al. (2018)
888
695
our dataset
our model
902
706
(Mycoplasma pneumoniae);  (Staphylococcus aureus);  (Streptococ-
cus pneumoniae);  (Chlamydia trachomatis);  (neisseria gonorrhoeae);
 (mycoplasma hominis).
Relation:
cure: (, ); (, ); (,
).
Pipeline model:
causality: (, ); (, ); (,
); (, ); (, ); (
, ); (, 	); (, ).
Our joint model:
causality: (, ); (, ); (,
); (, ); (, ); (
, ); (, ); (, 	); (,
); (, ); (, ).
Compared with the works of Gupta et al. [21] and Adel and Schütze [22], our model achieves
about 6% improvement on both EC and RE tasks, even without complex hand-crafted features.
Moreover, compared with the model of Gupta et al. [21] that relies on complex hand-crafted fea-
tures, our model also improves about 1%. For the NER task, we improve overall F1-score with about
3% and 2% compared to the works of Miwa and Sasaki [36] and Bekoulis et al. [2]. This indicates
that our model has better performance on the CoNLL-2004 dataset. It suggests that our neural net-
work model can obtain more semantic information than the feature-based model or the existing
neural network model. Unlike the prior studies that obtain the entity types and the corresponding
relations from the pairs of entities, our model takes the entire sentence as input, rather than just
the entity pairs.
In NER/RE task, Comparing with the work of Bekoulis et al. (2018) [2] on the CoNLL-2004
dataset, our model makes around 2% overall improvement. On our constructed dataset, our model
achieves more than 1% overall improvement in experiments. The comparative experimental results
with our model and Bekoulis et al. (2018) [2] are shown in Table 7.
From Table 7, we can find that more 24 entities and 7 relations are identified from CoNLL-
2004 dataset via our model comparing with the model of Bekoulis et al. (2018) [2], and our model
also recognizes additional 14 entities and 11 relations from our constructed dataset. Our model
has designed one more self-attention layer comparing with Bekoulis et al. (2018) [2], which can
learn the relationship between entities in the sentence or instruction globally. In fact, our model
has learned word dependencies in the sentence or instruction via the self-attention mechanism,
especially between the entities. Taking “ (penicillin is suitable for sensitive
bacteria)” as an example, and its self-attention weight map is shown in Figure 4. In our experiment,
the “” and “” represent the last tokens of the entities “” and “”, respectively. The
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
Joint Model with Self-attention Mechanism
59:15
Fig. 4. The weight map of “ (penicillin is suitable for sensitive bacteria)” by our
model with the self-attention mechanism.
weight by our model between the “” and “” is the maximum in Figure 4, denoting a sensitive
relationship between the two entities “” and “”. This verifies that the self-attention
mechanism can strengthen the connection between the related entities “” and “” via
learning entity dependencies.
In Example 5, the model proposed by Bekoulis et al. (2018) [2] fails to recognize the disease entity
“” and its relationships. But our joint model identifies the entity “” and its related relations,
including (, causality, ), (, causality, ), (, causality,
), (, causality, ), (, causality, ), (, causality, ), and
(, cure, ).
Example 5: : 	,

,

Acetylkitasamycin tablet: This product is mainly suitable for various infections caused by gram-
positive bacteria, especially for upper and lower respiratory tract infections caused by Staphylococcus
aureus, pneumococcus, and S. epidermidis, as well as skin soft tissue infections. According to the lit-
erature, this product has good curative effect on pertussis, scarlet fever, otitis media and so on.
Entity:
medicine:  (acetylkitasamycin tablets);  (this product).
bacteria:  (Gram-positive bacteria);  (Staphylococcus aureus);
 (pneumococcus);  (S. epidermidis).
Model of Bekoulis et al. (2018) [2]:
disease:  (pertussis);  (scarlet fever);  (otitis media);  (infections);
 (soft tissue infections);  (lower respiratory infections).
Our model:
disease:  (pertussis);  (scarlet fever);  (otitis media);  (infections);
 (soft tissue infections); (upper);  (lower respiratory infections).
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
59:16
M. Liu et al.
Relation:
Model of Bekoulis et al. (2018) [2]:
causality: (, ); (, ); (,
); (, ); (, ); (,
).
cure: (, ); (, ); (,
); (, ); (, ); (, ).
Our model:
causality: (,); (,); (,); (
,); (,); (,); (
,); (,); (,).
cure: (, ); (, ); (,
); (, ); (, ); (, ); (
, ).
Meanwhile, we find that the same model achieves better performance on our dataset than the
CoNLL-2004 dataset. The reason is that medical entities have obvious character characteristics in
the Chinese antibacterial medical instructions. We can see from Example 2 that the diseases usually
end with the Chinese characters “” and “”, e.g., “ (meningitis),” “ (osteomyelitis),”
and “ (Septicemia),” and the bacteria generally end with the Chinese character “”, e.g.,
“ (Escherichia coli)” and “ (Enterobacter).” Another interesting linguistic phe-
nomenon is that the name of medicine usually appears at the beginning of a medicine instruction.
Finally, we verify the impact of label embedding (LE) on the experiment results. After imple-
menting the model with the LE feature, we conduct a comparative experiment to uncover its signif-
icance. The experimental results are shown in Table 6. We can find that the performances of both
EC and NER decrease by around 1% when the LE feature is removed. Therefore, we can make the
conclusion that the category information of the medical entities is beneficial to relation extraction.
5.3
Error Analysis
The distributions of entity type and relation type are uneven, so the small number of entity or
relation instances in the training dataset makes our model under-fitting for a type of entity or
relation, which results in misidentification. In Example 6, “ (ascaris muscle)” rarely ap-
pears in our dataset and our model misidentifies it as the bacteria type. The relation between “
 (piperazine)” and “ (ascaris)” is also missed by our model due to its few occurrences.
Example 6: 	:
,
	,

Piperazine citrate syrup is used for ascaris and enterobiasis infection. Piperazine can paralyze as-
caris muscle, and its mechanism may make piperazine play an anticholinergic role at the neuromus-
cular junction of ascaris and block the excitatory effect of acetylcholine on ascaris muscle.
Entity:
medicine: 	 (piperazine citrate syrup); 	 (piperazine); 	 (acetyl-
choline).
disease:  (enterobiasis infection);  (Ascaris).
bacteria:  (Ascaris muscle).
Relation:
cure: (	, ); (	, ).
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
Joint Model with Self-attention Mechanism
59:17
Besides our constructed dataset, we also made the experiments to the remaining unlabeled 653
pieces of medicine Chinese instructions, the following Example 7 has shown the automatic la-
belling results with our model.
Example 7: :

,
	, , 	,

:

Rifampin injection: tuberculosis. This product is used in combination with other antituberculosis
medicines to treat various types of tuberculosis, including initial treatment, progressive, chronic and
drug-resistant cases. This product is also effective for most atypical mycobacterium strains. The other
infection: This product can treat refractory legionella and severe methicillin resistant staphylococcus
infection.
Entity:
medicine:  (rifampin injection);  (this product);  (this product);
 (antituberculosis medicines);  (this product); 	 (methicillin).
disease:  (infection); 	 (tuberculosis);  (staphylococcus infection);
 (tuberculosis);  (refractory legionella);  (severe resistant).
bacteria:
 (atypical mycobacterium strains).
Relation:
inhibition: (,
).
cure: (, ); (, ); (, 	); (
, 	).
We can find that entity extraction achieves good performance on the unlabeled medicine Chi-
nese instruction. Moreover, our model makes satisfactory result in relation extraction with the
unlabeled dataset to some extent. Our model recalls all the medical entities but the extracted
 (severe resistant) is not correct. All the extracted five relations are correct, but our model fails
to identify the other three relations, i.e., (, cure, ), (	, cure,
	), and (, anaphora, ).
6
CONCLUSIONS
In this work, we construct a language corpus with Chinese medical instructions according to our
specified annotation guidelines. A novel joint model is then developed to extract medical entities
and relations simultaneously. Moreover, the self-attention mechanism is introduced into the joint
model to learn the intra-sentence word dependencies. Although the proposed model does not use
any artificial features and NLP tools, its performance is the state of the art.
In the future, we would like to focus on data generalization and try to train the learning ability
of our model on the small dataset. We will conduct more experiments using emergent and popular
pre-trained language models, such as BERT and GPT, and evaluate their contributions to the tasks.
Our constructed dataset contains a large number of discontinuous entities, currently, only the main
parts of the discontinuous entities can be labelled. In the future, we will introduce more language
rules to solve the problem of discontinuous entity.
REFERENCES
[1] Harsha Gurulingappa, Abdul M. Rajput, Augus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo.
2012. Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from
medical case reports. J. Biomed. Inf. 45, 5 (2012), 885–892.
[2] Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Joint entity recognition and rela-
tion extraction as a multi-head selection problem. Expert Syst. Appl. 114 (2018), 34–45. https://bmcbioinformatics.
biomedcentral.com/track/pdf/10.1186/1471-2105-6-S1-S14.
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
59:18
M. Liu et al.
[3] Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine read-
ing. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP’16),
551–561.
[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia
Polosukhin. 2017. Attention is all you need. In Proceedings of 31st Conference on Neural Information Processing Systems
(NIPS’17), 5999–6009.
[5] Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization.
In Proceedings of the 6th International Conference on Learning Representations (ICLR’17).
[6] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Pro-
ceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies (NAACL-HLT’18), 464–468.
[7] Zhouhan Lin, Minwei Feng, Cicero N. Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A struc-
tured self-attentive sentence embedding. In Proceedings of the 6th International Conference on Learning Representations
(ICLR’17).
[8] Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. DiSAN: Directional self-
attention network for RNN/CNN-free language understanding. In Proceeding of the 32nd AAAI Conference on Artificial
Intelligence (AAAI’18), 5446–5455.
[9] Patrick Verga, Emma Strubell, and Andrew McCallum. 2018. Simultaneously self-attending to all mentions for full-
abstract biological relation extraction. In Proceedings of the 2018 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’18), 872–884.
[10] Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018. Deep semantic role labeling with
self-attention. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI’18), 4929–4936.
[11] Daniel Hanisch, Katrin Fundel, Heinz-Theodor Mevissen, Ralf Zimmer, and Juliane Fluck. 2005. ProMiner: Rule-based
protein and gene entity recognition. BMC Bioinf. 6, Suppl 1 (2005), S14. https://bmcbioinformatics.biomedcentral.com/
track/pdf/10.1186/1471-2105-6-S1-S14.
[12] Christopher S. G. Khoo, Syin Chan, and Yun Niu. 2000. Extracting causal knowledge from a medical database using
graphical patterns. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics (ACL’00),
336–343.
[13] Oana Frunza and Diana Inkpen. 2010. Extraction of disease-treatment semantic relations from biomedical sentences.
In Proceedings of the 2010 Workshop on Biomedical Natural Language Processing, 91–98.
[14] Maofu Liu, Li Jiang, and Huijun Hu. 2017. Automatic extraction and visualization of semantic relations between
medical entities from medicine instructions. Multimedia Tools Appl. 76, 8 (2017) 10555–10573.
[15] Abhyuday N. Jagannatha and Hong Yu. 2016. Bidirectional RNN for medical event detection in electronic health
records. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies (NAACL-HLT’16), 473–482.
[16] Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective
attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics
(ACL’16), 2124–2133.
[17] Chanqin Quan, Lei Hua, Xiao Sun, and Wenjun Bai. 2016. Multichannel convolutional neural network for biological
relation extraction, BioMed Research International 2016 (2016), 1–10. http://downloads.hindawi.com/journals/bmri/
2016/1850404.pdf.
[18] Bishan Yang and Claire Cardie. 2013. Joint inference for fine-grained opinion extraction. In Proceedings of the 51st
Annual Meeting of the Association for Computational Linguistics (ACL’13), 1640–1649.
[19] Sameer Singh, Sebastian Riedel, Brian Martin, Jiaping Zheng, and Andrew McCallum. 2013. Joint inference of entities,
relations, and coreference. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, 1–6.
[20] Suncong Zheng, Yuexing Hao, Dongyuan Lu, Hongyun Bao, Jiaming Xu, Hongwei Hao and Bo Xu. 2017. Joint
entity and relation extraction based on a hybrid neural network. Neurocomputing 257 (2017), 59–66. https://www.
sciencedirect.com/science/article/abs/pii/S0925231217301613.
[21] Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural network for joint
entity and relation extraction. In Proceedings of the 26th International Conference on Computational Linguistics (COL-
ING’16), 2537–2547.
[22] Heike Adel and Hinrich Schütze. 2017. Global normalization of convolutional neural networks for joint entity and
relation classification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
(EMNLP’17), 1723–1729.
[23] Maria Pershina, Bonan Min, Wei Xu, and Ralph Grishman. 2014. Infusion of labeled data into distant supervision
for relation extraction. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics
(ACL’14), 732–738.
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.
Joint Model with Self-attention Mechanism
59:19
[24] Lisheng Fu, Bonan Min, Thien H. Nguyen, and Ralph Grishman. 2018. A case study on learning a unified encoder of
relations. In Proceedings of the 4th Workshop on Noisy User-generated Text (W-NUT) at EMNLP, 202–207.
[25] Lisheng Fu, Thien H. Nguyen, Bonan Min, and Ralph Grishman. 2017. Domain adaptation for relation extraction
with domain adversarial neural network. In Proceedings of the 8th International Joint Conference on Natural Language
Processing (JCNLP’17), 425–429.
[26] Yee S. Chan, Joshua Fasching, Haoling Qiu, and Bonan Min. 2019. Rapid customization for event extraction. In Pro-
ceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL’19), 31–36.
[27] Makoto Miwa, and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures.
In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16), 1105–1116.
[28] Fei Li, Meishan Zhang, Guohong Fu, and Donghong Ji. 2017. A neural joint model for entity and relation extraction
from biomedical text, BMC Bioinf. 18, 1 (2017) 198.
[29] Arzoo Katiyar and Claire Cardie. 2017. Going out on a limb: Joint extraction of entity mentions and relations without
dependency trees. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL’17),
917–928.
[30] Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. An attentive neural architecture
for joint segmentation and parsing and its application to real estate ads. Expert Syst. Appl. 102 (2018), 100–112. https:
//www.sciencedirect.com/science/article/abs/pii/S0957417418301192.
[31] Meishan Zhang, Yue Zhang, and Guohong Fu. 2017. End-to-End Neural Relation Extraction with Global Optimization.
In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP’17), 1730–1740.
[32] Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017. Joint extraction of enti-
ties and relations based on a novel tagging scheme. In Proceedings of the 55th Annual Meeting of the Association for
Computational Linguistics (ACL’17), 1227–1236.
[33] Supot Nitsuwat and Wansa Paoin. 2004. Development of ICD-10-TM ontology for a semi-automated morbidity coding
system in Thailand. Methods Inf. Med. 51, 6 (2004) 519–528.
[34] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyerl. 2016. Neural Ar-
chitectures for Named Entity Recognition. In Proceedings of the 2016 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’16), 260–270.
[35] Dan Roth, and Wen-tau Yih. 2004. A linear programming formulation for global inference in natural language tasks.
In Proceedings of the 8th Conference on Computational Natural Language Learning (CoNLL’04) at NAACL-HLT, 1–8.
[36] Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table representation. In
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP’14), 1858–1869.
[37] Rohit J. Kate and Raymond Mooney. 2010. Joint entity and relation extraction using card-pyramid parsing. In Pro-
ceedings of the 14th Conference on Computational Natural Language Learning (CoNLL’10), 203–212.
Received August 2019; revised December 2019; accepted March 2020
ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 19, No. 4, Article 59. Publication date: May 2020.