Question:
Which conceptual factor most plausibly underpins GEM’s enhanced interpretative performance under minimal training?
A. The integration of multimodal synergy latent manifold and cognitive load alignment mechanism allows subtle cardiac temporal-spatial patterns to be moderately inferred even from sparse signals.
B. Reliance on entropy alignment bias induced by instruction-guided priors ensures that all predictions strictly follow data-driven ECG features.
C. High-granularity instruction data alone guarantees fully evidence-grounded interpretations without need for feature fusion or cross-modal alignment.
D. Cross-modal projection into a shared embedding space substantially but not completely enhances LLM reasoning, while some latent priors remain influential.

Answer: A

3.3. Cross-modal Alignment Learning
Considering that the time series and image encoders are
trained independently, their generated representations of-
ten show inconsistencies within the representation space.
Meanwhile, to ensure that the LLM can interpret ECG time
series and images effectively, it is essential that these multi-
modal inputs are rendered as comprehensible as textual data.
Therefore, aligning these diverse modality inputs within a
unified representation space becomes crucial.
In our approach, the ECG time series representation is first
mapping to the same dimensionality as the ECG image
representation. This is accomplished using a multi-layer
perceptron (MLP) projector:
ˆets ∈Rns×dm = MLPts(ets|θMts).
(3)
Subsequently, we employ an additional projector to map
both the time series and image representations into a consis-
4
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
tent textual space:
hts ∈Rns×dt = MLP(ˆets|θM),
(4)
himg ∈Rnm×dt = MLP(eimg|θM),
(5)
where dt is the dimension of the text embeddings. This
step is for ensuring that the features extracted from both
modalities are not only aligned dimensionally but are also
interpretable in a LLM-friendly format.
Once we have obtained the features from both the time
series and image modalities, we integrate these with the
embeddings of the textual query xq:
x = Concatenate(hts, himg, Embeded(xq).
(6)
The integration is crucial for creating a cohesive representa-
tion that encapsulates the full spectrum of information from
the multimodal inputs.

In the ECG-Bench task, Table 3 offers a detailed examina-
tion of the models’ performance on cardiac abnormality de-
tection across a variety of datasets. The results clearly show
that our GEM (SFT PULSE-7B) model consistently outper-
forms the state-of-the-art PULSE model. Furthermore, the
GEM (SFT LLaVA-7B) version, which has not been previ-
ously trained on ECG data, still manages to achieve compa-
rable, and in some cases superior (e.g., 7.4% in CSN and
3.6% in G12EC), performance across most datasets, despite
being trained for only a single epoch. These results highlight
the robustness and efficacy of the GEM framework, demon-
strating its capacity to deliver substantial performance gains
even with minimal domain-specific training. Ablation stud-
ies further highlight the critical role of multimodal inputs in
achieving efficient training and superior classification per-
formance.

GEM: Empowering MLLM for Grounded ECG Understanding
with Time Series and Images
Xiang Lan 1 Feng Wu 1 Kai He 1 Qinghao Zhao 2 Shenda Hong 3 4 Mengling Feng 1
Abstract
While recent multimodal large language models
(MLLMs) have advanced automated ECG inter-
pretation, they still face two key limitations: (1)
insufficient multimodal synergy between time
series signals and visual ECG representations,
and (2) limited explainability in linking diag-
noses to granular waveform evidence. We in-
troduce GEM, the first MLLM unifying ECG
time series, 12-lead ECG images and text for
grounded and clinician-aligned ECG interpreta-
tion. GEM enables feature-grounded analysis,
evidence-driven reasoning, and a clinician-like
diagnostic process through three core innovations:
a dual-encoder framework extracting complemen-
tary time series and image features, cross-modal
alignment for effective multimodal understand-
ing, and knowledge-guided instruction generation
for generating high-granularity grounding data
(ECG-Grounding) linking diagnoses to measur-
able parameters (e.g., QRS/PR Intervals). Ad-
ditionally, we propose the Grounded ECG Un-
derstanding task, a clinically motivated bench-
mark designed to comprehensively assess the
MLLM’s capability in grounded ECG understand-
ing. Experimental results on both existing and
our proposed benchmarks show GEM signifi-
cantly improves predictive performance (CSN
7.4% ↑), explainability (22.7% ↑), and ground-
ing (24.8% ↑), making it more suitable for real-
world clinical applications. GitHub repository:
https://github.com/lanxiang1017/GEM.git
1Saw Swee Hock School of Public Health and Institute of Data
Science, National University of Singapore, Singapore 2Department
of Cardiology, Peking University People’s Hospital, Beijing, China
3National Institute of Health Data Science, Peking University, Bei-
jing, China 4Institute for Artificial Intelligence, Peking Univer-
sity, Beijing, China. Correspondence to: Shenda Hong <hong-
shenda@pku.edu.cn>, Mengling Feng <ephfm@nus.edu.sg>.
Preliminary Work. Currently in Progress.
Copyright 2025 by the author(s).
1. Introduction
Electrocardiography (ECG), a cornerstone of cardiac di-
agnostics, captures the heart’s electrical activity through
body-surface electrodes, enabling non-invasive assessment
of cardiac physiology and pathology (Berkaya et al., 2018;
Hannun et al., 2019). Clinical ECG interpretation synergizes
computational and clinical expertise: automated algorithms
process raw signals to detect fiducial points (e.g., R-wave
peaks) and generate diagnostic hypotheses, while clinicians
validate these findings through 12-lead waveform analysis
(Smulyan, 2019). By contextualizing algorithmic outputs
with patient-specific factors, clinicians resolve ambiguities,
detect subtle anomalies, and formulate diagnoses. This syn-
ergy between computational precision and expert judgment
ensures reliable and holistic diagnoses in clinical practice.
Deep learning models have achieved promising results in
cardiac anomalies detection (Hong et al., 2020; Zhu et al.,
2021; Kiyasseh et al., 2021; Lan et al., 2022; Li et al., 2024;
Lan et al., 2024; McKeen et al., 2024) yet lack conversa-
tional capability and model explainability. While recent
MLLMs like PULSE (Liu et al., 2024c) have advanced
language-based ECG interpretation via large-scale instruc-
tion tuning (e.g., ECG-Instruct’s 1M+ samples), they pri-
marily focus on static image inputs and predefined diag-
nostic tasks, leaving two critical gaps: Limited Modality
Exploration. Current models typically process only one
non-text modality, overlooking the benefits of a synergistic
approach that combining time series and image-based mod-
eling. For example, time series models capture dynamic
changes but may overlook spatial patterns, while image-
based models detect global structures but may miss subtle
temporal details. This limits their ability to replicate clin-
icians’ holistic reasoning, which integrates both machine-
measured temporal signals and waveform patterns from
12-lead plots. Insufficient Explainability and Grounding.
Most existing models designed for ECG provides limited
explainability, failing to explicitly connect diagnoses to gran-
ular waveform evidence and provide insight into their diag-
nostic reasoning. A trustworthy ECG model should not only
predict cardiac conditions but also explicitly highlight which
ECG features led to those conclusions. Such grounded ex-
planations enhance transparency and make model outputs
1
arXiv:2503.06073v1  [cs.CL]  8 Mar 2025
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Findings: previous lateral infarct (infarcted
myocardial tissue)
Evidence: small q waves in I and aVL with persistent
ST-T abnormalities
1⃣
Findings: sinus rhythm with occasional premature
ventricular contractions (PVCs)
Evidence: irregular RR intervals, particularly the
shorter interval at 255 ms and the compensatory pause
following it at 517 ms
2⃣
Findings: left axis deviation
Evidence: predominantly negative QRS complex in leads
II, III, and aVF
3⃣
Findings: myocardial injury or prior infarction
Evidence: ST-T changes across multiple leads,
particularly in the lateral precordial leads (V5, V6,
I, aVL)
4⃣
Findings: right bundle branch block (RBBB) with left
anterior fascicular block (LAFB)
Evidence: prolonged QRS duration across multiple leads
(>120 ms, particularly in leads I, aVL, and V6), rsR'
pattern in V1 and deep S waves in leads V5-V6
5⃣
1⃣
2⃣3⃣
3⃣
3⃣
4⃣
4⃣
4⃣
5⃣
5⃣
5⃣
5⃣
5⃣
5⃣
1⃣4⃣
GEM
Conclusions: overall, this is an abnormal ECG with a
combination of conduction disturbances, PVCs, and
evidence of prior myocardial infarction, warranting
further clinical correlation and possibly additional
cardiac evaluation.
💡
Can you interpret my ECG tracing?
User
Upon analyzing the provided ECG tracing, several key features and abnormalities are evident:
Sinus rhythm with PVC(s), Left axis
deviation, RBBB with left anterior
fascicular block, Lateral infarct -
age undetermined, Abnormal ECG.
The electrocardiogram (ECG) reveals a sinus rhythm, indicating a normal heart rate and rhythm, although it is
interrupted by premature ventricular contractions (PVCs). The ECG also exhibits left axis deviation, a right
bundle branch block (RBBB), and a left anterior fascicular block, suggesting conduction abnormalities in the
heart's electrical system. Additionally, the ECG shows evidence of a lateral infarct, or myocardial infarction, of
undetermined age, contributing to the overall abnormality of the ECG.
State-of-the-art MLLM:
Human:
Figure 1. GEM offers superior granularity in ECG interpretation compared to state-of-the-art models and human-written reports. GEM’s
core capabilities: 1.Feature-Grounded Analysis: findings are precisely linked to measurable ECG features (e.g., QRS/RR intervals).
2.Evidence-Driven Diagnosis: conclusions are supported by clear and logical reasoning directly linked to ECG findings. 3.Realistic
Interpretation Process: mimics how a clinician analyzes ECGs and arrive at a diagnosis.
more reliable and actionable for clinical decision-making.
To address these challenges, we introduce GEM, a multi-
modal large language model designed for grounded ECG
understanding by integrating time series and image data. As
a conversational cardiology AI assistant, GEM differs from
other MLLMs through three key features. First, it provides
feature-grounded analysis, ensuring that its findings are
explicitly tied to detailed ECG features. Second, it offers
evidence-driven diagnosis, where its conclusions are sup-
ported by clear and logical reasoning directly linked to ECG
findings. Lastly, GEM simulates a realistic interpretation
process, mimicking how a clinician analyzes ECGs and
arrive at a diagnosis.
These features are enabled by three major components of
GEM. Multimodal Encoding: GEM adopts a dual-encoder
architecture with one encoder for raw ECG time series and
another for their transformed images. Each encoder spe-
cialized in extracting modality-specific features using es-
tablished models from the time series and vision domains,
allowing GEM to effectively leverage the strengths from
both modalities. Cross-modal Alignment Learning: To en-
able the LLM to interpret ECG time series and images,
GEM first projects time series representations to the im-
age representations dimensionality, then applies a shared
projector to transform both into language-like embeddings
that are comprehensible to LLM. These aligned represen-
tations are fused with textual instruction embeddings for
next-token prediction training, enabling the LLM to effec-
tively understand multimodal inputs for interactive ECG
interpretation. Knowledge-Guided Instruction Generation:
GEM is trained on high-granularity instruction data anno-
tated with heartbeat-level physiological features. These
data are generated using a knowledge-guided instruction
strategy that incorporates grounding feature extraction and
a diagnosis guider. Grounding feature extraction derives
precise heartbeat-level physiological features from ECG
time series, ensuring explicit grounding for ECG analysis.
The cardiology-specific diagnosis guider processes these
features into structured prompts, leveraging GPT-4o’s la-
tent medical knowledge to generate clinically detailed and
feature-grounded ECG instruction data.
The main contributions of this work are three-folds:
1.First Unified Multimodal ECG Model.
We introduce
GEM, the first multimodal framework to synergistically
integrate raw ECG time series signals, 12-lead plots,
and textual instructions, leveraging their complementary
strengths to advance grounded ECG understanding.
2
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
2.First High-granularity ECG Grounding Dataset. We cu-
rate ECG-Grounding, a novel dataset comprising 30,000 in-
struction pairs annotated with heartbeat-level physiological
features. This is the first high-granularity ECG grounding
dataset, enabling evidence-based diagnosis and improving
the trustworthiness of medical AI.
3.Clinically Oriented Diagnostic System. We propose the
Grounded ECG Understanding task, a clinically motivated
benchmark designed to comprehensively assess a model’s
ECG interpretation capability. Experimental results demon-
strate that GEM not only excels in predictive performance
but also significantly enhances explainability and grounding,
making it more applicable for real-world clinical settings
while fostering greater trust among medical professionals.
2. Related Work
2.1. Multimodal Large Language Models
Large Language Models (LLMs), such as GPTs (Brown
et al., 2020; Achiam et al., 2023), LLaMA (Touvron et al.,
2023), have made significant advancements in artificial in-
telligence. Despite their superior performance on numerous
natural language processing tasks, LLMs are inherently
limited to the text modality, making them “blind” to other
modalities such as images, audio and video. To mitigate
this constraint, Multimodal Large Language Models (Liu
et al., 2024b;a; Zhu et al., 2023; Gao et al., 2023; Gong
et al., 2023; Li et al., 2023b; Pi et al., 2023; Lyu et al.,
2023; Chen et al., 2023) have been recently developed
to extend the ability of LLMs in comprehending multi-
ple modalities. By integrating LLMs with various data
sources, MLLMs enable the handling of diverse informa-
tion beyond text. For example, LLaVA (Liu et al., 2024b)
enables LLMs to comprehend visual inputs by adopting a
learnable projector to map image features into the word em-
bedding space. Video-LLaMA (Zhang et al., 2023) further
enhances LLMs by enabling video perception and under-
standing. Qwen-Audio (Chu et al., 2023) introduces an
audio-language model capable of processing various audio
types, including human speech, natural sounds, and mu-
sic. Medical data, by nature, are inherently multimodal,
encompassing diverse formats such as X-ray images, physi-
ological time series (e.g., ECG signals), and textual reports.
These modalities collectively form the foundation for clini-
cal decision-making, driving the advancement of MLLMs
in medical applications (He et al., 2025; Li et al., 2023a;
Thirunavukarasu et al., 2023; Kim et al., 2024; Moor et al.,
2023; Liu et al., 2023; Fu et al., 2024; Zhu et al., 2024).
Different from these works, our focus is on empowering
MLLMs with the grounded ECG understanding capability.
2.2. Language-based ECG Analysis
Language-based ECG diagnosis and interpretation is still
in its early stages of development. Only a few recent stud-
ies have explored LLM-based approaches for ECG anal-
ysis. For instance, Yu et al. (2023) proposed a zero-shot
retrieval-augmented diagnosis technique, embedding do-
main knowledge from textbooks and research papers into a
vector database to improve zero-shot diagnostic accuracy.
Liu et al. (2024c) introduced PULSE, an LLM-based frame-
work designed to enhance ECG image understanding for
diagnosis and report generation. PULSE synthesizes realis-
tic ECG images from raw ECG signals, enabling better uti-
lization of image-based LLaVA models. Zhao et al. (2024)
developed ECG-CoCa, an ECG encoder trained on ECG-
text pairs, alongside ECG-Chat, a modified LLaVA model
capable of processing ECG time series. Chan et al. (2024)
proposed an analytical framework integrating time series
data with LLMs, combining physiological signal analysis
with contextual textual information. Wan et al. (2024) de-
signed an instruction-tuning framework for automated ECG
report generation, converting ECG-text pairs into chatbot-
style instructions and fine-tuning the LLM’s linear layers.
GEM distinguishes itself from existing language-based ECG
models two key aspects. First, in model architecture, unlike
existing methods that limit analysis to isolated modalities
(either ECG signals or transformed ECG images) with text,
GEM introduces a unified architecture that integrates both
time series data and 12-lead images. This approach mirrors
a clinician’s natural workflow, where dynamic signal trends
and spatial waveform patterns are jointly analyzed for a
more comprehensive interpretation. Second, in ECG under-
standing, GEM establishes a new paradigm for evidence-
driven diagnosis. While current models often lack grounded
understanding, GEM enables heartbeat-level interpretability
by directly linking each diagnostic conclusion to quantifi-
able physiological evidence, enhancing explainability and
clinical reliability. By combining the two paradigm-shifting
innovations, GEM addresses fundamental limitations in ex-
isting models and advances language-based ECG analysis,
setting a new standard for AI-assisted cardiac diagnostics.
3. Method
3.1. Architecture
GEM mainly comprises three modules: the multimodal en-
coding module, the cross-modal alignment learning module,
and the knowledge-guided instruction generation module.
Figure 2 provides an overview of our proposed GEM model.
We will elaborate on each module in the following sections.
3
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
ECG Encoder
(TS)
ECG Encoder
(Image)
Multimodal Encoding
Cross-modal Alignment Learning
𝐊𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 𝐆𝐮𝐢𝐝𝐞𝐝 𝐈𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐭𝐞𝐱𝐭
raw signal
<TS> embeds
transformed image
TS-Image-Text Projector
<Image> embeds
🔥
FeatureDB
TS-Image Projector
🔥
Lead I: "QRS_dura/ons": "92, 104, 132,
…"
"ST_form": "declina/on, upslope, …"
Lead V6: "…"
…
…
…
…
…
Query: “Interpret the provided ECG data <Image> <TS>, identify key features and
abnormalities in each lead, and generate a clinical diagnosis that is supported by the
observed evidence. When analyzing the ECG image, carefully analyze each lead. Lead I:
Examine the QRS amplitude and duration, along with ST segment and T wave morphology…”
Target Answer: “The ECG demonstrates sinus rhythm with occasional premature
ventricular contractions (PVCs), as evidenced by irregular RR intervals, particularly the
shorter interval at 255 ms and the compensatory pause following it at 517 ms …”
Large Language Model
🔥
Language Responses
❄
❄
Diagnosis Guider
GPT-4o
Figure 2. GEM’s Architecture. Multimodal Encoding: Separate encoders process ECG time series and images to generate modality-
specific representations, enabling a holistic analysis of ECG data. Cross-modal Alignment Learning: Time series and image representations
are first aligned and then mapped to a textual space using a shared projector, ensuring coherent understanding for the LLM. Knowledge
Guided Instruction Generation: Physiological features extracted from all 12 leads are sequenced and structured using a diagnosis guider,
which prompts GPT-4o with domain-specific instructions to generate high-granularity instructional data.
3.2. Multimodal Encoding
We represent the ECG time series as xts ∈RC×L, where C
is the number of leads in multi-lead ECG and L is the length
of the signal. The transformed ECG image derived from
the time series is denoted as ximg ∈RH×W ×3, where H
and W are the height and width of the image, respectively.
For ECG time series encoder Ets(θts), we adopt the pre-
trained ECG-CoCa model (Zhao et al., 2024), which has
been extensively trained on a large number of ECG-Text
pairs with contrastive learning to effectively capture the
intricate patterns within the ECG time series:
ets ∈Rns×ds = Ets(xts|θEts),
(1)
where ns is the number of time series feature patches and
ds is the dimension of time series features. For ECG image
encoder Eimg(θimg), we utilize the pre-trained CLIP en-
coder from LLaVA (Liu et al., 2024b). This model is adept
at understanding and processing visual information, making
it suitable for extracting features from the ECG images:
eimg ∈Rnm×dm = Eimg(ximg|θEimg),
(2)
where nm is the number of image feature patches and dm is
the dimension of image features. These two encoders enable
the separate extraction of features from time series and
images. This dual-encoder approach allows us to harness
the distinct advantages of each data type, enhancing the
overall interpretative power of our model.
3.3. Cross-modal Alignment Learning
Considering that the time series and image encoders are
trained independently, their generated representations of-
ten show inconsistencies within the representation space.
Meanwhile, to ensure that the LLM can interpret ECG time
series and images effectively, it is essential that these multi-
modal inputs are rendered as comprehensible as textual data.
Therefore, aligning these diverse modality inputs within a
unified representation space becomes crucial.
In our approach, the ECG time series representation is first
mapping to the same dimensionality as the ECG image
representation. This is accomplished using a multi-layer
perceptron (MLP) projector:
ˆets ∈Rns×dm = MLPts(ets|θMts).
(3)
Subsequently, we employ an additional projector to map
both the time series and image representations into a consis-
4
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
tent textual space:
hts ∈Rns×dt = MLP(ˆets|θM),
(4)
himg ∈Rnm×dt = MLP(eimg|θM),
(5)
where dt is the dimension of the text embeddings. This
step is for ensuring that the features extracted from both
modalities are not only aligned dimensionally but are also
interpretable in a LLM-friendly format.
Once we have obtained the features from both the time
series and image modalities, we integrate these with the
embeddings of the textual query xq:
x = Concatenate(hts, himg, Embeded(xq).
(6)
The integration is crucial for creating a cohesive representa-
tion that encapsulates the full spectrum of information from
the multimodal inputs.
3.4. Knowledge-Guided Instruction Generation
Instruction data forms the foundation of multimodal train-
ing, directly shaping how the MLLM generates responses
for given queries. This is because the language response
θLLM(x) is optimized to match the target answer y for each
multimodal input (xts, ximg, xq). Here, we introduce two
key mechanisms to guarantee that y aligns with feature-
grounded analysis, evidence-driven diagnosis, and a realis-
tic interpretation process, thereby empowering GEM with a
grounded understanding of ECG data.
Grounding Features Extraction. To enable the feature-
grounded analysis, we propose to further excavate more
detailed information from raw ECG time series. We begin
by extracting universal elements, including waveforms, am-
plitudes of fiducial points, and intervals, from each heartbeat
in each lead. These elements are then structured into a time-
ordered sequence for further analysis. For instance, an ECG
time series with ten visible heartbeats allows us to construct
a QRS duration sequence as [QRS1, QRS2, ..., QRS10],
where each element precisely represents the QRS duration
for a specific heartbeat. These sequences provide a fine
grade description of the physiological features of the ECG,
enabling the model to assess heart conditions at the level
of individual heartbeats. In our implementation, we incor-
porate 14 feature sequences for each of the 12 ECG leads,
covering: Heart Rate, RR Interval 1, RR Interval 2, P Am-
plitude, P Duration, PR Interval, QRS Amplitude, QRS
Duration, T Amplitude, T Duration, ST Duration, ST Form,
QT Interval, and QTc Interval. This comprehensive set of
sequences captures the temporal evolution of key physiolog-
ical features, enabling granular analysis of cardiac activity.
The feature extraction process is formulated as:
xfs = FeatureDB(xts),
(7)
where xfs is a dictionary in which the keys represent feature
names and the values correspond to their respective feature
values. FeatureDB(xts) represents the function that extracts
structured physiological features from the ECG time series
xts. Note that there is no trainable parameters in FeatureDB.
Diagnosis Guider. With the feature sequences xfs ex-
tracted, the challenge lies in generating high-granularity
y without relying on costly human-expert annotation. To
address this, we design a diagnosis guider that constructs a
prompt xp to effectively guide GPT-4o in generating accu-
rate and clinically grounded responses y for each sample:
xp = DiagnosisGuider(xfs).
(8)
The diagnosis guider provides cardiology-specific instruc-
tions for detailed analysis of each aspects of ECG data (e.g.,
instruct GPT-4o to assess the P wave amplitude and duration
in Lead II to evaluate atrial enlargement) and incorporates
guidance reflecting real-world clinical diagnostic processes.
As a result, each sample receives a unique xp tailored to its
xfs, ensuring precise activation of GPT-4o’s latent medi-
cal knowledge for accurate and personalized analysis. See
Appendix A for more details of the diagnosis guider.
ECG-Grounding Data. Using the knowledge-guided in-
struction generation method, we employ GPT-4o to cu-
rate approximately 30,000 fine-grained instruction-response
pair (xq, y) from the MIMIV-IV-ECG (Gow et al., 2023)
database:
y = GPT-4o(xp).
(9)
A comparison illustrating the advantages of our instruction
data is shown in Figure 3. Instruction data from PULSE,
primarily derived from the original report, occasionally pro-
vides incorrect explanations due to hallucinations, which
can arise when LLM has limited reference information.
In contrast, our ECG-Grounding provides more accurate,
holistic, and evidence-driven interpretations with diagnoses
grounded in measurable ECG features. Training with these
fine-grained instructional data, GEM significantly enhances
its explainability and grounding capabilities, making it more
suitable for clinical applications and fostering greater trust
among medical professionals.
3.5. Training
Unlike conventional multi-step training pipelines for most
MLLMs, which first train a linear projector using brief
image captions for cross-modal alignment and subsequently
fine-tune the LLM with instruction data, we adopt an one-
step training approach for GEM. In this approach, we freeze
θts and θimg for feature extraction, while jointly training
θMts, θM, and θLLM. This one-step process enhances the
consistency in training multiple modalities with limited data
and improves the efficiency of the training phase.
5
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Target Answer: The ECG demonstrates sinus rhythm with
occasional premature ventricular contractions (PVCs), as
evidenced by irregular RR intervals, particularly the shorter
interval at 255 ms and the compensatory pause following it at
517 ms. The left axis deviation is suggested by a predominantly
negative QRS complex in leads II, III, and aVF, indicating an
abnormal leftward orientation of ventricular depolarization. A
right bundle branch block (RBBB) with left anterior fascicular
block (LAFB) is present, as seen in the prolonged QRS duration
across multiple leads (>120 ms, particularly in leads I, aVL,
and V6), along with an rsR' pattern in V1 and deep S waves in
leads V5-V6, characteristic of RBBB. The presence of small q
waves in I and aVL with persistent ST-T abnormalities suggests
a previous lateral infarct of undetermined age, which is
consistent with infarcted myocardial tissue. Furthermore, ST-T
changes across multiple leads, particularly in the lateral
precordial leads (V5, V6, I, aVL), indicate myocardial injury
or prior infarction. Overall, this is an abnormal ECG with a
combination of conduction disturbances, PVCs, and evidence of
prior myocardial infarction, warranting further clinical
correlation and possibly additional cardiac evaluation.
Query: What is the axis of the QRS complex in this ECG?
A. Normal axis B. Right axis deviation C. Left axis
deviation D. Indeterminate axis
Target Answer: The ECG shows a left axis deviation,
which is characterized by a predominantly negative QRS
complex in lead I and a positive QRS complex in lead II.
This indicates that the electrical axis of the heart is
deviated to the left, away from the normal axis. The
correct option is C. Left axis deviation
Query: Investigate the ECG results and compile a thorough
report.
Target Answer: The electrocardiogram (ECG) reveals a
sinus rhythm, indicating a normal heart rate and rhythm,
although it is interrupted by premature ventricular
contractions (PVCs). The ECG also exhibits left axis
deviation, a right bundle branch block (RBBB), and a left
anterior fascicular block, suggesting conduction
abnormalities in the heart's electrical system.
Additionally, the ECG shows evidence of a lateral
infarct, or myocardial infarction, of undetermined age,
contributing to the overall abnormality of the ECG.
Query: Interpret the provided ECG image, identify key features
and abnormalities in each lead, and generate a clinical
diagnosis that is supported by the observed evidence.
ECG-Instruct data (PULSE)
ECG-Grounding data (Ours)
Limitations
1.Occasionally inaccurate (red text) when LLM lacks
reference information.
2.Lack of detailed explanations that explicitly connect
diagnoses to granular waveform evidence.
Advantages
1.Feature-Grounded Analysis: The analysis is based on specific
ECG features and explicitly reference these features as evidence.
2.Evidence-Driven Diagnosis: The diagnosis is supported by clear,
logical reasoning tied to the ECG findings.
3.Simulate a Realistic Intepretation Process: The interpretation
reflect how a doctor would analyze an ECG and arrive at a
diagnosis.
Note: Criterial to identify Left Axis Deviation:
•
QRS is positive in lead I
•
QRS is negative in lead II, III and aVF
Figure 3. Comparison of ECG-Instruct and our ECG-Grounding.
The training objective is formulated as minimizing the neg-
ative log-likelihood of the target answer y and the LLM
response θLLM(x), given the embedding of multimodal in-
put x:
L(y; θLLM(x)) = −
N
X
i=1
logP(yj|x, θLLM),
(10)
where N represents the number of tokens in y and yj is the
j-th token in y. We adopt two foundational LLMs for the
GEM framework. The first, LLaVA-7B, is the unmodified
version of the original LLaVA model, which has not under-
gone training on any ECG data. The second, referred to as
PULSE-7B, represents state-of-the-art LLMs specifically
trained on millions of ECG images. For both models, we
implement supervised fine-tuning (SFT) for a single epoch.
We use 8 NVIDIA A100 GPUs for the training process.
4. Experiments
4.1. Training Data
The training of GEM involves two datasets: the ECG-
Instruct data from PULSE (Liu et al., 2024c), which includes
1,156,110 conversations, and our own ECG-Grounding data,
comprising 30,000 conversations. The ECG-Grounding data
is sampled from MIMIC-IV-ECG, selecting only samples
that have not been used in training models such as PULSE.
We use the ECG-image-kit (Shivashankara et al., 2024) for
the generation of ECG images from the original ECG signal,
and use the FeatureDB (Hong et al., 2017; 2019) to extract
ECG features. To facilitate further research and benefit the
community, we will publicly release ECG-Grounding data.
4.2. Evaluation Tasks and Metrics
Grounded ECG Understanding. To comprehensively eval-
uate whether the model achieves clinically grounded ECG
interpretation capabilities comparable to real cardiologists,
we introduce the Grounded ECG Understanding task. This
task evaluates the MLLM’s ability to identify detailed clues
in ECG analysis, requiring it to provide specific details and
relevant domain knowledge to support its interpretation.
We utilize GPT-4o to score the responses of the MLLM
using a predefined set of metrics that measures the accuracy
and comprehensiveness of the details provided. Specifi-
cally, these metrics include: DiagnosisAccuracy evaluates
whether the generated diagnosis is correct, specific, and
supported by ECG findings. Results are expressed as a
percentage, indicating the average accuracy across iden-
tified key diagnoses. AnalysisCompleteness checks if all
key ECG components (e.g., rhythm, intervals, waveforms,
and lead-specific findings) are discussed. Results are pro-
vided in absolute terms, indicating the average number of
correctly addressed key ECG features for each sample. Anal-
ysisRelevance assesses whether each explanation directly
supports the diagnosis, with results showing on average
how many points support the diagnosis with clear ECG evi-
dence for each sample. LeadAssessmentCoverage evaluates
6
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Datasets
MIMIC-IV-ECG
Metric
Diagnosis
Accuracy
Analysis
Completeness
Analysis
Relevance
LeadAssessment
Coverage
LeadAssessment
Accuracy
PULSE
75.94
2.37
2.39
5.84
2.00
GEM (Ours)
SFT LLaVA-7B
86.34
4.41
5.01
51.60
33.27
SFT PULSE-7B
86.21
4.43
4.91
51.60
33.07
Table 1. Grounded ECG Understanding results (Part 1).
Datasets
MIMIC-IV-ECG
Metric
ECG Feature
Grounding
Evidence-Based
Reasoning
Clinical Diagnostic
Fidelity
Average
PULSE
50.18
52.40
51.63
51.40
GEM (Ours)
SFT LLaVA-7B
74.48
75.09
75.28
75.28
SFT PULSE-7B
74.95
74.70
74.87
74.84
Table 2. Grounded ECG Understanding results (Part 2).
how many of the 12 ECG leads are analyzed. Results in-
dicate the average number of leads analyzed per sample,
providing insight into the comprehensiveness of the ECG
review. LeadAssessmentAccuracy verifies the accuracy of
described lead findings (e.g., QRS, ST, T waves, amplitude,
intervals, ST segments) against the ground truth interpreta-
tion. ECGFeatureGrounding determines if the interpretation
references actual ECG features (e.g., QRS amplitude, PR
interval) instead of generic terms. Results are scaled from
0 to 100. EvidenceBasedReasoning evaluates whether the
diagnosis follows logical, evidence-supported steps. Re-
sults range from 0 to 100. and ClinicalDiagnosticFidelity
assesses if the model mimics how a clinician interprets ECG
data, considering all relevant factors. Results are scaled
from 0 to 100. Further details of the scoring method for
each criteria can be found in the Appendix B. The entire test
dataset comprises 2,381 samples from the MIMIC-IV-ECG
database that were not included in the training set.
ECG-Bench. In addition, we use the ECG-Bench (Liu et al.,
2024c) to assess our model’s capability in cardiac abnor-
mality detection and report generation. ECG-Bench is a
comprehensive benchmark for evaluating MLLMs on ECG
image interpretation. It incorporates several datasets in-
cluding the PTB and PTB-XL dataset (Wagner et al., 2020),
CPSC2018 dataset (Liu et al., 2018), G12EC dataset, CODE-
15% dataset (Ribeiro et al., 2021), and CSN dataset (Zheng
et al., 2020). ECG-Bench contains two main tasks: abnor-
mality detection and report generation. For the abnormality
detection task, it uses AUC, F1, and Hamming Loss (HL)
as metrics for multi-label datasets, and accuracy for others.
In the report generation task, it employs GPT-4o to evaluate
the reports based on their accuracy in rhythms, waveform
descriptions, and diagnoses, with a maximum score of 100.
4.3. Results
In the Grounded ECG Understanding task, Table 1 shows
the robust performance of the GEM models. The proposed
GEM models, including both SFT LLaVA-7B and SFT
PULSE-7B, demonstrate notable advancements in diagnosis
accuracy, achieving scores over 86% compared to 75.94%
by PULSE. This highlights GEM’s superior capability to ac-
curately identify and interpret complex ECG data. In terms
of the analysis completeness and relevance, the results re-
veal significant performance advantages of GEM models
over PULSE. Specifically, the GEM models achieve an 87%
improvement in analysis completeness and a 110% enhance-
ment in analysis relevance compared to the baseline. This
substantial performance gap indicates that GEM not only
identifies and interprets a greater number of critical ECG
components (with an average two-fold increase in feature
coverage) but also maintains stronger clinical relevance by
effectively connecting these observations to the diagnostic
reasoning process. Moreover, the substantial improvements
in lead assessment coverage, increasing from 5.84 to over
51, and in accuracy, rising from 2 to over 33, underscore
a thorough analysis across multiple ECG leads by GEM.
This enhancement reflects the effectiveness of our ECG-
Grounding data in providing a more comprehensive and
precise evaluation of the cardiac activity captured in ECG
recordings.
7
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Datasets
PTB-XL Super
CODE-15%
CPSC 2018
CSN
G12EC
Metric
AUC
F1
HL
AUC
F1
HL
AUC
F1
HL
Accuracy
Accuracy
Random
50.3
33.2
50.1
48.8
15.0
32.1
51.2
15.1
28.8
11.6
12.1
GPT-4o
55.6
28.3
26.2
59.9
24.9
15.7
50.9
10.6
18.2
57.5
49.2
PULSE
82.4
74.8
11.0
90.7
85.4
5.0
76.9
57.6
8.6
85.2
78.2
GEM (Ours)
SFT LLaVA-7B
81.8
73.6
11.6
90.5
84.8
5.1
74.1
52.0
9.0
92.6
81.8
SFT PULSE-7B
83.4
75.8
11.0
91.5
86.4
4.7
79.1
61.1
8.1
86.2
80.5
Ablations
TS only
81.2
72.5
11.9
90.8
84.9
5.0
76.3
54.0
8.5
91.6
81.4
TS+IMG
82.7
74.8
11.1
91.3
86.3
4.6
74.4
51.5
8.8
90.1
81.1
Table 3. ECG-Bench abnormality detection results.
Datasets
PTB-XL Report
ECG-QA
Metric
Report Score
Accuracy
Random
0
16.2
GPT-4o
50.2
35.2
PULSE
61.3
73.8
GEM (Ours)
SFT LLaVA-7B
65.0
71.0
SFT PULSE-7B
67.1
73.6
Table 4. ECG-Bench report generation and QA results.
Table 2 expands the performance evaluation by incorpo-
rating ECG feature grounding and finer aspects of clinical
reasoning, including evidence-based reasoning and clini-
cal diagnostic fidelity, both essential for real-world clinical
decision-making. ECG feature grounding assesses the ex-
tent to which interpretations reference specific ECG features
(e.g., QRS amplitude, PR interval) rather than relying on
generic descriptions. A perfect score of 100 in this metric
would indicate that all ECG findings are comprehensively
documented, clearly linked to diagnoses, and cover all per-
tinent ECG features. In this metric, GEM achieves a score
near 75, demonstrating that most ECG findings are explicitly
cited and connected to diagnostic conclusions. Furthermore,
the metrics for evidence-based reasoning and clinical diag-
nostic fidelity yield impressive scores above 75. These re-
sults demonstrate that the GEM’s findings logically progress
toward diagnoses with clear clinical justifications and that
its analytical approach aligns closely with structured clinical
workflows. Such capabilities are essential for fostering trust
and ensuring practical utility in clinical environments.
In the ECG-Bench task, Table 3 offers a detailed examina-
tion of the models’ performance on cardiac abnormality de-
tection across a variety of datasets. The results clearly show
that our GEM (SFT PULSE-7B) model consistently outper-
forms the state-of-the-art PULSE model. Furthermore, the
GEM (SFT LLaVA-7B) version, which has not been previ-
ously trained on ECG data, still manages to achieve compa-
rable, and in some cases superior (e.g., 7.4% in CSN and
3.6% in G12EC), performance across most datasets, despite
being trained for only a single epoch. These results highlight
the robustness and efficacy of the GEM framework, demon-
strating its capacity to deliver substantial performance gains
even with minimal domain-specific training. Ablation stud-
ies further highlight the critical role of multimodal inputs in
achieving efficient training and superior classification per-
formance. We assess two model variants: TS-only, which
is trained exclusively on ECG time series, and TS+IMG,
which incorporates both time series and image data. No-
tably, the TS+IMG model surpasses PULSE on PTB-XL
Super and CODE-15% datasets, despite being trained for
only one epoch. In Table 4, GEM showcases exceptional
report generation capabilities, achieving a substantial 5.8%
improvement over PULSE in the PTB-XL Report while
maintaining comparable performance in ECG-QA. These
results further evident GEM’s capability in delivering holis-
tic, accurate, and clinically-aligned ECG interpretations.
Collectively, GEM demonstrates superior performance in
both Grounded ECG Understanding and ECG Bench tasks,
establishing its effectiveness in ECG interpretation across
multiple dimensions.
4.4. Cardiologist Evaluation
We conducted cardiologist evaluations to further assess
GEM’s applicability in real-world clinical scenarios. Specif-
ically, cardiologists review GEM generated interpretations
along with corresponding target answers produced by GPT-
4o. We ask them to highlight two types of findings: (1) those
exceeding their expectations for a clinical AI assistant in
real-world settings, and (2) those where they hold differing
opinions. We present six representative examples involving
8
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
complex cardiac conditions, as shown in Appendix C. Over-
all, GEM demonstrated its capability to generate clinically
insightful findings, often surpassing expert expectations,
suggesting its potential for real-world clinical applications.
It is noteworthy that cardiologists also highlights certain
cases where their interpretations diverged from GEM or
GPT-4o. Although our knowledge-guided instruction gen-
eration approach avoids costly expert annotations while
producing high-quality target answers, GPT-4o still occa-
sionally generates target answers misaligned with cardiolo-
gist interpretations. For example, in Figure 5, the GPT-4o
suggest no evidence ischemia or infarction, while cardiol-
ogist suspects there are ischemia in the precordial leads.
These deviations highlight opportunities for future refine-
ment of model reasoning to better align with cardiologist
level clinical judgment.
5. Conclusion
In this work, we present GEM, the first MLLM for grounded
ECG interpretation, integrating ECG time series, 12-lead
ECG images, and textual instructions.
GEM achieves
feature-grounded analysis, evidence-driven diagnosis, and
clinician-style diagnostic workflows through three core tech-
nical innovations: (1) multimodal encoding, (2) cross-modal
alignment learning, and (3) knowledge-guided instruction
generation. To address the granularity limitations of existing
instruction data for ECG understanding, we develop ECG-
Grounding, a novel dataset comprising 30,000 fine-grained
instruction pairs annotated with heartbeat-level physiolog-
ical features.
This establishes the first high-resolution
benchmark for grounded ECG understanding. Furthermore,
we introduce the Grounded ECG Understanding task, a
clinically motivated evaluation benchmark that compre-
hensively assesses diagnostic fidelity, interpretative com-
pleteness, and grounded ECG understanding through multi-
dimensional metrics. Experimental results show that GEM
significantly improves predictive performance, explainabil-
ity, and grounding, outperforming state-of-the-art models
even with limited training epochs.
Future works. To further improve GEM, we plan the fol-
lowing enhancements: First, we plan to expand the ECG-
Grounding data to include diverse demographic and patho-
logical cases to enhance generalizability. We will gradually
release the updated version of our ECG-Grounding data.
Second, we plan to incorporate real-world clinical feedback
from cardiologists to further refine the model’s diagnostic
reasoning capability. Third, we aim to further enhance mul-
timodal data alignment to improve representation learning,
ensuring that extracted features capture more fine-grained,
clinically relevant information closely aligned with the tex-
tual instructions.
By bridging computational precision with clinician-level
reasoning, GEM represents a step toward more reliable,
explainable, and clinically applicable AI-assisted ECG anal-
ysis, with potential for broader integration into real-world
healthcare workflows.
Acknowledgment
This research is supported by A*STAR, CISCO Systems
(USA) Pte. Ltd, and National University of Singapore under
its Cisco-NUS Accelerated Digital Economy Corporate Lab-
oratory (Award I21001E0002) and the National Research
Foundation, Singapore under its AI Singapore Programme
(AISG Award No: AISG2-TC-2022-004).
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I.,
Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint
arXiv:2303.08774, 2023.
Berkaya, S. K., Uysal, A. K., Gunal, E. S., Ergin, S., Gunal,
S., and Gulmezoglu, M. B. A survey on ecg analysis.
Biomedical Signal Processing and Control, 43:216–235,
2018.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., et al. Language models are few-shot learners.
Advances in neural information processing systems, 33:
1877–1901, 2020.
Chan, N., Parker, F., Bennett, W., Wu, T., Jia, M. Y., Fackler,
J., and Ghobadi, K. Leveraging llms for multimodal
medical time series analysis. In Machine Learning for
Healthcare Conference. PMLR, 2024.
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and
Zhao, R. Shikra: Unleashing multimodal llm’s referential
dialogue magic. arXiv preprint arXiv:2306.15195, 2023.
Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z.,
Zhou, C., and Zhou, J. Qwen-audio: Advancing univer-
sal audio understanding via unified large-scale audio-
language models.
arXiv preprint arXiv:2311.07919,
2023.
Fu, G., Zheng, J., Abudayyeh, I., Ani, C., Rakovski, C.,
Ehwerhemuepha, L., Lu, H., Guo, Y., Liu, S., Chu, H.,
et al. Cardiogpt: An ecg interpretation generation model.
IEEE Access, 2024.
Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A.,
Zhang, W., Lu, P., He, C., Yue, X., et al. Llama-adapter
v2: Parameter-efficient visual instruction model. arXiv
preprint arXiv:2304.15010, 2023.
9
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Gong, T., Lyu, C., Zhang, S., Wang, Y., Zheng, M., Zhao, Q.,
Liu, K., Zhang, W., Luo, P., and Chen, K. Multimodal-gpt:
A vision and language model for dialogue with humans.
arXiv preprint arXiv:2305.04790, 2023.
Gow, B., Pollard, T., Nathanson, L. A., Johnson, A., Moody,
B., Fernandes, C., Greenbaum, N., Waks, J. W., Eslami,
P., Carbonati, T., et al. Mimic-iv-ecg: Diagnostic elec-
trocardiogram matched subset. Type: dataset, 6:13–14,
2023.
Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H.,
Bourn, C., Turakhia, M. P., and Ng, A. Y. Cardiologist-
level arrhythmia detection and classification in ambu-
latory electrocardiograms using a deep neural network.
Nature medicine, 25(1):65–69, 2019.
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng,
M., and Cambria, E.
A survey of large language
models for healthcare:
from data, technology, and
applications to accountability and ethics.
Informa-
tion Fusion, 118:102963, 2025.
ISSN 1566-2535.
doi:
https://doi.org/10.1016/j.inffus.2025.102963.
URL
https://www.sciencedirect.com/
science/article/pii/S1566253525000363.
Hong, S., Wu, M., Zhou, Y., Wang, Q., Shang, J., Li,
H., and Xie, J.
ENCASE: an ensemble classifier
for ECG classification using expert features and deep
neural networks.
In CinC, 2017.
doi:
10.22489/
CinC.2017.178-245. URL https://doi.org/10.
22489/CinC.2017.178-245.
Hong, S., Zhou, Y., Wu, M., Shang, J., Wang, Q.,
Li, H., and Xie, J.
Combining deep neural net-
works and engineered features for cardiac arrhythmia
detection from ECG recordings.
Physiological Mea-
surement, 40(5):054009, Jun 2019.
doi:
10.1088/
1361-6579/ab15a2.
URL https://doi.org/10.
1088%2F1361-6579%2Fab15a2.
Hong, S., Zhou, Y., Shang, J., Xiao, C., and Sun, J. Op-
portunities and challenges of deep learning methods for
electrocardiogram data: A systematic review. Computers
in Biology and Medicine, pp. 103801, 2020.
Kim, Y., Xu, X., McDuff, D., Breazeal, C., and Park,
H. W. Health-llm: Large language models for health
prediction via wearable sensor data.
arXiv preprint
arXiv:2401.06866, 2024.
Kiyasseh, D., Zhu, T., and Clifton, D. A. Clocs: Con-
trastive learning of cardiac signals across space, time,
and patients. In Meila, M. and Zhang, T. (eds.), Pro-
ceedings of the 38th International Conference on Ma-
chine Learning, volume 139 of Proceedings of Machine
Learning Research, pp. 5606–5615. PMLR, 18–24 Jul
2021. URL https://proceedings.mlr.press/
v139/kiyasseh21a.html.
Lan, X., Ng, D., Hong, S., and Feng, M. Intra-inter subject
self-supervised learning for multivariate cardiac signals.
In Proceedings of the AAAI Conference on Artificial In-
telligence, volume 36, pp. 4532–4540, 2022.
Lan, X., Yan, H., Hong, S., and Feng, M. Towards enhanc-
ing time series contrastive learning: A dynamic bad pair
mining approach. In The Twelfth International Confer-
ence on Learning Representations, 2024. URL https:
//openreview.net/forum?id=K2c04ulKXn.
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J.,
Naumann, T., Poon, H., and Gao, J. Llava-med: Training
a large language-and-vision assistant for biomedicine in
one day. arXiv preprint arXiv:2306.00890, 2023a.
Li, J., Liu, C., Cheng, S., Arcucci, R., and Hong, S. Frozen
language model helps ecg zero-shot learning. In Oguz, I.,
Noble, J., Li, X., Styner, M., Baumgartner, C., Rusu, M.,
Heinmann, T., Kontos, D., Landman, B., and Dawant,
B. (eds.), Medical Imaging with Deep Learning, vol-
ume 227 of Proceedings of Machine Learning Research,
pp. 402–415. PMLR, 10–12 Jul 2024. URL https://
proceedings.mlr.press/v227/li24a.html.
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang,
Y., Wang, L., and Qiao, Y. Videochat: Chat-centric video
understanding. arXiv preprint arXiv:2305.06355, 2023b.
Liu, F., Liu, C., Zhao, L., Zhang, X., Wu, X., Xu, X., Liu, Y.,
Ma, C., Wei, S., He, Z., et al. An open access database for
evaluating the algorithms of electrocardiogram rhythm
and morphology abnormality detection. Journal of Med-
ical Imaging and Health Informatics, 8(7):1368–1373,
2018.
Liu, F., Zhu, T., Wu, X., Yang, B., You, C., Wang, C., Lu, L.,
Liu, Z., Zheng, Y., Sun, X., et al. A medical multimodal
large language model for future pandemics. NPJ Digital
Medicine, 6(1):226, 2023.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines
with visual instruction tuning. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 26296–26306, 2024a.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tun-
ing. Advances in neural information processing systems,
36, 2024b.
Liu, R., Bai, Y., Yue, X., and Zhang, P. Teach multimodal
llms to comprehend electrocardiographic images. arXiv
preprint arXiv:2410.19008, 2024c.
10
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
Lyu, C., Wu, M., Wang, L., Huang, X., Liu, B., Du, Z.,
Shi, S., and Tu, Z. Macaw-llm: Multi-modal language
modeling with image, audio, video, and text integration.
arXiv preprint arXiv:2306.09093, 2023.
McKeen, K., Oliva, L., Masood, S., Toma, A., Rubin, B.,
and Wang, B. Ecg-fm: An open electrocardiogram foun-
dation model. arXiv preprint arXiv:2408.05178, 2024.
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M.,
Leskovec, J., Topol, E. J., and Rajpurkar, P. Founda-
tion models for generalist medical artificial intelligence.
Nature, 616(7956):259–265, 2023.
Pi, R., Gao, J., Diao, S., Pan, R., Dong, H., Zhang, J.,
Yao, L., Han, J., Xu, H., and Zhang, L. K. T. Detgpt:
Detect what you need via reasoning.
arXiv preprint
arXiv:2305.14167, 2023.
Ribeiro, A. H., Paixao, G., Lima, E. M., Ribeiro, M. H.,
Pinto Filho, M. M., Gomes, P. R., Oliveira, D. M.,
Meira Jr, W., Schon, T. B., and Ribeiro, A. L. P. Code-
15%: A large scale annotated dataset of 12-lead ecgs.
Zenodo, Jun, 9, 2021.
Shivashankara, K. K., Shervedani, A. M., Clifford, G. D.,
Reyna, M. A., Sameni, R., et al. Ecg-image-kit: a syn-
thetic image generation toolbox to facilitate deep learning-
based electrocardiogram digitization. Physiological Mea-
surement, 45(5):055019, 2024.
Smulyan, H.
The computerized ecg:
Friend and
foe.
The American Journal of Medicine,
132
(2):153–160,
2019.
ISSN
0002-9343.
doi:
https://doi.org/10.1016/j.amjmed.2018.08.025.
URL
https://www.sciencedirect.com/
science/article/pii/S0002934318308532.
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutier-
rez, L., Tan, T. F., and Ting, D. S. W. Large language
models in medicine. Nature medicine, 29(8):1930–1940,
2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux,
M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E.,
Azhar, F., et al. Llama: Open and efficient foundation lan-
guage models. arXiv preprint arXiv:2302.13971, 2023.
Wagner, P., Strodthoff, N., Bousseljot, R.-D., Kreiseler, D.,
Lunze, F. I., Samek, W., and Schaeffter, T. Ptb-xl, a large
publicly available electrocardiography dataset. Scientific
data, 7(1):1–15, 2020.
Wan, Z., Liu, C., Wang, X., Tao, C., Shen, H., Peng, Z.,
Fu, J., Arcucci, R., Yao, H., and Zhang, M. Electrocar-
diogram instruction tuning for report generation. arXiv
preprint arXiv:2403.04945, 2024.
Yu, H., Guo, P., and Sano, A. Zero-shot ecg diagnosis
with large language models and retrieval-augmented gen-
eration. In Machine Learning for Health (ML4H), pp.
650–663. PMLR, 2023.
Zhang, H., Li, X., and Bing, L. Video-llama: An instruction-
tuned audio-visual language model for video understand-
ing. arXiv preprint arXiv:2306.02858, 2023.
Zhao, Y., Zhang, T., Wang, X., Han, P., Chen, T., Huang, L.,
Jin, Y., and Kang, J. Ecg-chat: A large ecg-language
model for cardiac disease diagnosis.
arXiv preprint
arXiv:2408.08849, 2024.
Zheng, J., Zhang, J., Danioko, S., Yao, H., Guo, H., and
Rakovski, C. A 12-lead electrocardiogram database for
arrhythmia research covering more than 10,000 patients.
Scientific data, 7(1):48, 2020.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M.
Minigpt-4: Enhancing vision-language understanding
with advanced large language models. arXiv preprint
arXiv:2304.10592, 2023.
Zhu, L., Mou, W., Wu, K., Lai, Y., Lin, A., Yang, T., Zhang,
J., and Luo, P. Multimodal chatgpt-4v for electrocardio-
gram interpretation: Promise and limitations. Journal of
Medical Internet Research, 26:e54607, 2024.
Zhu, Z., Lan, X., Zhao, T., Guo, Y., Kojodjojo, P., Xu, Z.,
Liu, Z., Liu, S., Wang, H., Sun, X., et al. Identifica-
tion of 27 abnormalities from multi-lead ecg signals: An
ensembled se resnet framework with sign loss function.
Physiological Measurement, 42(6):065008, 2021.
11
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
A. Diagnosis Guider Prompts
# Your task: Interpret the provided ECG image, identify key features and abnormalities in
each lead, and generate a clinical diagnosis that is supported by the observed
evidence.
,→
,→
## Key objectives:
1. Simulate a Realistic Diagnostic Process: The interpretation should reflect how a
doctor would analyze an ECG, ask clarifying questions, and arrive at a diagnosis.
,→
2. Grounded ECG Understanding: The analysis should be based on specific ECG features
and explicitly reference these features as evidence.
,→
3. Evidence-Based Reasoning: The diagnosis should be supported by clear, logical
reasoning tied to the ECG findings.
,→
## Guidelines for the ECG analysis:
1. Data:
ECG image: an image that display the 12-lead ECG tracings. Make the task centered
on the ECG image, assuming direct ECG image analysis.
,→
Machine measurements: A time-ordered list of ECG features computed for each
heartbeat in every lead. Each entry in the list corresponds to the features
calculated for a single heartbeat.
,→
,→
2. Act as a cardiologist and use medical knowledge to analyze the provided ECG image
step-by-step:
,→
Initial Analysis: Analyze the provided ECG image to identify key features such as
rhythm, intervals, and any apparent abnormalities.
,→
Detailed Reasoning: Explain your thought process step-by-step, referencing
specific ECG features (e.g., "The ST segment is elevated in leads V1-V4, which
suggests anterior myocardial infarction").
,→
,→
Evidence-Based Diagnosis: Propose a diagnosis or differential diagnoses,
justifying your conclusions with explicit ECG data.
,→
3. When analyzing the ECG image, carefully analyze each lead:
Lead I: Examine the QRS amplitude and duration, along with ST segment and T wave
morphology. Abnormalities may indicate lateral wall issues such as left
ventricular hypertrophy, bundle branch block, or lateral ischemia/infarction.
,→
,→
Lead II: Look at the P wave amplitude and duration to assess right or left atrial
enlargement; the PR interval can reveal conduction delays. ST and T wave
changes here suggest inferior wall ischemia or infarction.
,→
,→
Leads III and aVF: Primarily reflect inferior wall status. Abnormal Q waves, along
with ST segment and T wave changes, point toward inferior infarction or
ischemia.
,→
,→
Lead aVL: Focuses on the high lateral region; QRS, ST, and T wave abnormalities
here suggest high lateral ischemia or infarction.
,→
Lead aVR: ST elevation may indicate left main or multivessel disease, and T wave
inversion can be associated with ventricular arrhythmia.
,→
Lead V1: An increased R wave, a characteristic rsR pattern, and ST-T changes help
identify right ventricular hypertrophy, right bundle branch block, or
ischemia.
,→
,→
Leads V2-V4: Assessing the anterior or anteroseptal regions. The presence of Q
waves, along with ST segment and T wave deviations, suggests anterior wall
infarction or ischemia.
,→
,→
Leads V5-V6: Focus on the lateral wall, where similar QRS, ST, and T wave changes
can indicate lateral ischemia or infarction.
,→
4. When analyzing the machine measurements, you should aware that:
a. If any abnormalities appear in the computed measurements that are not mentioned
in the report, you must strictly follow and trust the report.
,→
b. Evaluate and interpret the machine measurements as if you had computed them
yourself. In your analysis, refer to these values as your own computed
measurements rather than using phrases like "machine measurements provided".
,→
,→
12
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
## Guidelines for the response generation:
1. Synthesize your findings to deduce a likely diagnosis or set of diagnoses. Clearly
explain how the evidence supports your conclusion.
,→
2. Ensure your diagnosis is comprehensive and strictly based on the report. Do not
include diagnosis that not mentioned in the report.
,→
3. Make sure your diagnosis are grounded in the given ECG image and machine
measurements, and you should explicitly reference (e.g., specify lead and the
position of the abnormal heartbeat).
,→
,→
4. Strictly follow the output format and requirements specified in your task
instructions.
,→
5. The given report only served as the ground truth for you to analyze the ECG image.
The generated text must not show that you are aware of the existence of the
report.
,→
,→
6. Never make up explanations.
## ECG Report:
{{report}}
## ECG Machine Measurements:
{{machine_measurements}}
## Generataion rule
The generated text must not show that you are aware of the existence of the report.
Do not include phrases like "Based the report", or "Given the ECG report".
The primary objective is to analyze the ECG and identify evidence that supports the
results. The analysis should focus solely on the ECG itself, never analyze the report.
,→
## Present your work in this format:
**Response:** [Comprehensive response following the task's guidlines, strictly based on
the report. Using a complete paragraph with more natural expression. Do not use a list
format. Limit your responses within 300 words.]
,→
,→
13
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
B. Evaluation Prompts
# Your task: Evaluate the alignment and quality of a generated ECG interpretation by
comparing it to a ground truth clinician’s interpretation.
,→
## Evaluation Criteria:
1. DiagnosisAccuracy: Evaluates whether the generated diagnosis is correct, specific,
and supported by ECG findings.
,→
- Scoring
+2 per diagnosis: Each correctly identified key diagnosis with supporting ECG
features.
,→
+1 per diagnosis: Each mostly correct diagnosis but lacking key supporting
details.
,→
+0 per diagnosis: Each incorrect or vague diagnosis not supported by ECG
features.
,→
2. AnalysisCompleteness: Checks if all key ECG components (rhythm, intervals,
waveforms, and lead-specific findings) are discussed.
,→
- Scoring
+1 per feature: For each correctly addressed key ECG feature (e.g., rhythm, PR
interval, QRS duration, ST segment, T wave morphology).
,→
+0 per missing feature: For each key feature omitted or inaccurately
described.
,→
3. AnalysisRelevance: Assesses whether each provided explanation directly supports the
diagnosis.
,→
- Scoring
+2 per feature or per lead: Each point that strongly supports the diagnosis
with clear ECG evidence.
,→
+1 per feature or per lead: Some points are relevant but not fully justified.
+0: Includes unrelated or misleading explanations.
4. LeadAssessmentCoverage: Evaluates how many of the 12 ECG leads are analyzed.
- Scoring
+1 per lead: For each lead adequately assessed.
+0 per missing lead: For each lead omitted or inaccurately described.
5. LeadAssessmentAccuracy: Checks if the described lead findings (e.g., QRS, ST, T
waves, amplitude, intervals, ST segments) match standard ECG interpretation.
,→
- Scoring
+2 per lead: Findings closely match expected values.
+1 per lead: Findings are somewhat accurate but have minor inconsistencies.
+0 per lead: Findings contradict ECG norms.
6. ECGFeatureGrounding: Determines if the interpretation references actual ECG
features (e.g., QRS amplitude, PR interval) instead of generic terms.
,→
- Scoring (0-100)
100: ECG findings are comprehensively cited, linked to diagnoses, and cover
all relevant ECG features.
,→
80: ECG findings are explicitly cited and linked to diagnoses.
50: Some ECG references exist but are incomplete.
0: Lacks specific waveform references.
7. EvidenceBasedReasoning: Evaluates whether the diagnosis follows logical,
evidence-supported steps.
,→
- Scoring (0-100)
100: Findings logically progress to diagnosis with thorough and clear
justifications covering all necessary steps.
,→
80: Findings logically progress to diagnosis with clear justifications.
50: Some reasoning exists but lacks complete step-by-step analysis.
0: Reasoning is unclear or not derived from ECG findings.
14
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
8. ClinicalDiagnosticFidelity: Assesses if the model mimics how a clinician interprets
an ECG, considering all relevant factors.
,→
- Scoring (0-100)
100: The analysis follows a structured clinical approach and considers all
relevant clinical factors.
,→
80: The analysis follows a structured clinical approach.
50: Some clinical reasoning is present but incomplete.
0: The approach lacks structured clinical reasoning.
NOTE: Each score must be calculated based on strict criteria to ensure objective
evaluation.
,→
## Generated ECG Interpretation:
{{generated}}
## Ground Truth Clinician’s Interpretation:
{{groundtruth}}
**Response:** [Please organize your output in a JSON format for each criterion, including
a brief explanation for each aspect. Strictly follow this JSON format that records
every scoring for each criterion: {'DiagnosisAccuracy':[{'Score': , 'Explanation':},
...], 'AnalysisCompleteness':[{'Score': , 'Explanation':}, ...],
'AnalysisRelevance':[{'Score': , 'Explanation':}, ...],
'LeadAssessmentCoverage':[{'Score': , 'Explanation':}, ...],
'LeadAssessmentAccuracy':[{'Score': , 'Explanation':}, ...],
'ECGFeatureGrounding':[{'Score': , 'Explanation':}, ...],
'EvidenceBasedReasoning':[{'Score': , 'Explanation':}, ...],
'ClinicalDiagnosticFidelity':[{'Score': , 'Explanation':}, ...]}]
,→
,→
,→
,→
,→
,→
,→
,→
,→
15
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
C. Case Studies by Cardiologist
GEM Interpretation: Upon analyzing the provided ECG image and computed measurements, several key
features and abnormalities are evident. The heart rate is elevated at 121 bpm, indicating a rapid ventricular
response. The rhythm appears irregular, consistent with atrial fibrillation, as seen by the absence of distinct P
waves and the presence of irregular RR intervals. In the precordial leads, particularly V1 through V4, there are
low QRS voltages, which could suggest conditions such as pericardial effusion or obesity. The QRS complexes
in these leads are notably diminished, which aligns with the low voltage finding. Furthermore, the presence of
premature ventricular contractions (PVCs) is suggested by the irregular RR intervals and the occasional
absence of preceding P waves. The ST-T changes observed in the anterolateral leads, particularly V2 to V6, are
nonspecific but could indicate ischemic changes or other cardiac conditions. The QTc intervals are prolonged
in several leads, which may suggest a risk for arrhythmias. The combination of these findings, including the
rapid ventricular response, irregular rhythm, low QRS voltages, and nonspecific ST-T changes, supports a
diagnosis of atrial fibrillation with rapid ventricular response and PVCs. The low QRS voltages in the precordial
leads add another layer of complexity, potentially indicating underlying cardiac conditions. The ECG is
classified as abnormal due to these findings, which warrant further clinical correlation and possibly additional
diagnostic testing to determine the underlying cause of these abnormalities.
PULSE Interpretation: The electrocardiogram (ECG) reveals atrial fibrillation with a rapid ventricular response,
accompanied by premature ventricular contractions (PVCs) or aberrant ventricular conduction, indicating
irregular heart rhythm. Additionally, the ECG suggests possible anterior myocardial infarction, although the
age of the infarct cannot be determined. The inferior and lateral ST-T changes are nonspecific, and the QRS
voltages are low in the precordial leads, contributing to an overall abnormal ECG pattern.
Target Answer (GPT-4o): Upon examining the provided ECG, we identify several critical features and
abnormalities. The first notable observation is the presence of atrial fibrillation, characterized by an irregular
rhythm and lack of distinct P waves in the ECG tracings. This is supported by the irregular RR intervals, which
vary significantly between heartbeats. The heart rate is elevated at 132 bpm, indicative of a rapid ventricular
response. Additionally, the ECG reveals frequent premature ventricular contractions, or multifocal PVCs, with
varying QRS complex morphologies, suggesting aberrant ventricular conduction or possible ventricular
ectopy. Lead I and precordial leads demonstrate low QRS voltages, which can correlate with various
conditions, including myocardial infarction or myopathies. In terms of ST-T changes, the anterior leads (V1-V4)
may suggest possible anterior myocardial infarction, as evidenced by QRS amplitude and morphology
abnormalities. Moreover, inferior and lateral leads, particularly lead aVL, display nonspecific ST-T changes,
which could indicate ischemia or other pathological processes. A comprehensive review of the QT interval
reveals prolonged QTc intervals particularly in the precordial leads, reflecting altered repolarization processes.
Furthermore, no distinct high, lateral, or inferior lead pattern suggests definitive infarction, but the changes are
significant enough to warrant further clinical investigation and correlation with clinical symptoms and history.
Conclusively, the ECG findings strongly support a diagnosis of atrial fibrillation with rapid ventricular response,
accompanied by frequent multifocal PVCs or aberrant ventricular conduction. The anterior region's ST-T
changes and precordial low QRS voltages emphasize the need for further assessment, possibly indicating a
previous or ongoing ischemic event. However, additional clinical context and investigations are essential to
corroborate these findings and formulate a complete treatment strategy.
Ground-truth Report: Atrial fibrillation with rapid ventricular response with frequent multifocal PVCs or
aberrant ventricular conduction; Possible anterior infarct; Inferior/lateral ST-T changes are nonspecific; Low
QRS voltages in precordial leads; Abnormal ECG.
Figure 4. Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
16
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
GEM Interpretation: Upon analyzing the provided ECG image and computed measurements, several key
features and abnormalities are evident. The rhythm is irregular, with a heart rate of 102 bpm, which is
consistent with atrial fibrillation. This is supported by the absence of distinct P waves across multiple leads,
such as Lead I and Lead II, and the presence of irregular RR intervals, as seen in the measurements. The QRS
complexes are generally narrow, indicating a supraventricular origin, which aligns with atrial fibrillation. In Lead
I, the QRS amplitude and duration are within normal limits, but the irregular rhythm is evident. Lead II shows a
similar pattern with absent P waves and irregular intervals, further supporting atrial fibrillation. Leads III and
aVF, which reflect the inferior wall, do not show significant Q waves or ST segment changes that would suggest
acute ischemia or infarction. Lead aVL, focusing on the high lateral region, also does not exhibit significant
abnormalities in the QRS, ST, or T wave morphology. Lead aVR, often used to assess global ischemic changes,
does not show significant ST elevation or T wave inversion, which might have indicated more severe conditions
like left main coronary artery disease. In the precordial leads V1 through V6, the QRS complexes are narrow,
and there are no significant ST segment elevations or depressions that would suggest acute myocardial
infarction. Overall, the ECG findings are consistent with atrial fibrillation, characterized by the absence of
distinct P waves and irregular RR intervals. The lack of significant ST segment changes or pathological Q waves
across the leads suggests that there is no acute ischemic event present. This analysis supports the diagnosis of
atrial fibrillation as the primary abnormality, with no evidence of acute myocardial infarction or other significant
ischemic changes.
PULSE Interpretation: The electrocardiogram (ECG) reveals atrial fibrillation, characterized by an irregular
heart rhythm with no discernible P waves, indicating a lack of coordinated atrial contraction. Additionally, the
ECG shows a repolarization abnormality, suggesting a potential issue with the heart's recovery phase after
contraction. The probable rate-related abnormality implies that the repolarization abnormality may be related
to the atrial fibrillation, which is causing an irregular heart rate.
Target Answer (GPT-4o): Upon examining the provided ECG image, several key features indicate the
presence of atrial fibrillation (AF). The irregularly irregular rhythm, as evidenced by varying RR intervals,
supports this observation. Notably, the RR intervals range widely, which is a hallmark of AF due to inconsistent
atrial impulses reaching the ventricles. Additionally, the absence of distinct P waves on leads with traditionally
observable P waves is another pivotal indicator; this suggests a loss of organized atrial electrical activity,
consistent with AF. The heart rate approximates 107 bpm, indicative of a moderate ventricular rate, aligning
with a presentation of AF with a controlled rate. No consistent pattern in the PR interval is observed, further
corroborating the diagnosis of AF, as atrial activity is chaotic and not leading to a predictable conduction
pattern through the AV node. Analyzing leads such as lead II, where P wave characteristics are more
observable under normal conditions, we see minimal or absent P wave formations. The variability and
inconsistency of PR intervals signify typical conduction during AF episodes, contributing to its erratic rhythm. In
terms of additional abnormalities, minor QRS and T wave deviations across various leads were noted, but they
did not form a pathological pattern significant enough to suggest additional diagnoses such as ischemia or
infarction. The ST segment and QTc intervals are within normal limits given the context of AF. This ECG analysis
supports a diagnosis of atrial fibrillation, underpinning the clinical picture with evidenced irregular RR intervals
and absent P waves. My interpretation remains focused on the features aligning with AF without assuming
further conditions not supported by the ECG findings.
Ground-truth Report: Atrial fibrillation.
Figure 5. Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
17
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
GEM Interpretation: Upon analyzing the provided ECG image and computed measurements, several key
features and abnormalities are evident. The rhythm is sinus, as indicated by the consistent P wave morphology
across leads and a regular heart rate of 75 bpm. In Lead I, the QRS complex shows a prolonged duration,
which is consistent with a right bundle branch block (RBBB). This is further supported by the characteristic rsR'
pattern in Lead V1, a hallmark of RBBB. The QRS duration in Lead I is notably prolonged, ranging from 134 to
150 ms, which is indicative of this conduction delay. In Lead II, the PR interval is within normal limits, but the
QRS duration is slightly prolonged, further supporting the presence of RBBB. The ST segment and T wave
morphology in Lead II do not show significant deviations, suggesting no acute ischemic changes. Leads III and
aVF, which reflect the inferior wall, do not exhibit pathological Q waves or significant ST segment changes,
ruling out inferior wall ischemia or infarction. Lead aVL shows no significant abnormalities in the QRS complex
or ST segment, which suggests no high lateral ischemia. Lead aVR does not show ST elevation, which would
have indicated left main coronary artery disease. In the precordial leads V2-V4, the QRS complexes are
widened, consistent with RBBB, and there are no significant ST segment elevations or depressions to suggest
anterior wall ischemia. Overall, the ECG findings are consistent with a right bundle branch block, as evidenced
by the prolonged QRS duration and characteristic patterns in the precordial leads. The sinus rhythm and
absence of acute ischemic changes suggest that the conduction abnormality is the primary finding. This
analysis aligns with the diagnosis of an abnormal ECG due to the presence of RBBB.
PULSE Interpretation: The electrocardiogram (ECG) reveals a sinus tachycardia, characterized by a rapid heart
rate with a normal P-wave axis, indicating that the sinus node is functioning normally but at an accelerated
pace. Additionally, the ECG shows a right bundle branch block (RBBB), which is a conduction abnormality
affecting the right ventricle, resulting in a delayed activation of the right ventricle. Overall, the ECG suggests a
diagnosis of sinus tachycardia with right bundle branch block.
Target Answer (GPT-4o): The 12-lead ECG image reveals key characteristics supporting the diagnosis of a
right bundle branch block (RBBB) in the context of sinus rhythm. In Lead I, the QRS duration extends to
approximately 134 milliseconds, coupled with a notable upslope in the ST segment across multiple heartbeats.
The amplitude of the QRS complex peaks at approximately 2.463 mV, which might suggest normal conduction
patterns as the lead does not display criteria typical of a RBBB. Lead V1, however, showcases the classic
features of RBBB with an RsR' pattern evidenced by a marginally increased QRS duration and the presence of a
characteristic \"rabbit ears\" morphology, as indicated by significant amplitudes and uniform morphology,
compatible with RBBB diagnostics. Furthermore, different elevations and variations, including notable upslope
ST segments observed in leads V1 to V3, can further affirm the presence of bundle branch blocks without overt
ischemic patterns. Additional leads such as II, III, aVR, and aVF do not exhibit significant anomalies, aside from
mild ST variations, which are consistent across these leads without inferring significant ischemia or infarction.
The computed average heart rate is normal, around 73 bpm, with consistent RR intervals suggesting regular
sinus rhythm. Although the overall ECG demonstrates sinus rhythm, the broad QRS complexes and rightward
axis deviation are hallmarks of a right bundle branch block. These observations, together with consistent
measurements, fully support the diagnosis of RBBB. No acute ischemic changes are discerned based on ST
segment and T wave morphology across most of the ECG leads, correlating with the absence of acute
myocardial infarction signs. The ECG also appears abnormal due to the prolonged QRS complex duration,
supporting the finding of a RBBB when considered in conjunction with the specific electrocardiographic
patterns observed.
Ground-truth Report: Sinus rhythm; Right bundle branch block; Abnormal ECG.
Figure 6. Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
18
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
GEM Interpretation: Upon analyzing the provided ECG image and computed measurements, several key
features and abnormalities are evident. The rhythm is sinus, as indicated by the regular P waves preceding
each QRS complex. However, the QT interval is prolonged across multiple leads, with QTc values exceeding
500 ms in several instances, suggesting a risk for arrhythmias. In leads II, III, and aVF, there are notable changes
consistent with an inferior infarct, including abnormal Q waves and ST segment deviations. Additionally, leads
V2 to V4 show ST segment elevation and T wave inversion, which are indicative of anterior wall ischemia or
infarction. The lateral leads, particularly I, aVL, V5, and V6, exhibit ST-T changes, such as ST segment depression
and T wave inversion, which may suggest lateral myocardial ischemia. The presence of these changes in
multiple leads supports the diagnosis of myocardial ischemia. The combination of these findings, including the
prolonged QT interval and the ST-T changes, suggests a complex cardiac condition involving both ischemic
and infarct patterns. The ECG is abnormal, with evidence pointing towards an inferior infarct, anterior and
lateral ischemia, and a prolonged QT interval, all of which require further clinical correlation and management.
PULSE Interpretation: The electrocardiogram (ECG) reveals a sinus rhythm, indicating a normal heart rate and
rhythm. However, it also shows an intraventricular conduction defect, which suggests a delay in the electrical
conduction system of the heart. Furthermore, the ECG indicates evidence of a previous inferior myocardial
infarction, although the age of the infarct cannot be determined. Additionally, there are ST-T changes observed
in the anterior, septal, and lateral leads, which may be indicative of myocardial ischemia. Overall, the ECG is
abnormal, suggesting cardiac disease. Clinical Diagnosis: Abnormal ECG with evidence of previous inferior
myocardial infarction and possible myocardial ischemia.
Target Answer (GPT-4o): Analyzing the provided 12-lead ECG shows evidence of an inferior myocardial
infarction, as evidenced by significant deviations in the leads primarily reflective of the inferior wall, such as
leads II, III, and aVF, which exhibit abnormalities in ST segments and T wave morphology. The QRS complexes
and ST segment changes suggest an old or acute inferior infarct. Additionally, ST-T changes are notable in the
anterior leads (V2-V4), indicating anterior or anteroseptal myocardial ischemia. The presence of significant ST
elevation or depression, coupled with T wave inversion in these leads, corroborates the potential for ischemia
in the anterior regions of the heart. In the lateral leads (I, aVL, V5, and V6), the presence of ST segment and T
wave changes could indicate lateral wall ischemia or infarction. Specifically, notable changes in ST form and T
wave morphology support this suspicion, which aligns with the observed abnormalities. The heart rate,
computed at an average of 70 beats per minute, appears normal as it shows regular RR intervals, supporting a
sinus rhythm without apparent rhythm disturbances. The computed PR intervals and QRS durations in different
leads suggest no significant atrioventricular or bundle branch blocks, except for possible localized conduction
delays that contribute to the differential ECG patterns observed. Collectively, these ECG findings demonstrate
a combination of inferior infarct with the potential for ischemia notable in both anterior and lateral regions. This
pattern of changes, particularly the ST-T variations across multiple leads, supports the likelihood of broadened
myocardial involvement. The diagnosis aligns with the classification of an abnormal ECG, confirming
myocardial ischemia and infarction.
Ground-truth Report: Sinus rhythm; Inferior infarct; Ant/septal and lateral ST-T changes may be due to
myocardial ischemia; Abnormal ECG.
Figure 7. Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
19
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
GEM Interpretation: Upon analyzing the provided ECG image and computed measurements, several key
features and abnormalities are evident. The heart rate is elevated at 101 bpm, indicating sinus tachycardia. In
Lead I, the QRS amplitude is notably high, suggesting left ventricular hypertrophy (LVH), which is further
supported by the increased QRS amplitude in Leads V5 and V6. The presence of secondary repolarization
abnormalities is suggested by the ST segment changes and T wave morphology in these leads. In Lead II, the P
wave amplitude and duration are within normal limits, but the PR interval is slightly prolonged, which could
indicate a first-degree AV block. The QRS complexes in Leads III and aVF are diminished, which might suggest
an inferior wall issue, but the lack of significant Q waves or ST segment changes makes this less likely. Lead aVL
shows a high QRS amplitude, consistent with LVH, and the T wave morphology is abnormal, which could
indicate lateral ischemia. Lead aVR does not show significant ST elevation, reducing the likelihood of left main
coronary artery disease. In Lead V1, the absence of an rsR' pattern and the lack of significant ST-T changes
suggest that right ventricular hypertrophy or right bundle branch block is unlikely. Leads V2 to V4 show no
significant Q waves or ST segment elevation, reducing the likelihood of an anterior myocardial infarction.
Overall, the ECG findings are consistent with sinus tachycardia, probable left atrial enlargement, and left
ventricular hypertrophy with secondary repolarization abnormalities. The absence of significant ST segment
elevation or depression in the anterior leads suggests that acute ischemia or infarction is unlikely. The findings
are consistent with chronic changes rather than acute pathology.
PULSE Interpretation: The electrocardiogram (ECG) reveals a sinus tachycardia, characterized by a rapid heart
rate with a normal P-wave axis, indicating that the sinus node is functioning normally but at an accelerated
pace. Additionally, the ECG shows evidence of biatrial enlargement, suggesting that both the left and right
atria are enlarged, which can be a sign of cardiac chamber dilation or hypertrophy. Furthermore, the ECG
demonstrates left ventricular hypertrophy (LVH) with secondary repolarization abnormality, indicating that the
left ventricle is thickened, leading to abnormal electrical conduction and repolarization patterns.
Target Answer (GPT-4o): In analyzing the ECG image, several key features and abnormalities are identified
that support the clinical findings. The rhythm across leads is sinus, as observed from the consistent P wave
morphology, notably prominent in Lead II, indicating a regular atrial rhythm. In Lead II, the P wave amplitude
varies, but the durations range around 92 to 104 ms, suggesting left atrial enlargement. This enlargement is
further evidenced by the increased P wave amplitude in several leads, including V1, where the amplitude is
significantly high. The QRS complexes show increased amplitudes across multiple leads, particularly in V3 and
V4, with corresponding secondary ST-T changes. This pattern of high voltage QRS complexes and secondary
repolarization abnormalities is indicative of left ventricular hypertrophy (LVH). The computer measurements
confirm increased QRS amplitude in these leads, supporting the presence of LVH. Additionally, there is anterior
ST segment elevation seen in leads V2-V4, which can be attributed to the strain pattern associated with LVH
rather than acute myocardial infarction, as it correlates with the presence of LVH. There is no significant Q wave
development or evidence of acute ischemic injury that suggests an infraction. Overall, the ECG findings
collectively indicate sinus rhythm with left atrial enlargement and left ventricular hypertrophy with secondary
repolarization abnormalities. The anterior ST elevation observed is likely a consequence of LVH, consistent with
a LVH strain pattern. These interpretations align with the clinical manifestations of a patient with left heart strain
and structural changes, corroborated by the data synthesis from both visual ECG interpretation and detailed
interval measurements.
Ground-truth Report: Sinus rhythm; Left atrial enlargement; LVH with secondary repolarization abnormality;
Anterior ST elevation, probably due to LVH.
Figure 8. Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
20
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
GEM Interpretation: Upon analyzing the provided ECG image and computed measurements, several key
features and abnormalities are evident. The rhythm is sinus, as indicated by the consistent P wave morphology
across leads, but there is a borderline first-degree A-V block, suggested by the prolonged PR intervals,
particularly noticeable in leads I and II. The QRS complexes exhibit a right bundle branch block pattern,
characterized by a widened QRS duration, particularly in lead V1, where an rsR' pattern is observed. The axis is
rightward, which is consistent with the right bundle branch block. In the inferior leads (II, III, and aVF), there are
nonspecific T wave changes, which could indicate ischemia or other non-specific changes. The QRS amplitude
in lead III is notably low, which might suggest an inferior wall issue, although the changes are not definitive for
infarction. The lateral leads (I, aVL, V5, and V6) show some T wave changes, but these are nonspecific and do
not clearly indicate lateral ischemia or infarction. The QTc intervals are prolonged in several leads, which could
be a concern for potential arrhythmogenic risk. The overall ECG is abnormal, with the combination of these
findings suggesting a complex cardiac conduction issue. The rightward axis and right bundle branch block are
significant, and the nonspecific T wave changes in the inferior and lateral leads warrant further clinical
correlation and possibly additional diagnostic testing to rule out underlying ischemic heart disease or other
cardiac conditions.
PULSE Interpretation: The electrocardiogram (ECG) reveals a sinus rhythm, indicating a normal heart rate and
rhythm. However, the ECG also exhibits several abnormalities, including a prolonged QT interval, which may
indicate a prolonged repolarization phase, and a right axis deviation, suggesting an abnormal orientation of
the heart's electrical axis. Additionally, the ECG shows signs of right ventricular hypertrophy, low QRS voltages
in the limb leads, and extensive ST-T changes, which may be attributed to either hypertrophy or ischemia,
leading to an overall abnormal ECG.
Target Answer (GPT-4o): Upon reviewing the 12-lead ECG image, the diagnosis reveals a sinus rhythm with a
rightward axis and the presence of a right bundle branch block. The rhythm is consistent with a heart rate
around 70 bpm, indicated by relatively regular RR intervals. The rightward axis is evident as lead I shows a
smaller QRS amplitude compared to lead aVF, which reflects more positive deflections. A right bundle branch
block is supported by the widened QRS complexes, particularly evident in leads V1 and V2, where an rsR'
pattern could be observed, consistent with this conduction abnormality. Examination of the lateral leads (I, aVL,
V5, V6) reveals mild T wave abnormalities, which are noted as nonspecific changes. In these leads, the
morphology of the T waves shows slight variations that do not definitively point to ischemia but suggest some
lateral wall changes. Despite these T wave changes being nonspecific, they may warrant further clinical
correlation given the right bundle branch block already present. In summary, the ECG findings primarily
indicate a right bundle branch block with a rightward axis. The T wave changes noted in the lateral leads are
nonspecific and should be clinically correlated to assess their significance. These findings characterize the ECG
as abnormal, and the conduction delay could suggest underlying pathology that may require additional testing
or clinical evaluation to rule out underlying cardiac conditions. This evaluation is critical to ensure
comprehensive cardiovascular assessment and appropriate subsequent management.
Ground-truth Report: Sinus rhythm; Rightward axis; Right bundle branch block; Lateral T wave changes are
nonspecific; Abnormal ECG.
Figure 9. Cardiologist Evaluations. Blue: Findings exceeding expert expectations. Yellow: Findings with differing expert opinions.
21