Understanding Spatial Language in Radiology: Representation Framework, Annotation, and Spatial Relation Extraction from Chest X-ray Reports using Deep Learning
Table 4: Annotator agreement.
['[BOLD] Number of Reports', '[BOLD] Kappa ( [ITALIC] κ) Spatial Indicator', '[BOLD] Overall F1 Trajector', '[BOLD] Overall F1 Landmark', '[BOLD] Overall F1 Diagnosis', '[BOLD] Overall F1 Hedge']
The Kappa (κ) agreement between the two annotators has been calculated for Spatial Indicator (as this is a binary classification task) whereas we report the overall F1 agreement for annotating the spatial role labels (as this is a role identification task). The Kappa agreement is high for Spatial Indicators in both annotation rounds. The F1 agreements for the 4 spatial roles are fairly low in the first round with much improvement in the second round. This is mainly because it is relatively easy and unambiguous to locate a spatial preposition in a sentence compared to identifying the spatial roles. All conflicts were reconciled with an NLP expert (K.R.) following each round of annotation. The moderate agreement rate for Trajector and Diagnosis roles was likely due to ambiguity in distinguishing the two roles in a sentence, especially when the language pattern is different from the usual. Consider the examples below: Probably scarring in the left apex, although difficult to exclude a cavitary lesion. There are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex. In the first example, ‘scarring’ was annotated as a Trajector after reconciliation as its spatial location (‘left apex’) is described directly, although there is a higher chance of annotating it as a Diagnosis since most of the probable diagnoses terms are usually preceded by a Hedge term (‘Probably’ in this case). Similarly, ‘cavitary lesion’ is indirectly connected to the same body location (‘left apex’) and has been interpreted as an additional finding. So, ‘cavitary lesion’ was also annotated as a Trajector and not as a Diagnosis. In the second example, ‘cavitary lesion’ was annotated as a Diagnosis in context to the first ‘in’ in the sentence, whereas the same term ‘cavitary lesion’ was annotated as a Trajector when its role was identified in context to the second ‘in’. As previously noted, this difference where the same term can be both a Trajector and Diagnosis in different sentences is a consequence of focusing on explicitly representing the spatial language as described as well as the natural ambiguity between a finding and diagnosis in radiology. As a result, some downstream processing or interpretation is still required, which we leave to future work.