AI2D-RST: A multimodal corpus of 1000 primary school science diagrams
Table 4: Class-wise marginal κ scores for discourse relations
['Discourse relation', '[ITALIC] κ', 'z-score', 'p-value', 'Frequency in the entire corpus']
that the annotators used during the inter-annotator agreement experiment in a descending order. The results show that annotators consistently agree on common RST relations such as cyclic sequence, which is used to annotate recurring cycles formed by diagram elements, and preparation, which is used to describe the relationship between a title and an entire diagram. These RST relations are associated with visually distinctive macro-groups (cycles) and relatively fixed diagram elements (titles), which is likely to increase agreement. The same applies to frequently occurring relations defined between a label and an object or its part, such as property-ascription, identification and elaboration, whose specific use cases were defined in the annotation guide. In short, the development of an annotation guide seemed to support the consistent annotation of RST relations. Compared to previous studies of inter-annotator agreement using multimodal RST 88.4% of RST relations in the corpus.