Classifying Diagrams and Their Parts using Graph Neural Networks:A Comparison of Crowd-Sourced and Expert Annotations
Table 5: Mean accuracy, macro F1 and weighted F1 scores for graph classification. The results are averaged over 20 runs. The following abbreviations indicate the graph used: ‘AI2D’ for the original crowd-sourced graphs from AI2D, ‘G’ for the grouping graph and ‘G+C’ for the combination of grouping and connectivity graph from AI2D-RST. * indicates that the difference between AI2D and the best AI2D-RST graph is statistically significant at p<0.05 when comparing the results over 20 runs for the given metric using Mann–Whitney U test. + indicates the same for AI2D-RST grouping graph and the combination of grouping and connectivity graphs. The best result for each metric across all models and graphs is marked in bold.
['Classes Model', 'Original classes from AI2D ( [ITALIC] N=17) GAT', 'Original classes from AI2D ( [ITALIC] N=17) GAT', 'Original classes from AI2D ( [ITALIC] N=17) GAT', 'Original classes from AI2D ( [ITALIC] N=17) GCN', 'Original classes from AI2D ( [ITALIC] N=17) GCN', 'Original classes from AI2D ( [ITALIC] N=17) GCN', 'Original classes from AI2D ( [ITALIC] N=17) SAGE', 'Original classes from AI2D ( [ITALIC] N=17) SAGE', 'Original classes from AI2D ( [ITALIC] N=17) SAGE', 'Original classes from AI2D ( [ITALIC] N=17) SGC', 'Original classes from AI2D ( [ITALIC] N=17) SGC', 'Original classes from AI2D ( [ITALIC] N=17) SGC']
When classifying diagrams into the original semantic categories defined in AI2D (N=17), the AI2D graphs significantly outperform AI2D-RST when using the GraphSAGE architecture. For all other graph neural networks, the differences between AI2D and AI2D-RST are not statistically significant. This is not surprising as the AI2D graphs were tailored for the original classes, yet the AI2D-RST graphs seem to capture generic properties that help to classify diagrams into semantic categories nearly as accurately as AI2D graphs designed specifically for this purpose, although no semantic features apart from the layout structure are provided to the classifier.