Classifying Diagrams and Their Parts using Graph Neural Networks:A Comparison of Crowd-Sourced and Expert Annotations
Table 3: Mean accuracy, macro F1 and weighted F1 scores for node classification. The results are averaged over 20 runs. The following abbreviations indicate the graph used: ‘AI2D’ for the original crowd-sourced graphs from AI2D, ‘G’ for the grouping graph and ‘G+C’ for the combination of grouping and connectivity graph from AI2D-RST. An asterisk indicates that the difference between AI2D and the best AI2D-RST graph is statistically significant at p<0.05 when comparing the results for the given metric over 20 runs using Mann–Whitney U test. The best result for each metric is marked using bold.
['Model Graphs', 'GAT AI2D', 'GAT G', 'GAT G+C', 'GCN AI2D', 'GCN G', 'GCN G+C', 'SAGE AI2D', 'SAGE G', 'SAGE G+C', 'SGC AI2D', 'SGC G', 'SGC G+C']
Because the results are not entirely comparable due to different node types present in the two resources, it is more reasonable to compare architectures. SAGE, GCN and GAT clearly outperform SGC in classifying nodes from both resources, as does the random forest classifier. AI2D nodes are classified with particularly high accuracy, which may result from having to learn representations for only one node type, that is, the image constant (N=1000). AI2D-RST, in turn, must learn representations from scratch for both image constants (N=1000) and grouping nodes (N=7300).