Classifying Diagrams and Their Parts using Graph Neural Networks:A Comparison of Crowd-Sourced and Expert Annotations
Table 6: Baseline accuracy, macro F1 and weighted F1 scores for graph classification using dummy (D), random forest (RF; 100 estimators) and support vector machine (SVM; C=1.0) classifiers with balanced class weights. The results are averaged over 20 runs. All models were implemented using scikit-learn 0.21.3. Each diagram is represented by a 5-dimensional vector acquired by averaging the features for all nodes in the graph.
['Classes Dataset', 'AI2D ( [ITALIC] N=17) AI2D', 'AI2D ( [ITALIC] N=17) AI2D', 'AI2D ( [ITALIC] N=17) AI2D', 'AI2D ( [ITALIC] N=17) AI2D-RST (G)', 'AI2D ( [ITALIC] N=17) AI2D-RST (G)', 'AI2D ( [ITALIC] N=17) AI2D-RST (G)']
The macro F1 scores show that the RF classifier with 100 decision trees offers competitive performance for all target classes and both AI2D and AI2D-RST, in some cases outperforming graph neural networks. It should be noted, however, that the RF classifier is trained with node features learned using GraphSAGE.