Natural Language Processing for Achieving Sustainable Development:the Case of Neural Labelling to Enhance Community Profiling
Table 1: Results of ablation study (single-task).
['Model', '[ITALIC] test_set (T3) P', '[ITALIC] test_set (T3) R', '[ITALIC] test_set (T3) F1', '[ITALIC] real_simulation (T3) P', '[ITALIC] real_simulation (T3) R', '[ITALIC] real_simulation (T3) F1']
Models Performance. Notably, adding attention and integrating signal from descriptions to the base system caused significant improvements in performance. Significantly lower performance is observed in all settings from the test_set to the real_simulation evaluation setting. This is due to a substantial drop in precision, which proves the extreme difficulty of the task due to the significant imbalance between labels. Note, however, that recall remains stable over changes in evaluation setting. This is particularly important for a system which is meant to enhance the annotators’ speed, rather than to completely replace human experts: in this context, missing labels are more time consuming to recover than correcting false positives.