Interaction Matching for Long-Tail Multi-Label Classification
Table 2: Average precision (macro-averaged by label) performance on the Tutorial dataset. Significant results that are significant at p<0.01 are indicated by ↑.
['Model', 'Dev Set', 'Test Set']
Results. We use average precision (AP, macro-averaged by label), given the environment that users are presented with a list of possible software tools to complete the step of the tutorial. When applied to FastText, our approach improves the test set performance by 6% (relative improvement, significant at p<0.01). The FastText model achieved a perfect AP score of 1.0 for 55 of the labels found in the test set (meaning it was ranked highest among all labels whenever it appeared), whereas the interaction variant had a perfect score for 69 labels. This is a 25% improvement, most of which came from the less frequent half of the labels. The least frequent quarter saw an even bigger change, from 11 perfect scores to 26 (136%), four of which had no training samples. Our approach also slightly improves the performance on XML-CNN, though the results are not significant at p<0.01. Interestingly, the XML-CNN model appears to hamper performance in the development set.