MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification
Table 3: Performance (test accuracy (%)) on AG News with 10 labeled data per class with different mixup layers set for TMix. ∅ means no mixup.
['[BOLD] Mixup Layers Set', '[BOLD] Accuracy(%)']
Based on Jawahar et al. , the {3,4,5,6,7,9,12} are the most informative layers in BERT based model and each of them captures different types of information (e.g., surface, syntactic, or semantic). We chose to mixup using different subsets of those layers to see which subsets gave the optimal performance. When no mixup is performed, our model accuracy was 69.5%. If we just mixup at the input and lower layers ({0, 1, 2}), there seemed no performance increase. When doing mixup using different layer sets (e.g., {3,4}, or {6,7,9}), we found large differences in terms of model performances: {3,4} that mainly contains surface information like sentence length does not help text classification a lot, thus showing weaker performance. The 6th layer captures depth of the syntactic tree which also does not help much in classifications. Our model achieved the best performance at {7, 9, 12}; this layer subset contains most of syntactic and semantic information such as the sequence of top level constituents in the syntax tree, the object number in main clause, sensitivity to word order, and the sensitivity to random replacement of a noun/verb.