Snippext: Semi-supervised Opinion Mining with Augmented Data
Table 7. Ablation analysis of MixMatchNL. We evaluate performance with F1 score for AE and MF1 for ASC.
['Methods', 'AE@1000 rest', 'AE@1000 laptop', 'ASC@1000 rest', 'ASC@1000 laptop', 'AE@full rest', 'AE@full laptop', 'ASC@full rest', 'ASC@full laptop']
We analyze the effect of each component of MixMatchNL by ablation. We consider a few variants. First, we replace the MixDA component with regular DA’s (the “w/o. MixDA” row). Second, we disable the use of unlabeled data. The resulting method is equivalent to the BERT-FD baseline but with MixUpNL as regularization (the 3rd row). Third, to investigate if the guessed labels by pre-mature models harm the performance , we disable label guessing for the first 3 epochs (the 4th row). Results. The performance generally degrades as MixDA is removed (by up to 1.8% in Restaurant ASC@1000) and unlabeled data are removed (by up to 2.6%). The effectiveness of the two optimizations is similar among both AE and ASC tasks. As expected, both optimizations are more effective in the settings with less data (a total of 9.76% absolute improvement at size 1000 vs. 6.75% at full size). Finally, it is unclear whether discarding guessed labels from pre-mature models helps improve the performance (with only ∼1% difference overall).