One Model to Recognize Them All:Marginal Distillation from NER Models with Different Tag Sets
Table 2: F1 score results for tag set extension experiments. The best results are in bold.
['System', 'BN', 'BC', 'WB', 'MZ', 'TC', 'AVG']
For each extension setting, we train on the training sets with the corresponding source and target tag sets, and tune hyper-parameters and evaluate on the development and test sets of the target domain with the original full tag set. When Mardi has access only to the pre-trained NER models, Marginal CRF slightly outperforms it. However, Mardi provides dramatic flexibility on the required form of NER resources. Both Marginal CRF and Mardi-based systems considerably outperform the Post Processing baseline, showing the power of joint training a NER model over multiple tag sets. Finally, knowledge distillation of CRF models yields much more superior results than distillation of classification models on the NER task, which suggests that Mardi is indeed able to properly distill knowledge from a CRF teacher into a CRF student.