A Joint Named-Entity Recognizer for Heterogeneous Tag-setsUsing a Tag Hierarchy
Table 7: F1 for combining I2B2’06 and I2B2’14. The top two models were trained only on a single dataset. The lower table part holds the number of collisions at post-processing. (*) indicates results that are statistically significantly better than others in that column.
['F1 Model', 'Test Set  [ITALIC]  [BOLD] I2B2’06', 'Test Set  [ITALIC]  [BOLD] I2B2’14', 'Test Set  [ITALIC]  [BOLD] Physio']
To test this scenario, we trained the tested model types on the training sets of I2B2’06 and I2B2’14, which have different tag-sets. The models were evaluated both on the test sets of these datasets and on Physio, an unseen test-set that requires the combination of the two training tag-sets for full coverage of its tag-set. We also compared the models to single models trained on each of the training sets alone. Combining the two training sets using either MIndep or MMTL leads to substantial performance drop in 5 out of 6 test-sets compared to the best-performing single model. Indeed, the only competitive result, MMTL tested on Physio, had less than 100 collisions. This demonstrates the non triviality in real-world tag-set integration, and the difficulty of resolving tagging decisions across tag-sets.