Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media
Table 6: The class-level and overall results of our systems on the WNUT 2017 dataset. WNUT represents the winning system of the shared task (UH-RiTUAL), E2E is the end-to-end model, and Stacked shows the results of the stacked model. Both systems considerably outperform the state-of-the-art results. Between the end-to-end and the stacked models, the former gets better overall precision while the latter stands out on recall.
['Classes', 'Precision (%) Stacked', 'Precision (%) E2E', 'Precision (%) WNUT', 'Recall (%) Stacked', 'Recall (%) E2E', 'Recall (%) WNUT', 'F1 (%) Stacked', 'F1 (%) E2E', 'F1 (%) WNUT']
Stacked vs. end-to-end model. This means that the stacked model is slightly better at generalizing than the other models since it can detect a more diverse set of entities. It assigns a better F1 score to the stacked system (43.90%) than to the end-to-end model (42.79%) because the former finds more rare and emerging entities than the latter. This tends to capture the most frequent entities and leave behind the rare ones, which explains the different behaviors between the precision and recall of both models. State of the art comparison. In the case of the stacked system, the precision and recall outperform the winning system of the shared task (UH-RiTUAL) across all the classes. Moreover, even though the UH-RiTUAL system uses gazetteers, it only outperforms the recall of the end-to-end model on the corporation class. These results can be explained by the entity diversity of the dataset, where the emerging and rare properties are difficult to capture with external resources.