Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media
Table 7: We performed an ablation experiment on the stacked model. The results in the table are the average of the scores of three iterations.
['Model', 'F1', 'Delta']
Ablation experiment. We explored the contribution of the features and different aspects of our models. For instance, we tried a BLSTM network using pretrained word embeddings only. The results of this model set our baseline on a 39.78% F1-score This score is considerably close to the state-of-the-art performance, but improvements beyond that are small. The ablation reveals that weighting the classes is the most influential factor, which accounts for a 2.58% of F1 score improvement. This aligns with the fact that the data is highly skewed, and thus, the model should pay more attention to the less frequent classes. The second most important aspect is the POS tags, which enhance the results by 1.10%. Almost equally influential are the phonetic and phonological features that push the F1 score by 0.93%. According to the ablation experiment, using phonetic and phonology along with the pretrained word embeddings and POS tags can reach an F1 measure of 41.81%, which is a very similar result to the state-of-the-art score, but with a simpler and more suitable model for SM environments (i.e., without gazetteers or capitalization).