Empirical Evaluation of Character-Based Model on Neural Named-Entity Recognition in Indonesian Conversational Texts
Table 7: F1 scores of word embedding-only (word) and concatenation (concat) model on the test set of Small-Talk (left) and Task-Oriented (right) but only for entities containing at least one OOV word. Entries marked with an asterisk (*) indicate that the model does not recognize any entity at all.
['Entity', 'word', 'concat']
Next, we want to understand better how much gain we can get from character models on OOV cases. To answer this question, we ignored entities that do not have any OOV word on the test set and re-evaluated the word embedding-only and concatenation models. We see how the concatenation model consistently outperforms the word embedding-only model for almost all entities on both datasets. On Small-Talk dataset, the overall F1 score gap is as high as 15 points. It is also remarkable that the concatenation model manages to achieve 40 F1 points for GENDER on Small-Talk while the word embedding-only cannot even recognize any GENDER. Therefore, in general, this result corroborates our hypothesis that the character model is indeed better at dealing with the OOV problem.