E2EET: From Pipeline to End-to-end Entity Typing via Transformer-Based Embeddings
Table 2: A comparison of our mention-level (ML) model, with three different attention mechanisms, against state-of-the-art mention-level typing systems. The system using handcrafted features, AFET, is highlighted in grey. Grey cells with bold text indicate the best performing model not using handcrafted features. Non-highlighted cells with bold text are the best performing model even when compared to systems using handcrafted features.
['[BOLD] Model', '[BOLD] Wiki Acc', '[BOLD] Wiki Ma-F1', '[BOLD] Wiki Mi-F1', '[BOLD] Ontonotes Acc', '[BOLD] Ontonotes Ma-F1', '[BOLD] Ontonotes Mi-F1', '[BOLD] BBN Acc', '[BOLD] BBN Ma-F1', '[BOLD] BBN Mi-F1']
AFET, which relies on handcrafted features, is highlighted in grey. Another noticeable result is that, for the mention-level typing models, there is a clear relationship between the complexity of the attention model and the overall performance. The dynamic attention model in particular excels on the modified datasets, which are significantly cleaner than the original datasets as a result of being taken from the hand-labelled test set of their respective original dataset. The dynamic attention mechanism is clearly most effective when the dataset is clean and when there is consistency between the training and testing sets.