Back Attention Knowledge Transfer for Low-resource Named Entity Recognition
Table 2: Evaluation on German and Spanish NER
['[BOLD] Approach', '[BOLD] German', '[BOLD] Spanish']
Evaluation metric is F1-score. As shown in the table, the performances of BiLSTM+CRF and CharLM+BiLSTM+CRF can be further improved by integrating with BAN. After adding BAN to baseline model (i.e., BiLSTM+CRF), the performance of using first attention matrix, BANfirst , is almost equivalent to the model using last layer attention matrix and the BANave has the best performance. In addition, we can also note that the poorest performances of BAN are usually better than that of the base models. This indicates that the proposed BAN method is quite robust and the information transferred from English NER model is truly beneficial for original task. The baseline model achieves a F1-score of 81.41% and 82.49%, respectively. Adding back attention information, the F1-score is increased to 84.32% and 84.78%. Existing state-of-the-art systems utilize language model to produce contextual embedding, which are orthogonal to our work. Therefore, using the character-level language models to generate word embedding, the performance of CharLM+BiLSTM+CRF is close to the state-of-the-art. After adding BAN, the performances of each model are improved. The CharLM+BiLSTM+CRF+BANlast obtains the SOAT F1-score of 88.41% and 88.16%.