Part-of-Speech Tagging on an Endangered Language: a Parallel Griko-Italian Resource
Table 5: Tagging accuracy for each test narrative with and without active learning. We obtain significant improvements (shown in the Δ column) by adding each annotated narrative to the training set before retraining and tagging the next narrative. The last two columns further outline the improvements from active learning, showing performance on the last and longest narrative of the test set (story-9) in each iteration, also showing the best method in each iteration. The impact of using monolingual data and type-level cross-lingual projections disappears when more training data are available.
['Iteration', 'Narrative', 'Best accuracy without AL', 'Best accuracy with AL', 'Δ', 'Accuracy on story 9', 'Best method']
The results for each narrative in the active learning scenario (“with AL” column) report the best performing model that is trained on the concatenation of the UoI corpus and all the stories that were annotated in previous iterations. It is clear that the performance of the taggers improved continuously, as we added more training data. This is further outlined by each iteration’s tagging accuracy on story 9, the last and longest narrative of the test set. Of course, when a narrative is added in the training set, it is then excluded from the test set, and the performance is reported on the rest of the narratives. All methods display notable improvement as we added the annotated narratives to the training set. Firstly, it is notable that as the training set increases, the advantage of the model of \citetgarrette2013learning that leverages monolingual data diminishes, compared to our simpler crf-mod tagger, both with or without cross-lingual projected tags. Before the first iteration, the accuracy gap is 6.4 percentage points in favor of G&B. However, after adding around 4-5 narratives so that there are around 500 training sentences, our crf-mod+clp method surpasses the G&B method and keeps improving. As we add more training instances, the accuracy of the G&B method plateaus around 85% and does not improve further.