Evaluation of Neural NER Models on Turkish
Table 2: Turkish NER Studies
['Study', 'Approach', 'F1 Score']
The first neural network based study (Demir:14) used a regularized average perceptron algorithm and combined continuous vector representations of words and some language independent features (such as context, previous tags, and case features) in a semi-supervised fashion. The use of character embeddings rather than word embeddings was later explored in a stacked bidirectional LSTM network (Kuru:16). For each input character, the system outputs a tag probability and a Viterbi decoder converts character-level probabilities to word-level tag probabilities. The results demonstrated that a good tagging performance could be achieved without benefiting from an extensive list of word features and language dependent knowledge resources. The current state-of-the-art systems utilize bidirectional LSTM networks and experiment with different word representations. The first BiLSTM study (Gungor:19) concatenated word, character, and morphological embeddings as encoder inputs and used a CRF layer on top of the decoder. The tagging model was tested on four other morphologically rich languages (i.e., Czech, Hungarian, Spannish, and Finnish) and the results demonstrated that word representations once augmented with morphological and character embeddings achieve the highest performance. On the other hand, the second BiLSTM study (Gunes:18) combined word embeddings, writing style embeddings (e.g., all in uppercase letters or in sentence case letters) as input representations, and experimented with stacked layers of varying depth. There is only one work where deep bidirectional transformers were utilized (Stefan:20), and in that study both cased and uncased BERT models were evaluated on Turkish NER task. Our work lies on the path opened by these BiLSTM studies where different embeddings are learned and sequentially encoded by LSTMs. However, to the best of our knowledge, this work is the first Turkish NER study that compares language models learned by transformers with BiLSTM models in the same experimental setup. Moreover, our work explores the impact of context on task performance by exploring both context sensitive and insensitive word embeddings. However, there are a few other transformer-based large language models whose performances have not been reported for Turkish tagging task. Additionally, to our best knowledge, the impact of CRF on such models has not been evaluated before. Thus, our experiments on transformer-based networks are oriented around these research questions.