A Dataset of German Legal Documents for Named Entity Recognition
Table 4: Precision, recall and F1 values of the CRF and BiLSTM models for the fine- and coarse-grained classes
['[EMPTY]', '[BOLD] Prec %', '[BOLD] Rec %', '[BOLD] F1']
The dataset was thoroughly evaluated, see \newciteleitner2019fine for more details. As state of the art models, Conditional Random Fields (CRFs) and bidirectional Long-Short Term Memory Networks (BiLSTMs) were tested with the two variants of annotation. For CRFs, these are: CRF-F (with features), CRF-FG (with features and gazetteers), CRF-FGL (with features, gazetteers and lookup). For BiLSTM, we used models with pre-trained word embeddings [reimers2014germeval]: BiLSTM-CRF [huang2015bidirectional], BiLSTM-CRF+ with character embeddings from BiLSTM [lample2016neural], and BiLSTM-CNN-CRF with character embeddings from CNN [ma2016end]. To evaluate the performance we used stratified 10-fold cross-validation. The F1 score for the fine-grained classification reaches 95.46 and 95.95 for the coarse-grained one. CRFs reach up to 93.23 F1 for the fine-grained classes and 93.22 F1 for the coarse-grained ones. Both models perform best for judge, court and law.