HARE: a Flexible Highlighting Annotator for Ranking and Exploration
Table 2: Annotation and ranking evaluation results on mobility documents, using three embedding sources. Results are given with and without Viterbi smoothing, using binarization threshold=0.5 and no collapsing of adjacent segments. Pr=precision, Rec=recall, ρ=Spearman’s ρ Pr/Rec/F2 are macro-averaged over folds, ρ is over all test predictions.
['Embeddings', 'Smoothing', 'Annotation Pr', 'Annotation Rec', 'Annotation F-2', 'Ranking  [ITALIC] ρ']
Static and contextualized embedding models performed equivalently well on token-level annotations; BERT embeddings actually underperformed static embeddings and ELMo on both precision and recall. Interestingly, static embeddings yielded the best ranking performance of ρ=0.862, compared to 0.771 with ELMo and 0.689 with BERT. Viterbi smoothing makes a minimal difference in token-level tagging, but increases ranking performance considerably, particularly for contextualized models. It also produces a qualitative improvement by trimming out extraneous tokens at the start of several segments, as reflected by the improvements in precision.