Is It Worth the Attention?A Comparative Evaluation of Attention Layers forArgument Unit Segmentation
Table 1: The weighted F1-scores for the baseline and all four variations. Results are shown per variation and embedding. Each row shows the performance of one architecture with different word embeddings as input vector. The highest score for each architecture is marked in bold.
['Model', 'GloVe', 'BERT', 'Flair']
We evaluate the performance of all architectures on the Persuasive Essays data set detailed above. The models are re-initialized after every evaluation and do not share any weights. This allows us to answer the first research question of whether additional attention layers have a positive impact on the prediction quality. To answer the second research question, we re-run each training, replacing the GloVe with BERT and Flair embeddings. Both contextualized embedding methods are tested separately. This makes document-level or paragraph-level embeddings impractical for the data set. As a performance measure, we report the weighted F1-score instead of the macro F1-score , since it takes the imbalance of the samples per label into account. For our re-implementation of the baseline, we are able to approximately reproduce the results reported by Ajjour et al. Generally speaking, the added attention encodings do not improve upon the original architecture’s performance, no matter at which position they are added. For the Flair embeddings, this results in a 0.20 points performance drop in the F1-score measure. -id10 model is able to achieve results that outperform both, GloVe and Flair embeddings. Compared to the GloVe vectors, the models trained on the Flair embeddings mostly lose in F1-score performance as well.