Hierarchical Transformers for Long Document Classification
Table 5: The effect of including positional embeddings in ToBERT model. Fine-tuned BERT segment representations were used for these results.
['[EMPTY]', 'Position embeddings No', 'Position embeddings Yes']
It can be observed that position embeddings did not significantly affect the model performance for Fisher and 20newsgroups, but they helped slightly in CSAT prediction (an absolute improvement of 0.64% F1-score). We think that this is explained by the fact that Fisher and 20newsgroups are topic identification tasks, and the topic does not change much throughout these documents. However, CSAT may vary during the call, and in some cases a naive assumption that the sequential nature of the transcripts is irrelevant may lead to wrong conclusions.