Hierarchical Transformers for Long Document Classification
Table 4: Comparison of models using fine-tuned BERT segment-level predictions (P) instead of segment representations (H).
['[EMPTY]', 'Most frequent', 'Average', 'RoBERT', 'ToBERT']
For each document, having obtained segment-wise predictions we can obtain final prediction for the whole document in three ways: Compute the average of all segment-wise predictions and find the most probable class; Find the most frequently predicted class; Train a classification model. the Fisher dataset. We believe the improvements from using RoBERT or ToBERT, compared to simple averaging or most frequent operations, are proportional to the fraction of long documents in the dataset. CSAT and 20newsgroups have (on average) significantly shorter documents than Fisher, as seen in Fig. Also, significant improvements for Fisher could be because of less confident predictions from BERT model as this dataset has 40 classes. Fig. We used fine-tuned BERT segment-level predictions (P) for this analysis. It can be observed that ToBERT outperforms average voting in every interval. To the best of our knowledge, this is a state-of-the-art result reported on the Fisher dataset.