HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
Table 4: Human evaluation: proportions of rankings and mean ranks (MeanR; lower is better) of various models.
['Models', '1st', '2nd', '3rd', '4th', '5th', '6th', 'MeanR']
We also conducted human experiment with 20 randomly sampled documents from the CNNDM test set. We compared our model \sc HibertM against Lead3, DCA, Latent, BERT and the human reference We asked the subjects to rank the outputs of these systems from best to worst. We also converted the rank numbers into ratings (rank i to 7−i) and applied student t-test on the ratings. \sc HibertM is significantly different from all systems in comparison (p<0.05), which indicates our model still lags behind Human, but is better than all other systems.