Deep Attentive Ranking Networks for Learning to Order Sentences
Table 2: Kendall’s tau (τ) and perfect match ratio (PMR) on test set for various benchmark datasets. Note that Kendall’s tau metric, on which our method consistently outperforms other baselines, correlates with human judgements.
['[BOLD] Methods', '[BOLD] NIPS abstracts  [ITALIC]  [BOLD] τ', '[BOLD] NIPS abstracts  [BOLD] PMR', '[BOLD] AAN abstracts  [ITALIC]  [BOLD] τ', '[BOLD] AAN abstracts  [BOLD] PMR', '[BOLD] NSF abstracts  [ITALIC]  [BOLD] τ', '[BOLD] NSF abstracts  [BOLD] PMR', '[BOLD] arXiv abstracts  [ITALIC]  [BOLD] τ', '[BOLD] arXiv abstracts  [BOLD] PMR', '[BOLD] SIND captions  [ITALIC]  [BOLD] τ', '[BOLD] SIND captions  [BOLD] PMR', '[BOLD] ROCStory  [ITALIC]  [BOLD] τ', '[BOLD] ROCStory  [BOLD] PMR']
We consistently achieve better results than the state-of-the-art methods on τ on all the datasets. We reiterate that τ score correlates with human judgements. RankTxNet improves the previous state-of-the-art in terms of τ scores by the absolute percentage of 3.09% in NIPS abstracts, 4.48% in AAN abstracts, 2.98% in NSF abstracts, 1.3% in arXiv abstracts, 6.31% in SIND captions and 2.8% in ROCStory dataset. On SIND captions, we also get an improvement on PMR from 15.01% to 15.59%. On all other datasets, we show competitive PMR score. Among our approaches, pairwise and listwise methods always outperform the pointwise method. Specifically, ListMLE performs better than all other methods in most of the cases.