Sentence Ordering and Coherence Modeling using Recurrent Neural Networks
Table 3: Comparison on extractive summarization between models trained from scratch and models pre-trained with the ordering task.
['Model', 'SICK r', 'SICK  [ITALIC] ρ', 'SICK MSE', 'MSRP (Acc)', 'MSRP (F1)']
We train a model from scratch to perform extractive summarization in the above fashion. We then consider a model that is pre-trained on the ordering task and is fine-tuned on the above task. We use DailyMail for pre-training purposes and CNN for fine-tuning and evaluation. The labels in DailyMail are not used. To evaluate the quality of sentence embeddings derived from the model, we use the evaluation pipeline of \citeauthorkiros2015skip \shortcitekiros2015skip for tasks that involve understanding sentence semantics. These evaluations are performed by training a classifier on top of the embeddings derived from the model (holding the embeddings fixed) so that the performance is indicative of the quality of sentence representations. Results for only uni-directional versions of models are discussed here for a fair comparison. Similar to the skip-thought (ST) paper, we train two models - one predicting the correct order in the forward direction and another in the backward direction. The numbers shown for the ordering model were obtained by concatenating the representations from the two models.