Multi-Task Learning for Coherence Modeling
Table 4: Results of the binary discrimination task on the WSJ. * indicates significance (p<0.01) over all the other models based on the randomization test. Egrid models are significantly worse than MTLSOX and Concatgrs on the PRA metric and significantly worse than all models on TPRA.999Joty et al. (2018) reported 0.885 PRA for their Egrid CNNlex, which we were unable to replicate using their code; however, this is still lower compared to our results.
['Model', 'PRA', 'TPRA']
Binary Classification. The results demonstrate the effectiveness of our MTL approach using a supervision signal at the bottom layers based on the words’ GR types, which significantly outperforms all other approaches and achieves state-of-the-art results on the WSJ (0.932 PRA and 0.941 TPRA). The performance of the Egrid neural models shows that despite their ability to rank a document higher than its incoherent counterparts (0.876 and 0.846 PRA), they do not generalize when documents are compared against counterparts from the whole test set (0.656 and 0.566 TPRA). This could be partly attributed to the pairwise training strategy adopted by these models and their inability to compare entity-transition patterns across different topics. The table also shows that models that utilize compositions over textual units to form document representations (the last four models) are significantly more effective than those explicitly utilizing only the local transitions between sentences (LC model). Furthermore, we observe that incorporating GR types (MTL, MTLSOX and Concatgrs) gives significantly better results compared to the STL model that is GR-agnostic. The superiority of the MTL model over Concatgrs and MTLSOX demonstrates that learning the GR types, within an MTL framework, allows the model to learn richer contextual representations (but also to be more efficient at testing time compared to e.g., Concatgrs since it does not require external linguistic tools).