Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentation
Table 1: Performance of text segmentation models on five English evaluation datasets. GraphSeg model [glavavs2016unsupervised] was evaluated independently on different subcorpora of the Choi dataset (indicated with an asterisk).
['Model', 'Model\xa0Type', 'Wiki-727K', 'Wiki-50', 'Choi', 'Cities', 'Elements']
Both our Transformer-based models – TLT-TS and CATS – outperform the competing supervised model of \citeauthorkoshorek2018text \shortcitekoshorek2018text, a hierarchical encoder based on recurrent components, across the board. The improved performance that TLT-TS has with respect to the model of \citeauthorkoshorek2018text \shortcitekoshorek2018text is consistent with improvements that Transformer-based architectures yield in comparison with models based on recurrent components in other NLP tasks [vaswani2017attention, devlin2018bert]. The gap in performance is particularly wide (>20 Pk points) for the Elements dataset. This would suggest that TLT-TS and CATS offer more robust domain transfer than the recurrent model of \citeauthorkoshorek2018text \shortcitekoshorek2018text. Both our Transformer-based models, TLT-TS and CATS, outperform the unsupervised GraphSeg model (which seems to be only marginally better than the random baseline) by a wide margin. The coherence-aware CATS model is again significantly better (p<0.01 for FI and p<0.05 for CS and TR) than the TLT-TS model which was trained to optimize only the segmentation objective. This is quite encouraging as it suggests that it is possible to, via the zero-shot language transfer, rather reliably segment texts from under-resourced languages lacking sufficiently large gold-segmented data needed to directly train language-specific segmentation models (that is, robust neural segmentation models in particular).