Discourse Coherence in the Wild: A Dataset, Evaluation and Methods
Table 2: Distribution of coherence classes as a percentage of the training data.
['Domain', 'Raters', 'Coherence Class (%) Low', 'Coherence Class (%) Med', 'Coherence Class (%) High']
The resulting four domains each contain 1200 texts (1000 for training, 200 for testing). Each text has been scored as {low, medium, high} coherence by 5 MTurk raters and 3 expert raters. There is one consensus label for the expert ratings and another consensus label for the MTurk ratings. We observed that the MTurk raters tended to label more texts as “medium” coherence than the expert raters. Since the MTurk raters did not go through an extensive training session, they may be less confident in their ratings, defaulting to medium as the safe option.