An Annotated Corpus of Relational Strategies in Customer Service Under review in Language Resources and Evaluation
Table 2: Reviewer agreement on if any text should be selected. For example, row 3 is the number of requests with selections by at least three reviewers.
['[EMPTY]', 'TripAdvisor', 'Train', 'Airline', 'Telecom']
As it is difficult to determine how often all reviewers agree additional language is present from alignment scores alone, we measured reviewer agreement on the presence of additional language and multiple user intentions. For additional language presence, we calculated Fleiss’ κ over the annotations where the classes compared were if a reviewer did or did not select text. Despite the lower values, we are only measuring presence or absence of unnecessary language, and these two categories did not necessarily occur in equal frequencies. Under these conditions, according to Bakeman et al. Therefore, despite the lower values for κ, the individual reviewer annotations appear reliable and can be further improved when merged based on agreement as discussed in the following section.