An Annotated Corpus of Relational Strategies in Customer Service Under review in Language Resources and Evaluation
Table 3: Reviewer agreement on multi-intent detection. For example, row 3 is the number of requests flagged as containing multiple intentions by at least three reviewers.
['[EMPTY]', 'TripAdvisor', 'Train', 'Airline', 'Telecom']
Next, we considered the reviewers’ determination of multiple intentions. A κ was calculated over how reviewers flagged requests containing more than one user intention. This table demonstrates the difficulty of multi-intent detection, even for humans. The domain does not seem to be a factor as κ is similar across datasets. It is apparent, however, that in the forum setting, users are much more likely to insert multiple intentions in a single request than in a chat setting.