Inter-rater Agreement on Sentence Formality
Table 1: Agreement and correlation values on the actual set.
['[EMPTY]', '[BOLD] Forum', '[BOLD] Blog', '[BOLD] News', '[BOLD] Paper', '[BOLD] Overall']
We report four nonparametric correlation coefficients between the two raters, as well as cosine and Tanimoto similarity Each element in a rating vector corresponds to a sentence and the value of the element is the formality rating of the sentence. These results were obtained after pruning the X-marked sentences. Overall results (the rightmost column) indicate that the cosine and Tanimoto similarity between the raters were fairly high, which shows that the rating directions were preserved. In other words, if rater A rated sentence S1 as more formal than sentence S2, then rater B also rated S1 as more formal than S2, not the other way round. This shows the consistency of our raters and the importance of Likert scale in formality judgment. High similarity values were also obtained within specific categories (forum, blog, news and paper sentences), showing that rating consistency was maintained across categories. Similar results were obtained for the warm-up set as well. Inter-rater reliability was measured with Cohen’s κ and Krippendorff’s α. When category labels are not equally distinct from one another (as is our case), Krippendorff’s α must be computed. Note that Krippendorff’s α allows missing data as well, so we could have incorporated the X-marked sentences in α-computation. But to avoid complication, we chose not to do so, and quarantined the X-marked sentences for further analysis. This is a significant achievement given the conservativeness of κ, the subjectivity associated with formality judgment, our small dataset, and no prior instructions on what to consider formal and what to consider informal. (κBlog 0.164, κNews 0.019), which shows the merit of Likert-scale annotation for formality judgment. The overall κ-values were found to be statistically significant with p-value < 0.005.