From Paraphrase Database to Compositional Paraphrase Model and Back
Table 2: Inter-annotator agreement of ML-Paraphrase and comparison with ML dataset. Columns 2 and 3 show the inter-annotator agreement between the two annotators measured with Spearman ρ and Cohen’s κ. Column 4 shows the ρ between ML-Paraphrase and all of the ML dataset. The last column is the average human ρ on the ML dataset.
['Data', 'IA  [ITALIC] ρ', 'IA  [ITALIC] κ', 'ML comp.  [ITALIC] ρ', 'ML Human  [ITALIC] ρ']
We show inter-annotator Spearman ρ and Cohen ’s κ in columns 2 and 3, indicating substantial agreement on the JN and VN portions but only moderate agreement on NN. This suggests that the original NN portion, more so than the others, favored a notion of similarity more related to association than paraphrase.