Predicting ConceptNet Path QualityUsing Crowdsourced Assessments of Naturalness
Table 1. Results for the multi-response dataset.
['Opinion Split', '# Q', '# Correct', 'Avg Conf']
The first column lists the different possible opinion splits for a given question; for example, 13/3 corresponds to a question for which 13 people preferred one of the paths, and 3 people preferred the other. The second column shows how many of the 59 questions have a given opinion split. The third column shows the number of questions predicted correctly, i.e. the number of questions for which the prediction of our model coincides with the majority opinion of the human participants. The last column is the model’s average confidence in the majority consensus. The 8/8 row does not include prediction statistics because there is no majority consensus for these questions. Furthermore, there is a strong positive correlation between the confidence scores predicted by our model, shown in the last column, and the amount of disagreement among human annotators. Our model is thus able to distinguish between cases where the difference in naturalness is clear-cut and cases where human annotators would be undecided.