Aspect-based Sentiment Analysis of Scientific Reviews
Table 8. Correlation of recommendation scores with the final recommendation decision of the chair (p-value <10−150 in all these cases.)
['Correlation', 'Pearson’s', 'Spearman’s Rank', 'Kendall’s  [ITALIC] τ']
We investigate here how the final decision about the paper taken by the chair tallies with the individual reviewer decisions. In order to quantify individual reviewer decision, we consider their recommendation scores of 1-5 to a paper as reject and 6-10 as accept. Next we estimate what percentage of the reviewers disagree with the final recommendation of the chair. If the review scores were completely random this would be 50%. In the ICLR dataset we find this to be 24.9%. If the disagreement is calculated based on majority voting then this comes down to 14.8% which indicates that the chair’s decision may significantly vary from the recommendation of the majority of the reviewers. Also, it indicates that the reviewers tend to be more judicious in deciding which half their review scores will fall in between (1-5) and (6-10) bins than assigning the relative scores within each bin if the chair’s decision is assumed to be correct. We also explicitly compute the correlation of the recommendation scores with the final recommendation decision. Once again we observe that the correlation values never go beyond 0.78.