KPQA: A Metric for Generative Question Answering Using Word Weights
Table 2: Inter annotator agreement
['[BOLD] Dataset', '[BOLD] Krippendorff’s  [ITALIC] α', '[BOLD] # Annotators (avg.)']
Filtering Noisy Workers Some workers did not follow the instructions, producing poor-quality judgements. To solve this problem, we filter noisy responses using z-score as in jung2011improving. We first compute the z-score among ten responses for each sample. Then, we consider the responses whose z-score is higher than 1 as noise and remove them up to five in the order of the z-score. As a result, all of the samples have at least five annotations after removing the noisy responses. We use the average score of the annotators for each sample as a ground-truth evaluation score to assess the quality of the evaluation metric. Inter-Annotator Agreement The final dataset is further validated with Krippendorff’s alpha krippendorff2011computing, a statistical measure of inter-rater agreement for multiple annotators. These coefficient numbers indicate a “near-perfect” agreement according to one of the general guidelines landis1977measurement for kappa-like measures.