Controlled Crowdsourcing for High-Quality QA-SRL Annotation
Table 5: Performance analysis when considering PropBank as reference (all roles, core roles, and adjuncts).
['[EMPTY]', 'This work  [BOLD] P', 'This work  [BOLD] R', 'This work  [BOLD] F1', 'He et al. ( 2015 )  [BOLD] P', 'He et al. ( 2015 )  [BOLD] R', 'He et al. ( 2015 )  [BOLD] F1']
S4SS0SSS0Px3 Agreement with PropBank Data It is illuminating to observe the agreement between QA-SRL and PropBank (CoNLL-2009) annotations Hajič et al. (evaluation is automatic and the metric is similar to our UA). while considering the PropBank data as the reference set. Our recall of PropBank roles is notably high, reconfirming the coverage obtained by our annotation protocol.