A Crowdsourced Frame Disambiguation Corpus with Ambiguity
Table 2: Aggregated evaluation results.
['[EMPTY]', 'Eval. Metric', '[BOLD] OS', '[BOLD] OS+', '[BOLD] FS', '[BOLD] TC']
FS performs the worst out of all models on R-Set, because it cannot find newly added frames from the latest FN release, but improves on the F-Set (FS can find candidate frames for LUs not in FN). The scores on the F-Set were lower for all baselines, suggesting that sentences with LUs not in FN are more difficult to classify – this could be because FN is missing frames that can express the full meaning of these LUs. TC has a good performance, but is far from being unbeatable – when measuring Kendall’s \uptau over the R-Set, OS+ performs better than TC.