Evaluating Scoped Meaning Representations
Table 3: Results of comparing 19,000 Boxer-produced DRSs to DRSs produced by amr2drs, for different number of restarts. For three or more restarts, we always use the smart role and concept mapping.
['[BOLD] Restarts', '[BOLD] P%', '[BOLD] R%', '[BOLD] F1%', '[BOLD] Time (h:m:s)']
We compare the sets of DRSs using different numbers of restarts to find the best trade-off between speed and accuracy. The optimal scores are obtained using a Prolog script that performs an exhaustive search for the optimal mapping. As expected, increasing the number of restarts benefits performance. \newcitesmatch:13 consider four restarts the optimal trade-off between accuracy and speed, showing no improvement in F-score when using more than ten restarts. We see that for long sentences, 5 and 10 restarts are not sufficient to get close to the optimal, while for short sentences 5 restarts might be considered enough. In general, the best trade-off between speed and accuracy is approximately 20 restarts.