AMR quality rating with a lightweight CNN
Table 2: Main results. Pearson’s corr. coefficient (row 1-3) is better if higher; root mean squared error (RMSE, row 4-6) is better if lower. Results are averaged over 10 runs. The quality dimensions are explained in §2. † (‡): p < 0.05 (p < 0.005), significant difference in the correlations with two-tailed test using Fisher ρ to z transformation (fisher1915frequency).
['[EMPTY]', 'Quality Dim.', 'LG-LSTM', 'ours', 'change %']
With regard to estimated Smatch F1, we achieve a correlation with the gold scores of 0.695 Pearson’s ρ. This constitutes a significant improvement of appr. 5%. Similarly, recall and precision correlations improve by 6.36% and 3.83 % (from 0.676 to 0.719 and 0.600 to 0.623). While the improvement in predicted recall is significant at p<0.05 and p<0.005, the improvement in predicted precision is only significant at p<0.05. When we consider the RMSE, we find that the method improves over the baseline by -1.54% in estimated F1 in estimated Smatch F1 and -4.23% in estimated Smatch recall. On the other hand, the error in estimated precision remains unchanged.