AMR quality rating with a lightweight CNN
Table 4: Effects of our data de-biasing steps on model performance.
['data', 'method', 'Pearson’s  [ITALIC] ρ P', 'Pearson’s  [ITALIC] ρ R', 'Pearson’s  [ITALIC] ρ F1', 'error RMSE']
We can make three main observations: (i) from the first to the second de-biasing step, the baseline and our model have in common that Pearson’s ρ and the error decrease. While we cannot exactly explain why ρ decreases, it is clear that the error decrease is caused by the random re-split that balances the target scores in the three folds. (ii) The second de-biasing step leads to a decrease in ρ and an increase in error, for both models. This indicates that we have successfully removed shallow biases from the data that can give away the source of the parse. (iii) On all considered versions of the data, the method performs better than the baseline.