An Empirical Study on Explainable Prediction of Text Complexity: Preliminaries for Text Simplification
Table 9. Edit distance (ED) and Fréchet Embedding Distance (FED) scores for simplifications of simple sentences.
['[BOLD] Dataset', '[BOLD] Sentence Pairs', '[BOLD] Metric', '[BOLD] ACCESS', '[BOLD] DMLMTL']
We’d also understand how large the changes are and whether they are in the wrong direction (e.g., whether the models oversimplify the input and/or deviate from its meaning). We compute the Edit Distance (ED) and Fréchet Embedding Distance (FED) ED measures lexical difference of two sentences and FED measures their semantic difference in a pre-trained embedding space. The changes by DMLMTL are worse: 6 to 13 edits on average, again towards oversimplification. The pre-trained models also tend to deviate the simple sentence from its original meaning: the semantic distance (FED) between the aligned simple sentence and its complex version is almost doubled after it is further simplified by ACCESS, and even larger (e.g., 6 times on Newsela) when it’s changed by DMLMTL. All these qualitative and quantitative results suggest that the state-of-the-art, pre-trained models tend to oversimplify and distort the meanings of input that is already simple. Evidently, the lack of transparency and explainability has limited the application of these end-to-end black-box models in reality, especially to out-sample data, context, and domains. The pitfall can be avoided with the proposed pipeline and simply with explainable complexity prediction as a preliminary step.