Sentence Simplification: Challenges for Encoder-Decoder Models
Table 1: Results for the end-to-end sentence simplification task. Our complete model is S4+gv+bce. Section 4 details the metrics. Values in bold are closest to the ground-truth.
['[EMPTY]', '[BOLD] B-1', '[BOLD] B-2', '[BOLD] B-3', '[BOLD] B-4', '[BOLD] Rouge', '[BOLD] Flesch', '[BOLD] Avg.Words', '[BOLD] Edit Dist.']
The base model for S4 is an encoder-decoder model with attention and word-copy feeding. We use suffixes to show the components added or removed in each model variant: -attn for removing attention, -feed for removing word-copy, +gv for allowing a mix of trainable and pre-trained embeddings, and +bce for training with the loss function for word-copying. Removing either the attention (S4-attn) or the feeding (S4-feed) causes a drop in BLEU and Rouge, indicating that the generated sentences are further from the simplified ground-truth. The attention is the more important of the two, with removal leading to an enormous 35.56 BLEU-4 point drop – we also observed semantic divergence from the input sentence after three to four words. Attention not only enables word copying but also reduces the effective depth of the network and avoids compressing the entire encoded sentence into a single fixed size vector. Adding either the pre-trained word-vectors or the custom loss function improves the BLEU and Rouge scores. S4+bce has a higher BLEU-4 by 4.9 points, and S4+gv+bce leads to a further 3.8 increase. Compared to the ground-truth sentences, S4+gv+bce is closest in sentence lengths, while S4+bce is closest in Flesch score. The attention component alone performs around 81 in B-1, and the model variants (+gv) that use a larger vocabulary perform slightly better. Better word replacement needs non-trivial amount of training data, and new ideas seem necessary for further improving attention alignment.