Automatic Generation of Headlines for Online Math Questions
Table 3: Comparison of different models on the EXEQ-300k and OFEQ-10 test sets for F1 scores of R1 (ROUGE-1), R2 (ROUGE-2), RL (ROUGE-L), BLEU-4, and METEOR.
['Models', 'EXEQ-300k R1', 'EXEQ-300k R2', 'EXEQ-300k RL', 'EXEQ-300k BLEU-4', 'EXEQ-300k METEOR', 'OFEQ-10k R1', 'OFEQ-10k R2', 'OFEQ-10k RL', 'OFEQ-10k BLEU-4', 'OFEQ-10k METEOR']
All models perform better on EXEQ-300k than OFEQ-10k. For extractive models, we find that Lead obtains a good performance on EXEQ-300k, while TextRank performs well on OFEX-10k. Since OFEX-10k contains more sentences for each question, TextRank is more likely to pick out the accurate sentence. Unsurprisingly, abstractive models perform better than extractive models on both datasets. Compared to ordinary Seq2Seq, PtGen gets better performance, since it uses a copying strategy to directly copy tokens from the source question. The transformer can outperform PtGen, which implies that by utilizing multi-head attention mechanism, we obtain a better learning of representation. MathSum significantly outperforms other models for all evaluation metrics on both datasets. Thus, MathSum initially addresses some of the challenges of NAME task and generates satisfactory headlines for questions.