Product Title Generation for Conversational Systems using BERT
Table 3: Evaluation Results. R-1, R-2, and R-L denote ROUGE-1, ROUGE-2, and ROUGE-L metrics.
['[BOLD] Method', '[BOLD] R-1', '[BOLD] R-2', '[BOLD] R-L', '[BOLD] avg. # of duplicates', '[BOLD] Human Evaluation']
We observe that transformer and BERT have ROUGE-1 F1 scores of 0.92715 and 0.92298 respectively, outperform seq2seq-based approaches in terms of both ROUGE and avg. # duplicates metrics. The Ptr-Net model outperforms basic seq2seq+attention approach, however, paradoxically Ptr-Net with coverage underperforms compared to Ptr-Net model. The coverage loss is present to specifically address the issue of repetition of words and instead of fixing it, lead to an increase in the avg. # of duplicates to 0.3041 from 0.239839. While transformer model does have a better ROUGE score than BERT model, BERT has lower repeated words in output (0.17262 compared to 0.18795) which has a greater impact on the title qualitatively. Since, BERT is pre-trained on a large corpus, it should be able to generalize better especially in low data scenarios. Human evaluation of generated titles reinforce this, showing that BERT performs better than transformer based model on output quality.