Relevance-Promoting Language Model for Short-Text ConversationThe work described in this paper is substantially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14204418). It was mainly done when Xin Li was an intern at Tencent AI Lab.
Table 1: Experimental results on the automatic metrics. The best results are in bold.
['[BOLD] Model', '[BOLD] Relevance', '[BOLD] Relevance', '[BOLD] Relevance', '[BOLD] Relevance', '[BOLD] Relevance', '[BOLD] Diversity', '[BOLD] Diversity']
In terms of Bleu, the proposed model with beam search decoding, namely, OURS-bm, consistently achieve the best scores. Besides, OURS-bm outperforms all compared models on the keyword-overlapping-based Hit metrics, suggesting that our model, armed with Supervised Source Attention component (SSA) and Topic Inference (TI) component, is beneficial for the generation of informative topical words related to the query. Surprisingly, OURS-bm also obtains better Dist metrics than the baseline models. After replacing the beam search with top-k sampling, our model (OURS-tk) is further enhanced in diversity modeling, reaching 0.107 and 0.544 on Dist-1 and Dist-2 respectively. In order to track the source of the performance gains, we also conduct the ablation study on the OURS-tk. As expected, the model without relevance-promoting design, i.e., OURS-tk w/o SSA & TI, is the worst one on the relevance metrics. OURS-k w/o SSA and OURS-tk w/o TI, the variants incorporating either TI or SSA for relevance modeling, boost the Relevance score by ∼17% and ∼18% respectively. Although they are comparable on the relevance metrics but the former achieves higher diversity scores (Dist-2: 0.521 v.s. 0.441). We attribute this phenomenon to the TI component, which exploits the usage of more related topical words mentioned in the multiple references. With the help of both SSA component and TI component, OURS-tk becomes the best model on Relevance and Dist metrics, demonstrating the necessity of the relevance modeling for the transformer language model. Another interesting finding is that the SSA component decreases the Fluency score (see the results of OURS-tk w/o TI), which indicates that fighting against explanation-away issue by incorporating additional query context may be coupled with corrupting the language model.