A Document-grounded Matching Network for Response Selection in Retrieval-based Chatbots
Table 4: Performance of DGMN across different length of grounded documents on all data sets.
['Doc Length', 'Original Persona (0,25]', 'Original Persona (25,30]', 'Original Persona (30,35]', 'Original Persona (35,52]', 'Revised Persona (0,30]', 'Revised Persona (30, 35]', 'Revised Persona (35, 40]', 'Revised Persona (40,55]', 'CMUDoG (0, 150]', 'CMUDoG (150, 250]', 'CMUDoG (250, 350]', 'CMUDoG (350,515]']
Finally, we study the relationship between the performance of DGMN and document length by binning text examples in both data into different buckets according to the document length. On the PERSONA-CHAT data, both short profiles and long profiles lead to performance drop, while on the CMUDoG data, the longer the documents are, the better the performance of DGMN is. The reason behind the difference might be that profiles in the PERSONA-CHAT data are handcrafted by crowd workers, and thus semantics among different sentences are relatively independent, while documents in the CMUDoG data come from Wikipedia, and there is rich semantic overlap among sentences. Therefore, short profiles contain less useful information and long profiles contain more irrelevant information, and both will make the matching task more challenging. On the other hand, the longer a wiki document is, the more relevant information it can provide to the matching task.