ConveRT: Efficient and Accurate Conversational Representations from Transformers github.com/PolyAI-LDN/polyai-models
Table 3: R100@1×100% scores on Reddit test set and amazonQA. polyai-dual and ConveRT networks are fine-tuned on the training portion of amazonQA. Note that amazonQA by design supports only single-context response selection.
['[EMPTY]', 'Reddit', 'AmazonQA']
Response Selection on Reddit. Even single-context ConveRT achieves peak performance in the task, with substantial gains over the previous best reported score of \newciteHenderson:2019acl. It also substantially outperforms all the other models which were not pretrained directly on the response selection task, but on a standard LM task instead. The strongest baselines, however, are two dual-encoder architectures (i.e., use-large, use-qa and polyai-dual); this illustrates the importance of explicitly distinguishing between inputs/contexts and responses when modeling response selection. Multi-context ConveRT achieves a state-of-the-art Reddit response selection score of 71.8% We observe similar benefits in other reported response selection tasks. We also note the results of 1) using only the sub-network that models the interaction between the immediate context and the response (i.e., the hTxhy interaction), and 2) artificially replacing the concatenated extra contexts z with an empty string. The respective scores are 65.7% and 65.6%. This suggests that multi-context ConveRT is also applicable to single-context scenarios when no extra contexts are provided for the target task. Other Response Selection Tasks. We see similar trends as with Reddit evaluation. Fine-tuned ConveRT reaches a new state-of-the-art score, and the strongest baselines are again dual-encoder networks. Fine-tuned polyai-dual, which was pretrained on exactly the same data, cannot match ConveRT’s performance.