ConveRT: Efficient and Accurate Conversational Representations from Transformers github.com/PolyAI-LDN/polyai-models
Table 5: Results on dstc7-ubuntu. (*) Scores for GPT and BERT taken from Vig:2019dstc.
['[EMPTY]', '[BOLD] R100@1', 'MRR']
It is an encouraging finding, given that multi-context ConveRT relies on simple context concatenation without any additional attention mechanisms. We leave the investigation of such more sophisticated models to integrate additional contexts for future work. Multi-context ConveRT can also match or even surpass the performance of another dual-encoder architecture from \newciteHumeau:2019arxiv. Their dual encoder (i.e., bi-encoder) is based on the BERT-base architecture Humeau et al. Training with that model is roughly 5× slower, and the pretraining objective is more complex: they use the standard BERT pretraining objective plus next utterance classification. Moreover, their model is trained on 32 v100 GPUs for 14 days, which makes it roughly 50× more expensive than ConveRT.