Game-Based Video-Context Dialogue
Table 3: Performance of our baselines, discriminative models, and generative models for recall@k metrics on our Twitch-FIFA test set. C and V represent chat and video context, respectively.
['Models', 'r@1', 'r@2', 'r@5']
We first discuss results of our simple non-trained and trained baselines (see Sec. The ‘Most-Frequent-Response’ baseline, which just ranks the 10-sized response retrieval list based on their frequency in the training data, gets only around 10% recall@1. Our other non-trained baselines: ‘Chat-Response-Cosine’ and ‘Nearest Neighbor’, which ranks the candidate responses based on (Twitch-trained RNN encoder’s vector) cosine similarity with chat-context and K-best training contexts’ response vectors, respectively, achieves slightly better scores. We also show that our simple trained baselines (logistic regression and nearest neighbor) also achieve relatively low scores, indicating that a simple, shallow model will not work on this challenging dataset. Our dual encoder models are significantly better than random choice and all our simple baselines above, and further show that they have complementary information because using both of them together (in ‘Triple Encoder’) improves the overall performance of the model. Finally, we show that our novel TriDAF model with self-attention performs significantly better than the triple encoder model. Next, we evaluate the performance of our generative models with both retrieval-based recall@k scores and phrase matching-based metrics as discussed in Sec. Starting with a simple sequence-to-sequence attention model with video only, chat only, and both video and chat encoders, the recall@k scores are better than all the simple baselines. Moreover, using both video+chat context is again better than using only one context modality. Finally, we show that the addition of the bidirectional attention flow mechanism improves the performance in all recall@k scores. Das et al.