TripleNet: Triple Attention Network for Multi-Turn Response Selection in Retrieval-based Chatbots
Table 2: Ablation studies on Ubuntu Dialogue Corpus. The letter ‘A’ stands for the subsection in triple attention, and ‘M’ the is triple matching part.
['[EMPTY]', 'R2@1', 'R10@1', 'R10@2', 'R10@5']
The performance of the model is similar to the baseline model DAM. This indicates that our four-level hierarchical representation may play a similar role to the five stacks Transformer in DAM. We then remove the triple attention part, which means we match the pairs ⟨C,R⟩ and ⟨Q,R⟩ with their original representation in each level; the score of R10@1 drops 1.4%, which shows the effect of triple attention. We also have tried to remove all the parts related to the query (-Query). That means the attention and matching parts are only calculated within the pair ⟨C,R⟩. It is worth mentioning that the information of the query is still contained at the end of the context. The performance also has a marked drop (1.6% in R10@1), which shows that it is necessary to model the query separately. To find out which subsection in those parts is more important, we remove each one of them. Triple attention matching ablation. The attentions in the other two pairs ⟨C,Q⟩ and ⟨Q,R⟩ all lead to a slight performance drop (0.3 and 0.5 in R10@1), which may be because they overlap with each other for updating the representation of the triple. Hierarchical representation ablation. To our surprise, when we remove char (-char) and context level calculation (-context), we observe that the reduction (0.5 in R10@1) is more significant than the other two, indicating that we should pay more attention to the lowest and highest level information. Also by removing the other two levels, there is also a significant reduction from TripleNet, which means each level of the three is indispensable for our TripleNet .