Dual Multi-head Co-attention for Multi-choice Reading Comprehension
Table 11: Results with and without self-attention on DREAM dataset. “SA” means self-attention and “CA” means co-attention. “SA+CA” means straightforwardly using CA to replace SA in ALBERT.
['model', 'dev', 'test']
Our overall architecture can be split into two steps from the view of attention, of which the first is self-attention and the second is co-attention. To examine whether the structure can be further simplified, that is only using co-attention, we straightforwardly change all of the Multi-head Self-attention of ALBERT model to our Dual Multi-head Co-attention, while still using its pre-trained parameters. To conclude, a better way for modeling is our pre-trained LM plus DUMA model, which is to firstly build a global relationship using self-attention of the well trained encoder and then further enhance the relationship between passage and question-answer and distill more matching information using co-attention.