Dual Multi-head Co-attention for Multi-choice Reading Comprehension
Table 5: Comparison between DUMA and TB-based DUMA on DREAM dataset.
['model', 'dev', 'test']
The kernel module of Transformer is one-layer encoder block, which consists of Multi-head Attention, layer normalization and Feed-Forward Network (FFN), as described in [transformer]. In this paper, we call this module Transformer Block (TB). In consideration of the extensive application and great success of TB for global encoding, we investigate the difference between it and Multi-head Attention while modeling relationships between passage and question for MRC tasks. Thus, we use TB to replace the Multi-head Attention module in our DUMA to form TB-based DUMA (TB-DUMA) model. It can be seen that TB has no obvious difference with Multi-head Attention in modeling relationships. However, our proposed DUMA holds more straightforward structure and equally efficient performance.