A Gated Self-attention Memory Network for Answer Selection
Table 2: Results on the TrecQA and WikiQA datasets
['[BOLD] Model', '[BOLD] TrecQA MAP', '[BOLD] TrecQA MRR', '[BOLD] WikiQA MAP', '[BOLD] WikiQA MRR']
The full model [BERT + GSAMN+ Transfer Learning] outperforms the previous state-of-the-art methods by a large margin. Note that by simply fine-tuning the pre-trained BERT embeddings, one can easily achieve very competitive performance on both datasets. This is expected as BERT has been pre-trained on a massive amount of unlabeled data. However, our proposed techniques do add a significant amount of performance. The gain from using all of our proposed techniques is larger than the difference between fine-tuning BERT model compared to previous systems in the TrecQA dataset. We aim to analyze the relative effectiveness of different components of our full model. From the original BERT baseline, we add one component at a time and evaluate the performance of the partial models on the datasets. Transfer Learning] have better performance than the original BERT baseline. However, both of the partial variants still perform worse than the one with all the techniques. This shows that although each of our proposed components is effective by itself, we need to combine them together in order to achieve the best performance. It was an iterative process to arrive at the current design of GSAMN. We aim to analyze whether the improvement in performance comes from the inductive bias that we introduced into the architecture or simply from having more parameters due to added complexity. To this end, we evaluated the performances of two variants [BERT + Transformers] and [BERT + Transformers + Transfer Learning]. These model variants simply use two Transformer layers Vaswani et al. on top of BERT instead of using our GSAMN architecture. Finally, we have a comparison between our full model and the Compare-Aggregate framework. Most previous Compare-Aggregate architectures use traditional word embeddings such as word2vec Mikolov et al. et al. In contrast, our full model uses BERT which is an arguably more powerful language representation model. To this end, we implemented a Compare-Aggregate variant that uses dynamic-clip attention Bian et al. We use ELMo Peters et al. We use ELMo instead of BERT because BERT is in subword-level while one of the intuitions behind the Compare-Aggregate variant is about comparing word-level representations. In addition, we have tested the variant [BERT + Compare-Aggregate] but found it to be worse than the version [ELMo + Compare-Aggregate].