Contrastive Multi-document Question Generation
Table 4: Additional retrieval performance using self-critic (rennie2017self) baseline in the policy gradient, applicable to datasets with no oracle questions. It shows that our framework is also effective using a different baseline. The superscript null-neg denotes models that do not use negative attentions when generating questions. This shows the importance of the negative set in promoting specificity in the generated question. It further corroborates that the non-uniform weighted-sum scheme among D+ improves performance because not all documents in D+ revolve around the same topic, and the model learns to address this nature of the dataset through unequal weights and generate a more representative question.
['[BOLD] Model', '[BOLD] Out-Sample IR mAP', '[BOLD] Out-Sample IR RPrec', '[BOLD] Out-Sample IR MRR (=MRR@10)', '[BOLD] Out-Sample IR nDCG', '[BOLD] Search-Engine Augmented IR mAP', '[BOLD] Search-Engine Augmented IR RPrec', '[BOLD] Search-Engine Augmented IR MRR', '[BOLD] Search-Engine Augmented IR MRR@10', '[BOLD] Search-Engine Augmented IR nDCG']
Entropy regularization improves the search-engine augmented IR scores, in particular, MRR. Coordinator: The input size is 20 with the dimensionality of the embeddings and hidden states as 768. The number of recurrent layers is 2, with 4 attention heads in each layer. The epsilon value used in the layer normalization is set to 1e−5. The number of cluster embeddings is 2 (positive or negative). The standard deviation of the truncated normal initializer for weight matrices is 0.02. λ1,λ2,λ3=1.0,100.0,0.1. Maximum generation length is 20 tokens. We use the BERT (devlin2018bert) version of Adam optimizer (Kingma2014AdamAM) with weight decay of 0.01 and learning rate of 1e−5. We trained the coordinator model by maximizing Precision@10 with oracle questions as the policy gradient baseline. This shows that our proposed model framework is effective even with any of the two policy gradient baselines. Conceptually, the coordinator model would generate a common question that can better retrieve the documents from the positive set, aided by the negative set.