Adversarial Training for Community Question Answer Selection Based on Multi-scale Matching
Table 1: Performance on SemEval 2017 dataset. “Contrs” denotes non-primary submission.
['Method', 'MAP', 'MRR']
The term “adversarial” means that we employ an additional generative model to produce challenging adversarial samples to fool the discriminative model during training. From the table we can see that using Multi-scale Matching consistently improves the performance. With only a discriminative model, MAP is increased from 14.67 to 14.80. With adversarial training, MAP is increased from 17.25 to 17.91. For single-scale model, the MAP is increased from 14.67 to 17.25, while for multi-scale model, the number is increased from 14.80 to 17.91. This demonstrates the effectiveness of utilizing a generative model to produce challenging negative samples. Since in SemEval 2017 dataset each question is associated with 100 candidate answers and only 2.8% of them are labeled as relevant, the class labels are severely imbalanced. By having a generative model, we are able to select more challenging adversarial samples as training proceeds, resulting in a more robust discriminator. The MAP is 13.31 and 14.33 for single-scale and multi-scale model respectively, significantly outperforming a random baseline and a information-retrieval (IR) based baseline. This shows that our generators also learned useful information. Consequently, the negative samples selected by the generators are much more challenging that those selected randomly or by a IR based approach.