Gated Convolutional Bidirectional Attention-based Model for Off-topic Spoken Response Detection
Table 5: The performance of GCBiA with negative sampling augmentation method conditioned on over 0.999 on-topic recall.
['Model', 'Seen PPR3', 'Seen AOR', 'Unseen PPR3', 'Unseen AOR']
To augment training data and strengthen the generalization of the off-topic response detection model for unseen prompts, we proposed a new and effective negative sampling method for off-topic response detection task. Comparing with the previous method of generating only one negative sample for each positive one, we generated two. The first one is chosen randomly as before, and the second one consists of words shuffled from the first one. This method contributes to the diversity of negative samples of training data. The size of our training data reaches 1.67M, compared with 1.12M in the previous negative sampling method. To make training data balanced, we gave the weight of positive and negative samples: 1 and 0.5, respectively. Our model GCBiA equipped with negative sampling augmentation can achieve 88.2% and 69.1% average off-topic response recall on seen and unseen prompts, conditioned on 0.999 on-topic recall.