Generating Semantically Valid Adversarial Questions for TableQA
Table 2: Automatic evaluation metrics on WikiSQL test set for generated adversarial questions. Delex represents entity delexicalization. Ecr, Qfr and Afr represent sentence level entity coverage rate, query and answer flip rate. The best scores for Seq2seq-based models are in bold.
['[EMPTY]', 'Original Questions', '[BOLD] BLEU -', '[BOLD] METEOR -', '[BOLD] SimLe -', '[BOLD] Ecr (%) -', '[BOLD] Qfr (%) -', '[BOLD] Afr (%) -', '[BOLD] Perplexity 816']
Since local models only make changes to a single token, compared to Seq2seq-based models, they can easily achieve much higher scores in BLEU and METEOR, which are based on string matching. However, such gap is reduced significantly between local and Seq2seq-based models in SimLe. This suggests that although questions generated by Seq2seq-based models are different in textual realization from the original questions, the semantic meaning is greatly preserved. Since SimLe is still based on the bag-of-words assumption that ignores syntactic structure, local models inevitably have higher scores than those generated by Seq2seq-based models from scratch, which contain more significant lexical and syntactic variations. We argue that these variations are beneficial because they mimic the diversity of human language. Due to the limitations of automatic metrics in evaluating semantic validity and fluency, we introduce human evaluation to substantiate our findings. We sample 100 questions from the WikiSQL test set and recruit three native expert annotators to annotate the adversarial examples generated by each model in the tasks of semantic validity and fluency. Here we focus on the Seq2seq-based models as well as the best-performed local attack models from