Generating Semantically Valid Adversarial Questions for TableQA
Table 3: Human evaluation results on 100 questions per task from each model. Validity is the percentage of semantically valid questions. Fluency is the mean rank of each model over sampled questions. The best scores are in bold. The Fleiss’ Kappa for inter-annotator agreement is 61.0, which falls in the interval of substantial agreement.
['[EMPTY]', '[BOLD] Validity (%)↑', '[BOLD] Fluency\xa0(rank)↓']
For example, 90.3% of the generated questions of our Wseq-S are semantically valid, fortifying the effectiveness of entity delexicalization and SimiLe. Also, all Seq2seq-based models achieve higher semantic validity than the local models. It demonstrates the capability of Seq2seq-based models in generating more semantically equivalent questions. Although local models only introduce minor word-level changes, most of them actually greatly alter the meaning of the original questions and make the generated adversarial questions semantically invalid. Additionally, it suggests that existing automatic metrics are not sufficient to evaluate semantic validity. On the other hand, SAGE scores lower than Wseq-S and Wseq in terms of semantic validity. However, given its high flip rates, SAGE still generates many more semantically valid adversarial questions than any other Seq2seq-based model. Similar to automatic fluency evaluation (perplexity), Wseq tops all models in human evaluation, which only ranks behind the original question by a small margin. This verifies that Wseq is effective in improving generation fluency. SAGE yields good fluency, which is significantly better than the Seq2seq model without entity delexicalization. The decrease in fluency from SAGE compared to Wseq are caused by two reasons: (1) Adding the adversarial loss harms text quality; (2) SimiLe drives the model to generate questions closer to the original ones, which increases semantic validity but sacrifices fluency, which is the same case in Wseq-S.