Question Answering via Integer Programming over Semi-Structured Knowledge
Table 7: Drop in solver scores (on the development set, rather than the hidden test set) when questions are perturbed
['Solver', 'Original Score (%)', '% Drop with Perturbation [t] absolute', '% Drop with Perturbation relative']
For each of the 108 development questions, we generate 10 new perturbed questions, using the 30 most frequently occurring words in step (5) above. While this approach can introduce new answer options that should be considered correct as well, only 3% of the questions in a random sample exhibited this behavior. As one might expect, the PMI approach suffers the most at a 33% relative drop. TableILP ’s score drops as well (since answer type matching isn’t perfect), but only by 12%, attesting to its higher resilience to simple question variation.