Stress Test Evaluation for Natural Language Inference
Table 8: Model Performance on grammaticality test
['[BOLD] Sys  [BOLD] tem', 'Adj SWAP  [BOLD] Mat', 'Adj SWAP  [BOLD] Mis', 'KB Swap  [BOLD] Mat', 'KB Swap  [BOLD] Mis', 'CN Swap  [BOLD] Mat', 'CN Swap  [BOLD] Mis', 'FN Swap  [BOLD] Mat', 'FN Swap  [BOLD] Mis']
This stress test consists of an adversarial example set which tests model robustness to spelling errors. Spelling errors occur often in MultiNLI data, due to involvement of Turkers and noisy source text, which is problematic as some NLI systems rely heavily on word embeddings. We construct a stress test for “spelling errors” by performing two types of perturbations: AdjSWAP: Swap adjacent characters in a single word sampled randomly from the hypothesis. For example, “I saw Tipper with him at teh movie.”. KBSWAP: Substitute a single alphabetical character randomly sampled from the hypothesis with the character next to it on the English keyboard. For example, “Agencies have been further restricted and given less choice in selecting contractimg methods.” We additionally perform perturbations on only function words (conjunctions, pronouns and articles), and on only content words (nouns and adjectives) in the hypothesis to study the effects. We do not address perturbations in verbs and adverbs in the content word vs. function word analysis.