Topics to Avoid: Demoting Latent Confounds in Text Classification
Table 3: Classification accuracy with topic-demoting methods, TOEFL dataset.
['[EMPTY]', '[BOLD] In- Domain', '[BOLD] Out-of- Domain']
We evaluate our topical confound demotion method on the L1ID task. This corpus reflects eleven L1s: Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Spanish, Telugu, and Turkish. The training data include 11,000 authors (1,000 per L1) and the development set has 1,100 essays per L1. We evaluate on the development set. Each essay is also marked with a prompt ID which was given to the authors to write the essay. There are 8 prompts in total, based on which we construct 8 versions of train and test set. In each version, we remove essays marked with one of the prompts from both the train and the development sets, and consider the removed essays from the development set an “out-of-domain” test set. We begin with experiments on the TOEFL17 dataset, where predicting L1 is an easier task due to the lower proficiency of the authors. one demotes latent log-odds-based topics via gradient reversal (gr-lo), and another uses our proposed novel learning procedure but demotes baseline LDA topics (alt-lda). We report both in-domain accuracy and out-of-domain results; the latter is obtained by averaging the accuracy of each set “–PK” over K∈{0,…,7}.