When Choosing Plausible Alternatives, Clever Hans can be Clever
Table 2: Applicability (App.), Productivity (Prod.) and Coverage (Cov.) of the various words in the alternatives of the COPA dev set.
['Cue', 'App.', 'Prod.', 'Cov.']
Table 2 shows the five tokens with highest coverage. For example, a is the token with the highest coverage and appears in either a correct alternative or wrong alternative in 21.2% of COPA training instances. Its productivity of 57.5% expresses that it appears in in correct alternatives 7.5% more often than expected by random chance. This suggests that a model could rely on such unbalanced distributions of tokens to predict answers based only on alternatives without understanding the task.