Training Classifiers with Natural Language Explanations
Table 6: F1 scores obtained using explanations as functions for data programming (BL) or features (Feat), optionally with no discriminative model (-DM) or using a perfect parser (+PP).
['[EMPTY]', 'BL-DM', 'BL', 'BL+PP', 'Feat', 'Feat+PP']
Once we have relevant logical forms from user-provided explanations, we have multiple options for how to use them. Srivastava et al. propose using these logical forms as features in a linear classifier, essentially using a traditional supervision approach with user-specified features. We choose instead to use them as functions for weakly supervising the creation of a larger training set via data programming Ratner et al. We attribute this difference primarily to the ability of data programming to utilize a larger feature set and unlabeled data. We also observe qualitatively that data programming exposes the classifier to additional patterns that are correlated with our explanations but not mentioned directly. For example, in the Disease task, two of the features weighted most highly by the discriminative model were the presence of the trigrams “could produce a” or “support diagnosis of” between the chemical and disease, despite none of these words occurring in the explanations for that task.