Training Classifiers with Natural Language Explanations
Table 3: F1 scores obtained by a classifier trained with BabbleLabble (BL) using 30 explanations or with traditional supervision (TS) using the specified number of individually labeled examples. BabbleLabble achieves the same F1 score as traditional supervision while using fewer user inputs by a factor of over 5 (Protein) to over 100 (Spouse).
['# Inputs', 'BL 30', 'TS 30', 'TS 60', 'TS 150', 'TS 300', 'TS 1,000', 'TS 3,000', 'TS 10,000']
On average, it took the same amount of time to collect 30 explanations as 60 labels. We observe that in all three tasks, BabbleLabble achieves a given F1 score with far fewer user inputs than traditional supervision, by as much as 100 times in the case of the Spouse task. Because explanations are applied to many unlabeled examples, each individual input from the user can implicitly contribute many (noisy) labels to the learning algorithm.