Learning Norms from Stories: A Prior for Value Aligned Agents
Table 2: Results for Goofus & Gallant classification experiments.
['[BOLD] Model', '[BOLD] Test acc', '[ITALIC] F1 [BOLD] -score', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] MCC']
First, it shows that humans have strong agreement with the G&G ground truth labels. Among the non-transformer models, DPCNN better classifies normative and non-normative behavior from the G&G dataset. This is likely because the CNN can identify the global sentence structure better than a simple bi-directional LSTM cell. While the BERT-Base and XLNet-Base models struggle to classify events from the G&G corpus (achieving accuracies of %61.4 and %60.6 respectively), fine-tuning drastically improves each model’s performance. BERT-GG obtains the best results in each of our metrics, obtaining a 21.33% accuracy improvement over the DPCNN.