Decomposing GeneralizationModels of Generic, Habitual, and Episodic Statements
Table 1: Bias (log-odds) for answering true
['[EMPTY]', '[BOLD] Property', '^ [ITALIC] β0', '^ [ITALIC] σann', '^ [ITALIC] σitem']
To estimate pe for each property, we fit a random effects logistic regression to the binary responses for that property, with random intercepts for both annotator and item. The fixed intercept estimate ^β0 for this model is an estimate of the log-odds that the average annotator would answer true on that property for the average item; and the random intercepts give the distribution of actual annotator (^σann) or item (^σitem) biases. We note a substantial amount of variability in the bias different annotators have for answering true on many of these properties. This variability is evidenced by the fact that ^σann and ^σitem are similar across properties, and it suggests the need to adjust for annotator biases when analyzing these data, which we do both here and for our bulk annotation. To adjust for these biases, we construct a normalized score for each predicate and argument using mixed effects logistic regressions. These mixed effects models all had (i) a hinge loss with margin set to the normalized confidence rating; (ii) fixed effects for property—particular, kind, and abstract for arguments; particular, hypothetical, and dynamic for predicates—token, and their interaction; and (iii) by-annotator random intercepts and random slopes for property with diagonal covariance matrices. The rationale behind (i) is that true should be associated with positive values; false should be associated with negative values; and the confidence rating should control how far from zero the normalized rating is, adjusting for the biases of annotators that responded to a particular item.