Studying the Inductive Biases of RNNswith Synthetic Variations of Natural Languages
Table 5: Accuracy (A) and recall (R) in predicting subject and object agreement with different case systems.
['Case system', 'Flexible word order Subject A', 'Flexible word order Object A/R', 'VOS Subject A', 'VOS Object A/R', 'OVS Subject A', 'OVS Object A/R']
Unambiguous case marking dramatically improved subject and object plurality prediction compared with the previous experiments; accuracy was above 98% for all three word orders. Partial syncretism hurt performance somewhat relative to the unambiguous setting (except with flexible word order), especially for object prediction. The fully syncretic case system, which marked only the plurality of the head of each argument, further decreased performance. At the same time, even this limited marking scheme was helpful: accuracy in the most challenging setting, flexible word order (subject: 96.0%; object: 86.1%), was not very different from the results on unmodified English (95.7% and 90.0%). This contrasts with the poor results on the flexible setting without cases (subject: 88.6%; object: 60.2%). On the rigid orders, a fully syncretic system still significantly improved agreement prediction. The moderate effect of case syncretism on performance suggests that most of the benefits of case marking stems from the overt marking of the heads of all arguments.