How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?
Table 4: Accuracy of models on targeted syntactic evaluation. RC: Relative Clause, PP: Prepositional Phrase, VP : Verb Phrase. Closeness in the mean arithmetic rank of models (other than SRNs) across tasks suggests that within the current space of sequential recurrent models, none dominates the others.
['[EMPTY]', 'SRN', 'GRU', 'LSTM', '[BOLD] DRNN', '[BOLD] SDRNN', '[BOLD] Ab-DRNN', 'ON-LSTM']
Word-level language modeling is a task that helps in the evaluation of the model’s capacity to capture the general properties of language beyond what is tested in specialized tasks focused on, e.g., subject-verb agreement. We use perplexity to compare our model’s performance against standard sequential recurrent architectures. Targeted syntactic evaluation Marvin and Linzen This includes subject-verb agreement, reflexive anaphora, and negative polarity items (NPI). In the case of long-range dependencies and NPI involving relative-object clauses, our models perform substantially better than LSTMs. High variability in the performance of the models in the case of NPIs might be due to non-syntactic cues as pointed out by Marvin and Linzen However, SRNs alone are not sufficient for most purposes.