How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?
Table 2: Number prediction % accuracy with an increasing number of non-attractor intervening nouns (n). The distance between the main subject and the corresponding verb is held constant at 7 and the attractor count at 1.
['Model', 'n=0', 'n=1', 'n=2']
So far in the literature, when looking at intervening material in agreement tasks, the research has tended to focus on agreement attractors, the intervening nouns with the opposite number to the main noun Kuncoro et al. However, we posit that the role of non-attractor intervening nouns may also be important when understanding a model’s decisions. For long-range dependencies in agreement tasks, a model may be influenced by the presence of non-attractor intervening nouns instead of purely capturing the verb’s relationship with the main subject. Hence an analysis done solely based on the number of agreement attractors may be misleading. This indicates that the models are also using cues present in the vicinity of the main verb to predict its number, apart from agreement with the main noun.