Constituency Parsing with a Self-Attentive Encoder
Table 3: Development-set F1 scores when attention is constrained to not exceed a particular distance in the sentence during training and at test time. In the relaxed setting, the first and last two tokens of the sentence can attend to any word and be attended to by any word, to allow for sentence-wide pooling of information.
['Distance', 'aF1 (strict)', 'F1 (relaxed)']
Note that when a window of size 10 was imposed at training time, this was per-layer and the series of 8 layers actually had an effective context size of around 80 – which was still insufficient to recover the performance of our full parser (with either approach to windowing). The side-by-side comparison of strict and relaxed windowing shows that the ability to pool global information, using the designated locations that are always available in the relaxed scheme, consistently translates to accuracy gains but is insufficient to compensate for small window sizes. This suggests that not only must the information signal from long-distance tokens be available in principle, but that it also helps to have this information be directly accessible without an intermediate bottleneck.