Constituency Parsing with a Self-Attentive Encoder
Table 2: Development-set F1 scores when attention is constrained to not exceed a particular distance in the sentence at test time only. In the relaxed setting, the first and last two tokens of the sentence can attend to any word and be attended to by any word, to allow for sentence-wide pooling of information.
['Distance', 'aF1 (strict)', 'F1 (relaxed)']
We can also examine our model’s use of long-distance context information by applying windowing to the attention mechanism. We begin by taking our trained model and windowing the attention mechanism at test-time only. When we began to investigate how the model makes use of long-distance attention, we immediately found that there are particular attention heads at some layers in our model that almost always attend to the start token. This suggests that the start token is being used as the location for some sentence-wide pooling/processing, or perhaps as a dummy target location when a head fails to find the particular phenomenon that it’s learned to search for. In light of this observation, we introduce a relaxed variation on the windowing scheme, where the start token, first word, last word, and stop token can participate in all possible uses of attention, but pairs of other words in the sentence can only attend to each other if they are within a given window. We include three other positions in addition to the start token to do our best to cover possible locations for global pooling by our model. Even when we allow global processing to take place at designated locations such as the start token, our model is able to make use of long-distance dependencies at up to length 40.