Learning to Embed Words in Context for Syntactic Tasks
Table 7: Attachment F1 (%) on validation set using different models and window sizes. For TE columns, the input does not include any type embeddings at all, only token embeddings. Best result in each column is in boldface.
['[ITALIC] w or  [ITALIC] w′', 'Baseline', 'DNN TE', 'seq2seq TE']
The baseline head predictor actually does best with w=0. The predictors with token embeddings are able to leverage larger context: with DNN token embeddings, performance is best with w′=1 while with seq2seq token embeddings, performance is strong with w′=1 and 2. When using token embeddings, we actually found it beneficial to drop the center word type embedding from the input, only using it indirectly through the token embedding functions. We use w=−1 to indicate this setting.