Long Short-Term Memoryas a Dynamically Computed Element-wise Weighted Sum
Table 3: Performance on the universal dependencies parsing benchmark, measured by unlabeled (UAS) and labeled attachment score (LAS).
['[BOLD] Model', '[BOLD] UAS', '[BOLD] LAS']
We directly use their released implementation, which is evaluated on the Universal Dependencies English Web Treebank v1.3 Silveira et al. In our experiments, we use the existing hyperparameters and only replace the LSTMs with the simplified architectures. This latter result indicates that the additive recurrent connection in the memory cell – and not the multiplicative recurrent connections in the content layer or in the gates – is the most important computational element in an LSTM. As a corollary, this result also suggests that a weighted sum of context words, while mathematically simple, is a powerful model of contextual information.