Finding Syntactic Representations in Neural Stacks
Table 2: Number prediction accuracies attained by the three stack RNN classifiers and the LSTM baseline.
['[BOLD] Number of  [BOLD] Attractors', '[BOLD] Stack ( [ITALIC] ut=1)', '[BOLD] Stack ( [ITALIC] dt=1)', '[BOLD] Stack', '[BOLD] LSTM']
While the stack classifier with ut fixed to 1 and the LSTM baseline achieve the best overall accuracy, the stack with unrestricted ut and sigmoid dt and the stack with dt fixed to 1 exceed the baseline on sentences with at least 2 attractors. We take this to suggest that the hierarchical bias provided by the stack can improve performance on syntactically complex cases. Overall, our stack language models show no improvement over the LSTM baseline in terms of perplexity and classification accuracy. Although the ut=1 language model is comparable to the LSTM on these metrics, it ultimately achieves worse scores than the baseline. However, we have now seen that the pushing behavior of the model reflects subcategorization properties of lexical items that play an important role in determining their syntactic behavior, and that these properties allow reasonable parses to be extracted from this model. These observations show that the ut=1 model has learned to encode structural representations using the stack.