Unsupervised Learning of Syntactic Structure with Invertible Neural Projections
Table 3: Unsupervised POS tagging results of our approach on WSJ, with random initialization of syntax model.
['[BOLD] System', '[BOLD] M-1', '[BOLD] VM']
In the above experiments we initialize the structured syntax components with the pre-trained Gaussian or discrete baseline, which is shown as a useful technique to help train our deep models. We further study the results with fully random initialization. While the performance with 4 layers is comparable to the pre-trained Gaussian initialization, deeper projections (8 or 16 layers) result in a dramatic drop in performance. This suggests that the structured syntax model with very deep projections is difficult to train from scratch, and a simpler projection might be a good compromise in the random initialization setting.