Compositional Sentence Representation from Character within Large Context Text
Table 2: Size of compositional model at each level, represented by (# layers) × (# memory in each layer). Note that the complexity of the model increases as the level of hierarchy increases, following the assumption that the complexity of composition increases as the level of language increases.
['[EMPTY]', 'CC', 'CW', 'CS']
The configuration of the HCRN is represented by the hierarchy of the compositional model and its size, CCsize−CWsize−CSsize, where the size is represented by the number of layers and the number of hidden units in each layer. Initialization of CC: Random VS. Pre-trained The test set classification error rate of sentence-hierarchy learning with and without the pre-trained CC are compared to evaluate how the pre-trained CC provides useful initialization for sentence-hierarchy learning. After that, the whole architecture consisting of the CC, CW, and MLP is jointly trained. In addition, pre-training on two different training dataset sizes (50 % and 100 % ) are compared. Pre-training consistently reduces the test error rate on the various architectures as shown in Fig. 3(a). Moreover, as shown in Fig. 3(b), improvement by pre-training is significant especially when fewer training data are available, where the model is liable to overfit to a given small number of data.