TextNAS: A Neural Architecture Search Space tailored for Text Representation
Table 7: Detailed settings for experiments of natural language inference.
['Exp', 'lr', 'training epoch', '[ITALIC] l2', 'dropout rate', 'penalization']
In NLI experiments, we evaluate the result model of TextNAS by training it from scratch. We set the dimension of hidden units as 512 for all layers in the sentence encoder and 2400 for the three fully-connected layers before softmax output. All 24 layers in the sentence encoder are linearly combined to produce the ultimate sentence embedding vector. All convolutions follow an ordering of ReLU, convolution operation and batch normalization. We also employ layer normalization after the outputs of bidirectional GRU and self-attention layers. Dropout is adopted on the output of each word-embedding, GRU and fully-connected layer. We set batch size as 32 and max input length as 128. Adam optimizer with cosine decay of learning rate and warm up over the first epoch are utilized to train the model. Learning rate: 1×10−4, 2×10−4, 3×10−4 Training epochs: 8, 12, 16, 20, 30 l2 regularization: 1×10−2, 2×10−2, 5×10−2, 1×10−1, 2×10−1, 5×10−1 Dropout ratio: 0.1, 0.2, 0.3 Penalization: 0, 1×10−1, 1×10−2, 1×10−3