TextNAS: A Neural Architecture Search Space tailored for Text Representation
Table 6: Detailed settings for experiments of text classification.
['Exp', 'batch size', 'max length', '[ITALIC] l2', 'lr', 'sliding window', 'hidden size']
In all the experiments, we apply dropout (ratio=0.5) to the embedding layers, final output layers and self-attention layers. In addition, in the bidirectional GRU layers, we apply dropout (ratio=0.5) on the input and output tensors. Besides, for several time-consuming experiments, we employ sliding window trick to accelerate the training procedure. Given a sentence, we utilize a sliding window to segment the long input sentence into several sub-sentences, where window_size and stride are pre-defined hyper-parameters. The sub-sentences are fed separately to the neural network to output fixed-length vector representation for each sub-sentence. Then, a max pooling operator is applied on top to calculate the vector representation for the entire sentence. In all experiments using sliding window, we set window_size as 64 and stride as 32.