Continual and Multi-Task Architecture Search
Table 1: Test results on GLUE tasks for various models: Baseline, ENAS, and CAS (continual architecture search). The CAS results maintain statistical equality across each step.
['Models', 'QNLI', 'RTE', 'WNLI']
Baseline Models: We use bidirectional LSTM-RNN encoders with max-pooling Conneau et al. Further, we used the ELMo embeddings Peters et al. On top of these strong baselines, we add ENAS approach. ENAS Models: that we only replace the LSTM-RNN cell with our ENAS cell, rest of the model architecture in ENAS model is same as our baseline model. Next, we apply our continual architecture search (CAS) approach on QNLI, RTE, and WNLI, where we sequentially allow the model to learn QNLI, RTE, and WNLI (in the order of decreasing dataset size, following standard transfer setup practice) We train on QNLI task, RTE task, and WNLI task in step-1, step-2, and step-3, respectively. We observe that even though we learn the models sequentially, we are able to maintain performance on the previously-learned QNLI task in step-2 (74.1 vs. 74.2 on validation set which is statistically equal, and 73.6 vs. 73.8 on test). Next, we observe a similar pattern when we extend CAS to the WNLI dataset (scores are statistically equal on validation set). Further, if we compare the performance of QNLI from step-1 to step-3, we see that they are also stat. equal on val set (73.9 vs. 74.2). This shows that our CAS method can maintain the performance of a task in a continual learning setting with several steps.