Continual and Multi-Task Architecture Search
Table 2: Ablation (val) results on CAS conditions.
['Model', 'Accuracy on QNLI']
CAS Condition Ablation : We also performed important ablation experiments to understand the importance of our block sparsity and orthogonality conditions in the CAS approach (as discussed in Sec. Our full model (with both Condition 2.1 and 2.2) achieves a validation performance of 74.1. Next, we separately experimented with each of Condition 2.1 and 2.2 and observe that using only one condition at a time is not able to maintain the performance w.r.t. step-1 QNLI performance (the decrease in score is statistically significant), suggesting that both of these two conditions are important for our CAS approach to work. Further, we remove both conditions and observe that the performance drops to 69.1. Finally, we also replaced the QNLI cell structure with the RTE cell structure along with removing both conditions and the performance further drops to 54.1. This shows that using the cell structure of the actual task is important.