Language Models with Transformers
Table 5: Efficiency of different search methods on PTB and WT-2.
['[BOLD] Search Method', '[BOLD] Search Cost (GPU days)  [BOLD] PTB', '[BOLD] Search Cost (GPU days)  [BOLD] WT-2', '[BOLD] Method Class']
As can be seen, BERT-CAS is cheaper than all others. The results indicate that by leveraging the prior knowledge of the design of the neural networks for specific tasks, we could only optimize the architectures in a small confined sub-space, that leads to speed up the search process. For example, BERT-CAS is directly based on BERT, applying search upon such effective neural networks could facilitate the adaptation to similar tasks.