DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
Table 4: Inference latency (in seconds) on SQuAD datasets for BERT-base vs DeFormer-BERT-base, as an average measured in batch mode. On the GPU and CPU batch size is 32 and on the phone (marked by *) batch size is 1.
['[EMPTY]', 'BERT', 'DeFormer-BERT']
Device Results: To evaluate the impact on different devices, we deployed the models on three different machines (a GPU, CPU, and a mobile phone). On all devices, we get more than three times speedup.