Contextual Embeddings: When Are They Worth It?
Table 1: Performance and sample complexity of random (R) and GloVe (G) relative to BERT (B) for NER, sentiment analysis (Sent.), and language understanding (GLUE) tasks. Second column shows BERT accuracy; third/fourth columns show the accuracy gap between BERT and random/GloVe; fifth/sixth columns show sample complexity ratios, the largest n∈{1,4,16,64,256} for which BERT outperforms random/GloVe when trained on n-times less data. We observe that non-contextual embeddings can often (1) perform within 10% absolute accuracy of the contextual embeddings, and (2) match the performance of contextual embeddings which are trained on 1x-16x less data. This sheds light on a tradeoff between the upfront cost of labeling training data and the inference-time computational cost of the embeddings.
['[BOLD] Task', '[BOLD] Task', '[BOLD] Performance gap', '[BOLD] Performance gap', '[BOLD] Performance gap', '[BOLD] Sample  [BOLD] complexity ratio', '[BOLD] Sample  [BOLD] complexity ratio']
We show that the amount of downstream training data is a critical factor in determining the relative performance of contextual vs. non-contextual embeddings. In light of this, ML practitioners may find that for certain real-world tasks the large gains in efficiency are well worth the cost of labeling more data.