Contextual Embeddings: When Are They Worth It?
Table 2: For our complexity, ambiguity, and unseen prevalence metrics, we slice the validation set using the median metric value, and compute the average error rates for BERT and random on each slice. We show that the gap between BERT and random errors is larger on the slice above the median than below it in 19 out of 21 cases, in absolute (Abs.) and relative (Rel.) terms.
['[BOLD] Task', '[BOLD] Complexity Abs.', '[BOLD] Complexity Rel.', '[BOLD] Ambiguity Abs.', '[BOLD] Ambiguity Rel.', '[BOLD] Unseen Abs.', '[BOLD] Unseen Rel.']
we show that for each of the metrics defined above, the accuracy gap between BERT and random embeddings is larger on inputs for which the metrics are large. In particular, we split each of the task validation sets into two halves, with points with metric values below the median in one half, and above the median in the other. We see that in 19 out of 21 cases, the accuracy gap between BERT and random embeddings is larger on the slice of the validation set corresponding to large metric values, validating our hypothesis that contextual embeddings provide important boosts in accuracy on these points.