Semantics-aware BERT for Language Understanding
Table 6: Analysis on SNLI and SQuAD 2.0 datasets.
['[BOLD] Model', '[BOLD] SNLI  [BOLD] Dev', '[BOLD] SQuAD 2.0  [BOLD] EM', '[BOLD] SQuAD 2.0  [BOLD] F1']
Since SemBERT absorbs contextual semantics in a deep processing way, we wonder if a simple and straightforward way integrating such semantic information may still work, thus we concatenate the SRL embedding with BERT subword embeddings for a direct comparison, where the semantic role labels are copied to the number of subwords for each original word, without CNN and pooling for word-level alignment. From the results, we observe that the concatenation would yield an improvement, verifying that integrating contextual semantics would be quite useful for language understanding. However, SemBERT still outperforms the simple BERT+SRL model just like the latter outperforms the original BERT by a large performance margin, which shows that SemBERT works more effectively for integrating both plain contextual representation and contextual semantics at the same time.