Subword-augmented Embedding for Cloze Reading Comprehension
Table 3: Case study on CMRC-2017.
['Model', 'Operation', 'CMRC-2017 Valid', 'CMRC-2017 Test']
We also list different integration operations for word and subword embeddings. From the results, we can see that Word + BPE outperforms Word + Char which indicates subword embedding works essentially. We also observe that mul outperforms the other two operations, concat and sum. This reveals that mul might be more informative than concat and sum operations. The superiority might be due to element-wise product being capable of modeling the interactions and eliminating distribution differences between word and subword embedding. Intuitively, this is also similar to endow subword-aware “attention” over the word embedding. In contrast, concatenation operation may cause too high dimension, which leads to serious over-fitting issues, and sum operation is too simple to prevent from detailed information losing.