Learning Word Embeddings with Domain Awareness
Table 1: Text classification results on ATIS and IMDB.
['Embeddings', 'ATIS', 'IMDB']
Overall, in-domain word embeddings tend to outperform out-of-domain ones when in-domain data is relatively large and in a contrary when in-domain data is limited. The aggression methods, i.e., S⊕T, Avg(S,T), normally perform in between the embeddings from the source and the target domain; while the embeddings trained on the concatenated corpus (S+T) tend to perform similar to the ones trained in the source domain because the Wiki corpus dominate the in-domain corpora. Against all aggression methods and baselines from other studies, our SG-DI and CBOW-DA constantly outperform them with a large margin. Particularly, no matter whether there is enough in-domain data, SG-DI and CBOW-DA effectively integrate domain knowledge into embeddings and thus result better performance in text classification. When there are more in-domain data, CBOW-DA demonstrates a larger margin on top of other baselines, as well as SG-DI. Especially, the performance of CBOW-DA on ATIS achieve a record score 97.75, which is, to the best of our knowledge, the state-of-the-art result reported on this dataset. To compare between SG-DI and CBOW-DA, the reason leads to the difference of their performance could be that the attention mechanism works more smooth than the indicator.