Improving Sparse Word Representations with Distributional Inference for Semantic Composition
Table 2: Comparison of composition by union and composition by intersection. Not all features are displayed for space reasons.
['[BOLD] Composition by  [ITALIC] union  [BOLD] Distributional Features', '[BOLD] Composition by  [ITALIC] union  [BOLD] Co-occurrence Count', '[BOLD] Composition by  [ITALIC] intersection  [BOLD] Distributional Features', '[BOLD] Composition by  [ITALIC] intersection  [BOLD] Co-occurrence Count']
We are then in a position to compose the offset representation of white with the vector for clothes by the union or the intersection of their features. It is worth noting that any arithmetic operation can be used to combine the counts of the aligned features, however for this paper we use pointwise addition for both composition functions. One of the advantages of this approach to composition is that the inherent interpretability of count-based models naturally expands beyond the word level, allowing us to study the distributional semantics of phrases in the same space as words. Due to offsetting one of the constituents, the composition operation is not commutative and hence avoids identical representations for house boat and boat house. However, the typed nature of our vector space results in extreme sparsity, for example while the untyped VSM has 130k dimensions, our Apt model can have more than 3m dimensions. We therefore need to enrich the elementary vector representations with the distributional information of their nearest neighbours to ease the sparsity effect and infer missing information. Due to the syntactic nature of our composition operation it is not straightforward to apply common dimensionality reduction techniques such as SVD, as the type information needs to be preserved.