Evaluating Neural Word Representations inTensor-Based Compositional Settings
Table 5: Results for paraphrase detection (MSR) and dialog act tagging (SWDA) tasks. All top results significantly outperform corresponding nearest competitors (for accuracy): p<0.05, χ2 test.
['[EMPTY]', '[BOLD] Baseline', '[BOLD] Baseline', '[BOLD] Co-occurrence  [BOLD] GS11', '[BOLD] Co-occurrence  [BOLD] GS11', '[BOLD] Co-occurrence  [BOLD] KS14', '[BOLD] Co-occurrence  [BOLD] KS14', '[BOLD] Neural word embeddings  [BOLD] Unlemmatized', '[BOLD] Neural word embeddings  [BOLD] Unlemmatized', '[BOLD] Neural word embeddings  [BOLD] Lemmatized', '[BOLD] Neural word embeddings  [BOLD] Lemmatized']
For each vector space and model, a number of different thresholds were tested on the first 2000 pairs of the training set, which we used as a development set; in each case, the best-performed threshold was selected for a single run of our “classifier” on the test set (1726 pairs). Additionally, we evaluate the NWE model with a lemmatized version of the corpus, so that the experimental setup is maximally similar for all vector spaces. Disfluency markers and commas are removed from the utterance raw texts. For GS11 and KS14 the utterance tokens are POS-tagged and lemmatized; for NWE, we test the vectors in both a lemmatized and an un-lemmatized version of the corpus. We split the training and testing utterances as suggested by \newciteStolcke.etal00. Utterance vectors are then obtained as in the previous experiments; they are reduced to 50 dimensions using SVD and a k-nearest-neighbour classifier is trained on these reduced utterance vectors (the 5 closest neighbours by Euclidean distance are retrieved to make a classification decision).