Bad Form: Comparing Context-Based and Form-BasedFew-Shot Learning in Distributional Semantic Models
Table 1: Results for DN and Chimera tasks. The best result per category (context-based or hybrid) in every column is marked in bold, while setups that were not evaluated have been filled with a dash. Purely form-based methods are not evaluated on the Chimera tasks. The stem-based model is not compatible with the filtered tasks.
['[EMPTY]', '[BOLD] Method', '[BOLD] Definitional Nonce MRR', '[BOLD] Definitional Nonce Median', '[BOLD] Filtered DN MRR', '[BOLD] Filtered DN Median', '[BOLD] Chimera L2', '[BOLD] Chimera L4', '[BOLD] Chimera L6', '[BOLD] Full Chimera L2', '[BOLD] Full Chimera L4', '[BOLD] Full Chimera L6']
While the Form-Context model performs significantly better than A La Carte on both tasks, the original DN task is completely dominated by the stem-based model. This shows that using known related word forms is an extremely effective approach for estimating a new embedding. The removal of these related words from the training data heavily impacts the scores for all form-based and hybrid methods, but the Form-Context model still manages to perform strongly on the Filtered DN task, showing that the model is robust to varying amounts of information in both the form and context. On the original task, there are no clear performance trends, presumably caused by the small size of the test set. In that respect, the Full Chimera task is much more useful, allowing for a better comparison between models. The additive model with a window achieves the best score on all trials for the Full Chimera task, as well as one for the original (L6, with strong performance L2 and L4 as well). The most advanced additive model (window, subsampling and negative sampling) and the nonlinear A La Carte model also perform very strongly on the Full Chimera task.