Measuring Thematic Fit with Distributional Feature Overlap
Table 3: Results for Padó, McRae and Ferretti, Instruments and Locations, with WO computed on PPMI matrix, varying the number of fillers (i.e. 10, 30 and 50) and the types of dependency contexts (i.e. ALL, SO and PREP). The best results of our system are in bold. A baseline reimplementing Baroni and Lenci (2010) – with 10, 30 and 50 fillers – and state of the art results from previous literature are reported for comparison.
['[BOLD] Weight', '[BOLD] N', '[BOLD] # Fillers', '[BOLD] Padó  [BOLD] ALL', '[BOLD] Padó  [BOLD] SO', '[BOLD] Padó  [BOLD] PREP', '[BOLD] Mcrae  [BOLD] ALL', '[BOLD] Mcrae  [BOLD] SO', '[BOLD] Mcrae  [BOLD] PREP', '[BOLD] Ferretti - Instruments  [BOLD] ALL', '[BOLD] Ferretti - Instruments  [BOLD] SO', '[BOLD] Ferretti - Instruments  [BOLD] PREP', '[BOLD] Ferretti - Locations  [BOLD] ALL', '[BOLD] Ferretti - Locations  [BOLD] SO', '[BOLD] Ferretti - Locations  [BOLD] PREP']
Settings. To prove our hypotheses and verify the consistency of the system, we tested a large range of settings, varying: the number of fillers used to build the prototype, with the most typical values in the literature ranging between 10 and 50. We report the results for 10, 30 and 50 fillers the types of the dependency relations used for calculating the overlap: we report results for the SO, PREP and ALL settings; the value of N, that is the number of top contexts that we take into account when computing the weighted overlap. Baseline and State of the Art. In the first three rows, the table shows the scores obtained by our system varying the types of dependency contexts (i.e. ALL, SO, PREP) and the number of fillers considered for the prototype (i.e. 10, 30 and 50). The other rows respectively show i) the scores obtained by calculating the vector cosine between the role prototype vector (i.e. the vector obtained by summing the most typical fillers, with no salience ranking of the dimensions) and the candidate filler vector and ii) the scores reported in the literature for the best unsupervised and supervised models. At the best of our knowledge, the result for the instruments is the best reported until now in the literature. This is particularly interesting because – as pointed out by Sayeed and Demberg I ate a pizza with Mark). Unlike all the other datasets, instrument prototypes built with more fillers tend to be more noisy and therefore to pull down both the vector cosine and WO performance Systems based on semantic role labeling have an advantage in this sense, as they do not have to deal with prepositional ambiguity. As briefly mentioned above, in our experiments, we tested both PPMI and PLMI as weighting measures.