A Large-Scale Multilingual Disambiguation of Glosses
Table 4: Disambiguation precision (Prec.) and coverage (Cov.) percentage (%) of the three different disambiguation strategies on the 300 sample definitions.
['[EMPTY]', '[BOLD] English  [BOLD] Prec.', '[BOLD] English  [BOLD] Cov.', '[BOLD] Spanish  [BOLD] Prec.', '[BOLD] Spanish  [BOLD] Cov.', '[BOLD] Italian  [BOLD] Prec.', '[BOLD] Italian  [BOLD] Cov.']
We first carry out an intrinsic evaluation of the resource, by manually assessing the quality of disambiguation on some randomly extracted samples of definitions. We rely on three human judges and evaluate samples of 100 items for three languages. We evaluated the disambiguation output before and after the refinement step, and compared against a baseline where each definition is disambiguated in isolation with Babelfy. Although the disambiguation of context-free definitions improves only slightly with respect to the disambiguation of definitions in isolation, this improvement is consistent across languages. Furthermore, our system significantly increases the precision after the refinement step. Refinement reduces the coverage by 35% for English, and by 43% for Spanish and Italian, but increases precision by almost 11% for English, 20% for Spanish and 13% for Italian.