The Paradigm Discovery Problem
Table 4: PDP and PCFP results for all languages and models, averaged over 4 runs. Metrics are defined in § 3.3. An refers to the Analogy metric and LE to the Lexicon Expansion metric.
['[EMPTY]', 'Cells', 'Paradigms', '[BOLD] PDP F_{\\mathrm{cell}}', '[BOLD] PDP F_{\\mathrm{par}}', '[BOLD] PDP F_{\\mathrm{grid}}', '[BOLD] PCFP An', '[BOLD] PCFP LE']
For reference, we also report a supervised benchmark, sup, which assumes a gold grid as input, then solves the PCFP exactly as the benchmark does. In terms of the PDP, clustering assigns lexicon forms to paradigms (46–82%) more accurately than to cells (26–80%). Results are high for English, which has the fewest gold cells, and lower elsewhere. In German, Latin, and Russian, our benchmark proposes nearly as many cells as gold k, thus performing similarly. For English, it overestimates the true number and performs worse. For Arabic, it severely underestimates k but performs better, likely due to the orthography: without diacritics, the three case distinctions become obscured in almost all instances. In general, fixing the true number of cells can be unhelpful because syncretism and the Zipfian distribution of cells creates situations where certain gold cells are too difficult to detect. Allowing the system to choose its own number of cells lets it focus on distinctions for which there is sufficient distributional evidence.