Bilingual Document Alignment with Latent Semantic Indexing
Table 2: Distribution of missed pairs over domains with a soft similarity threshold of .95. Domains with a single miss are aggregated under “other”.
['[BOLD] domain', '[BOLD] missed pairs']
As we can see, errors are concentrated in only a few of the 203 domains in the test set. We will briefly discuss the top five here. The errors in www.lagardere.com originate from mixed-language pages, typically pages with the boilerplate text for the user interface in one language and the actual content in the other. The missed pairs in meatballwiki.org can be attributed to red herrings: URL pairs that erroneously suggest a correspondence between the two pairs in question. The web site www.toucherdubois.ca provides teaching resources (including images and lesson plans) for teaching students about “the sociocultural heritage of the people of Madawaska” in Canada and the US. Some of the pages consist of little text wrapped around image resources; lesson plans are often very similar in terms of the vocabulary used, thus confusing the LSA model. The missing pairs from www.rfimusique.com and www.taize.fr are pairs of pages with a low payload-to-boilerplate (or near-boilerplate) ratio , i.e., they are dominated by text that can be found on multiple pages, thus leading to document alignment errors.