Cross-lingual Document Retrieval using Regularized Wasserstein Distance
(a)
['[EMPTY]', 'WikiEn-Fr', 'WikiEn-Ge', 'WikiEn-Fi', 'WikiEn-Gr']
(ℓ1−ℓ2) Wikipedia datasets: (i) English-French, (ii) English-German, (iii) English-Greek and (iv) English-Finnish. Each dataset defines two retrieval problems: for the first (ℓ1→ℓ2) the documents of ℓ1 are retrieved given queries in ℓ2; for the second (ℓ2→ℓ1) the documents of ℓ1 are the queries. To construct the Wikipedia excerpts, we use the comparable Wikipedia corpora of linguatools. Our datasets comprise 10K pairs of comparable documents; the first 500 are used for evaluating the retrieval approaches. The use of the remaining 9.5K, dubbed “BiLDA Training” is described in the next paragraph. In the pre-processing steps we lowercase the documents, we remove the stopwords, the punctuation symbols and the numbers. We keep documents with more than five words and, for efficiency, we keep for each document the first 500 words.