Cross-genre Document Retrieval:Matching between Conversational and Formal Writings
Table 4: Evaluation on the development and evaluation sets for summary, plot, and all (summary + plot). Elastic10: Elasticsearch with k = 10, Structw,l,m: structure matching using words, lemmas, embeddings, Rerank1,λ: unweighted and weighted reranking.
['[BOLD] Model', '[BOLD] Development  [BOLD] Summary', '[BOLD] Development  [BOLD] Summary', '[BOLD] Development  [BOLD] Plot', '[BOLD] Development  [BOLD] Plot', '[BOLD] Development  [BOLD] All', '[BOLD] Development  [BOLD] All', '[BOLD] Evaluation  [BOLD] Summary', '[BOLD] Evaluation  [BOLD] Summary', '[BOLD] Evaluation  [BOLD] Plot', '[BOLD] Evaluation  [BOLD] Plot', '[BOLD] Evaluation  [BOLD] All', '[BOLD] Evaluation  [BOLD] All']
The highest P@1 of 39.53% is achieved on the evaluation set using lemmas. Although it is about 8% lower than the one achieved by Elasticsearch, we hypothesize that this approach can correctly retrieve documents for certain queries that Elasticsearch cannot. The scores from Elastic10 and Struct∗ for each document are fed into the binary classifier that decides whether or not to accept the top-1 result from Elasticsearch. The Rerankλ row shows the results when the optimized weights are used, which gives an additional 3% boost on the development set but not on the evaluation set.