Optimizing Component Combination in a Multi-Indexing Paragraph Retrieval System
Table 1: Evaluation of our system, with and without weight tuning, and comparison with the QuestionCube system from [Molino et al.2012].
['[BOLD] System', '[BOLD] MRR']
Our metric of choice is the Mean Reciprocal Rank (MRR), which gives a score of 1/r for each question, where r is the position of the paragraph containing the right answer in the results list. For the weight tuning experiment, we used 20 rounds of cross-validation to avoid over-fitting. In each round, the tuning was done on 190 questions, and then evaluated on the remaining 10. The result shown in the table is the average of those 20 MRR scores.