Improving Term Frequency Normalization for Multi-topical Documents, and Application to Language Modeling Approaches
Table 2: Performances of Dir, DirV and JMV2 (MAP, Pr@5, Pr@10). Bold faced numbers indicate runs showing significant improvement over Dir.
['[BOLD] MAP', 'Dir SK', 'Dir SV', 'Dir LV', 'DirV SK', 'DirV SV', 'DirV LV', 'JMV2 SK', 'JMV2 SV', 'JMV2 LV']
As for topic measurement τ(D), we selected the information quantity (ε(D)) since JMV2 and DirV using the information quantity is better than those using vocabulary size. We used MLE (Maximum Likelihood Estimation) for P(w|θD) to calculate the information quantity without any smoothing. We selected Dir as the baseline due to its superiority over JM in all test collections. To obtain the best performance of each run, we searched 20 different values between 0.01 and 0.99 for λ, and 22 values between 100 and 30,000 for μ. To check whether or not the proposed method (DirV and JMV2) significantly improves the baseline, we performed the Wilcoxon sign ranked test to examine at 95% and 99% confidence levels. We attached † and ‡ to the performance number of each cell in the table when the test passes at 95% and 99% confidence level, respectively. The results are summarized as follows: DirV significantly improves MAP of Dir for verbose type of query (SV and LV). Exceptionally, TREC10 did not show an improvement for verbose type of query. DirV does not significantly improve MAP of Dir for keyword type of query (SK), but improves precisions (Pr@5 or Pr@10). Especially, on TREC7 and TREC8, Pr@10 is significantly improved over Dir. Although other test collections do not statistically show a significant improvement, there is large portion of the numerical increase. DirV or JMV2 show improvement on a specific test collection even for keyword type of query. For DirV, TREC10 is such a collection by showing a significant improvement of MAP. For JMV2, WT2G is such a test collection. Overall, DirV is slightly better than JMV2 in most of test collections. WT2G is an exceptional collection to show that JMV2 significantly improves DirV.