Neural Ranking Models with Weak Supervision
Table 5. Performance of the rankprob model with embedding vector representation in fully supervised setting, weak supervised setting, and weak supervised plus supervision as fine tuning. indicates that the improvements over all other models are statistically significant, at the 0.05 level using the paired two-tailed t-test, with Bonferroni correction.
['[BOLD] Method', '[BOLD] Robust04  [ITALIC] MAP', '[BOLD] Robust04  [ITALIC] P@20', '[BOLD] Robust04  [ITALIC] nDCG@20', '[BOLD] ClueWeb  [ITALIC] MAP', '[BOLD] ClueWeb  [ITALIC] P@20', '[BOLD] ClueWeb  [ITALIC] nDCG@20']
How useful is learning with weak supervision for supervised ranking? In this set of experiments, we investigate whether employing weak supervision as a pre-training step helps to improve the performance of supervised ranking, when a small amount of training data is available. In all the supervised scenarios, we performed 5-fold cross-validation over the queries of each collection and in each step, we used the TREC relevance judgements of the training set as supervised signal. For each query with m relevant documents, we also randomly sampled m non-relevant documents as negative instances. Binary labels are used in the experiments: 1 for relevant documents and 0 for non-relevant ones. The reason for the poor performance of the supervised model compared to the conventional learning to rank models is that the number of parameters are much larger, hence it needs much more data for training.