Repurposing Entailment for Multi-Hop Question Answering Tasks
Table 6: F1a precision and recall on MultiRC Dev with 2 kinds of relevance losses. IR Sum is the sum of attention probability mass on irrelevant sentences. BCE is Binary Cross Entropy loss.
['[EMPTY]', 'F1a precision', 'F1a recall']
Relevance Loss. The sentence-level relevance model provides a way to dig deeper into the overall QA model’s behavior. When sentence-level supervision is available, as in the case of MultiRC, we can analyze the impact of different auxiliary losses for the relevance module. Overall, we find that two types of behaviors emerge from different loss functions. For instance, trying to minimize the sum of attention probability mass on irrelevant sentences i.e. ∑iαi(1−yi), called IR Sum Loss, causes the attention scores to become ”peaky” i.e, high for one or two sentences, and close to zero for others. This leads to higher precision but at significantly lower recall for the QA system, as it now uses information from fewer but highly relevant sentences. Binary cross entropy loss (BCE) allows the model to attend to more relevant sentences thereby increasing recall without too much drop in precision.