SubjQA: A Dataset for Subjectivity and Review Comprehension
Table 9: MTL gains/losses over the fine-tuning condition (F1 and Exact match), across subj./fact. QA.
['[EMPTY]', '[BOLD] Fact. A  [BOLD] F1', '[BOLD] Fact. A  [BOLD] E', '[BOLD] Subj. A  [BOLD] F1', '[BOLD] Subj. A  [BOLD] E', '[BOLD] Fact. Q  [BOLD] F1', '[BOLD] Fact. Q  [BOLD] E', '[BOLD] Subj. Q  [BOLD] F1', '[BOLD] Subj. Q  [BOLD] E', '[BOLD] Overall  [BOLD] F1', '[BOLD] Overall  [BOLD] E']
After fine-tuning over each domain in the MTL setting, the subjectivity-aware model achieves an average F1 of 76.3% across the different domains, with a minimum of 58.8% and a maximum of 82.0% on any given domain. Under both the F1 and the Exact match metrics, incorporating subjectivity in the model as an auxiliary task boosts performance across all domains. Although there are gains also for subjective questions and answers, it is noteworthy that the highest gains can be found for factual questions and answers. This can be explained by the fact that existing techniques already are tuned for factual questions. Our MTL extension helps in identifying factual questions, which further improves the results. However, even if subjective questions are identified, the system is still not tuned to adequately deal with this input.