ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Them
Table 9: Selected shallow features one may hypothesize impact evaluation scores negatively (SI).
['[BOLD] Feature', '[EMPTY]', '[BOLD] Count', '[BOLD] P-value']
Further investigation of problematic cases in a paradigm of no-box debugging with GEval tool [gralinski-etal-2019-geval] revealed the most worsening features, that is features It seems that our system tend to return ranges without adjacent punctuation. This is the case of sentences such as The new CIA Director Haspel, who ‘tortured some folks,’ probably can’t travel to the EU, where only the quoted text was returned whereas annotation assumes it should be returned with apostrophes and comma. This remark can be used to slightly improve overall results with simple post-processing. Returned and conjunction refers to the cases where it connects two propaganda spans. The system frequently returns them as single span, contrary to what is expected in the gold standard. One of the frequent predictors of low accuracy is a comma character present within the span to be classified. It can be probably attributed to the fact that its presence is a good indicator of span linguistic complexity. Another determinant of inefficiency turned out to be a negation—around a half of the sentences containing word not were misclassified by the system. Suggested features of a quotation mark before the span and the digram according to after the span are related to reported or indirect speech. Explanation of worsening effect of other features is not as evident as in the case of mentioned above. Moreover, it seems there is no obvious way of improving the final results with our findings and a more detailed analysis might be required.