Unmasking Bias in News
Table 3: Results of the proposed masking technique (k=500 and n=5) applied to mask topic-related information or style-related information. NB: Naive Bayes; RF: Random Forest; SVM: Support Vector Machine. The last two rows show the results obtained by applying the system from [5] to our cleaned dataset (Section 3).
['[BOLD] Masking  [BOLD] Method', '[BOLD] Classifier', '[BOLD] Macro F1', '[BOLD] Accuracy', '[BOLD] Precision left', '[BOLD] Precision right', '[BOLD] Precision main', '[BOLD] Recall left', '[BOLD] Recall right', '[BOLD] Recall main', '[BOLD] F1 left', '[BOLD] F1 right', '[BOLD] F1 main']
We also include the macro F1 score because of the unbalance test set. For these experiments we extract the character 5-grams from the transformed texts, taking into account that as more narrow is the domain more sense has the use of longer n-grams. When k>5000, we clearly can see that the topic-related model, in which the k most frequent terms are masked, is decreasing the performance. This could be explained by the fact that relevant topic-related terms start to be masked too. However, a different behavior is seen in the style-related model, in which we tried to maintain only the style-related words without masking them. In this model, the higher is k the better is the performance. This confirms that for the used dataset, taking into account only style-related information is not good, and observing also topic-related information benefits the classification.