Towards Detection of Subjective Bias using Contextualized Word Embeddings
Table 1. Experimental Results for the Subjectivity Detection Task
['[EMPTY]', '[BOLD] Models/Metrics', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] F1', '[BOLD] Acc']
Optimized BERT-based models: We use BERT-based models optimized as in (2020roberta) and (2020albert), pretrained on a dataset as large as twelve times as compared to BERTlarge, with bigger batches, and longer sequences. ALBERT, introduced in (2020albert), uses factorized embedding parameterization and cross-layer parameter sharing for parameter reduction. These optimizations have led both the models to outperform BERTlarge in various benchmarking tests, like GLUE for text classification and SQuAD for Question Answering. Distilled BERT-based models: Secondly, we propose to use distilled BERT-based models, as introduced in (DistilBert). They are smaller general-purpose language representation model, pre-trained by leveraging distillation knowledge. This results in significantly smaller and faster models with performance comparable to their undistilled versions. We finetune these pretrained distilled models on the training corpus to efficiently detect subjectivity. BERT-based ensemble models : Lastly, we use the weighted-average ensembling technique to exploit the predictions made by different variations of the above models. Ensembling methodology entails engendering a predictive model by utilizing predictions from multiple models in order to improve Accuracy and F1, decrease variance, and bias. Our proposed methodology, the use of finetuned optimized BERT based models, and BERT-based ensemble models outperform the baselines for all the metrics.