Support-BERT: Predicting Quality of Question-Answer Pairs in MSDN using Deep Bidirectional Transformer
TABLE II: Classification results with BERT pre-training+finetuning
['Training', 'Test', 'Accuracy', 'Precision', 'Recall', 'Specificity', 'F-1 Score']
Starting from the checkpoint of BERT model trained on Wikipedia in an unsupervised setting, support-BERT was pre-trained on the MSDN community support data for 10 epochs. Note that during pre-training, next sentence prediction involves using sentences defined by words between two full stops. The pre-training for 10 epochs took 48 hours in our machine. Finetuning the model using Q&As drastically improved the performance. In this case, next sentence prediction model involves using questions as a paragraph (involving more than one actual sentences) as first sentence and answers as a paragraph (involving more than one actual sentences) as next sentence. For test set containing acceptance vs. unaccepted ratio as 1:1, the model was able to identify quality Q&As 82% of the time, whereas for test set containing acceptance vs. unaccepted ratio as 1:3, the accuracy value was 0.7741.