Neural Arabic Question Answering
Table 6: Results of BERT as a document reader on ARCD-Test under different data regimes and of our open domain system SOQAL when returning the top k answers
['[BOLD] Method', '[BOLD] ARCD-Test EM', '[BOLD] ARCD-Test F1', '[BOLD] ARCD-Test SM']
Transfer Learning. To evaluate the effectiveness of using translated data as training data on the ARCD test set we train BERT under the following data regimes: (a) Arabic-SQuAD only, (b) ARCD-Train only and (c) We remark that training under regimes (a) or (b) had very similar results, this gives strong evidence that Arabic-SQuAD could be in fact sufficient for obtaining powerful MRC models. When combining both datasets, we obtain an improvement of 8.3% on the F1 score with a total score of 61.3; the training on ARCD allowed the model to better adapt to its differing answer distribution. The close F1 and SM scores indicate that the system is able to correctly retrieve the answer when it selects the correct paragraph, the issue then lies with it not scoring highly enough the correct paragraph. We also report the accuracy when the system outputs the top 3 and top 5 results (choosing the best answer out of them).