BERT Based Multilingual Machine Comprehension in English and Hindi
Table 1. Important hyperparameter values for m-BERT fine-tuning and evaluation.
['Hyperparameter', 'Value']
Training phase For each training setup 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters) was fine-tuned on the entire training set of original SQuAD 2.0 dataset, containing 150K QA pairs. No fine-tuning on Hindi was involved in this setup. Hindi monolingual QA augmentation: Each zero-shot checkpoint was further fine-tuned on QH−PH variant of the training split of translated SQuAD dataset, which consists of around 10.5K QA pairs. There was a one-to-one mapping between a zero-shot checkpoint and the new checkpoint, thus maintaining the same number of checkpoints for this setup. Hindi cross-lingual QA augmentation: Each checkpoint after Hindi monolingual QA augmentation was further fine-tuned for this setup on QE−PH variant of the training split of translated SQuAD dataset. Similar to the previous setup, one-to-one mapping was maintained between the checkpoints. It is important to note that the choice of cross-lingual variant for augmentation (QE−PH or QH−PE) was completely arbitrary and any of them would have given better results as we see in the later section.