BERT-Based Arabic Social Media Author Profiling
Table 1: Tweet level results on DEV
['[EMPTY]', '[BOLD] Age', '[BOLD] Dialect', '[BOLD] Gender']
Our baseline is a GRU network for each of the three tasks. We use the same network architecture across the 3 tasks. For each network, the network contains a layer unidirectional GRU, with 500 units and an output linear layer. The network is trained end-to-end. Our input embedding layer is initialized with a standard normal distribution, with μ=0, and σ=1, i.e., W∼N(0,1). We use a maximum sequence length of 50 tokens, and choose an arbitrary vocabulary size of 100,000 types, where we use the 100,000 most frequent words in TRAIN. To avoid over-fitting, we use dropout [srivastava2014dropout] with a rate of 0.5 on the hidden layer. For the training, we use the Adam [kingma2014adam] optimizer with a fixed learning rate of 1e−3. We employ batch training with a batch size of 32 for this model. We train the network for 15 epochs and save the model at the end of each epoch, choosing the model that performs highest accuracy on DEV as our best model. We report all our results using accuracy. Our best model obtains 42.48% for age, 37.50% for dialect, and 57.81% for gender. All models obtain best results with 2 epochs.