Question Dependent Recurrent Entity Network for Question Answering
Table 3: On the left, an example from CNN news article, and on the right, the model selection Hyper-parameters.
['[BOLD] Parameter', '[BOLD] Values']
The CNN dataset is already tokenized and cleaned, therefore we did not apply any text pre-processing. As it was done in other models, the set of possible answers is restricted to the set of hidden entities in the text, that are much less, around 500, compared to all the words (120K) in the vocabulary. Compared to the model used for bAbI, we changed the activation function of the output layer, using a sigmoid instead of parametric ReLU, since after several experiments we noticed that such activation was hurting the model performance. Moreover, the input was not split into sentences, thus we divided the text into sentences using the dot token (”.”). sentence splitting in general is itself a challenging task, but in this case the input was already cleaned and normalised. However, the sentence may be very long, thus we intrdocued a windowing mechanism. The same approach has been used in the End To End Memory Network [sukhbaatar2015end] as a way to encode the input sentence. This method takes each entity marker (@entityi) and it creates a window of b words around it. Formally, {wi−(b−1)2,…,wi,…,wi+(b−1)2}, where wi represent the entity of interest. For the question, a single window is created around the placeholder marker (the word to predict). Moreover, we add 2(b−1) tokens for the entities at the beginning and at the end of the text. To check whether our QDREN could improve the existent REN and whether the window-based approach makes any difference in comparison with plain sentences, we separately trained four different models: REN+SENT, REN+WIND, QDREN+SENT and QDREN+WIND. Where SENT represent simple input sentences, and WIND the window as a input. For each of this model, we conduct a separated model selection using a various number of hyper-parameters.