Emergent Predication Structurein Hidden State Vectors of Neural Readers
Table 2: Accuracy on Who-did-What dataset. Each result is based on a single model. Results for neural readers other than NSE are based on replications of those systems. All models were trained on the relaxed training set which uniformly yields better performance than the restricted training set. The first group of models are explicit reference models and the second group are aggregation models. + indicates anonymization with better reference identifier.
['Who did What', 'Validation', 'Test']
Aggregation readers can have difficulty when anonymization is not done. The Stanford Reader achieves just better than 45% on the Who-did- What dataset while the Attention-Sum Reader can get near 60% But if we anonymize the Who-did-What dataset and then re-train the Stanford Reader, the accuracy jumps to near 65%. Anonymization greatly reduces the number of output word embeddings eo(a) to be learned. We need to learn only output embeddings for the relatively small number of entity identifiers needed for the question. Anonymization suppresses the semantics of the reference phrases and leaves only a semantics-free entity identifier. This suppression of semantics may facilitate the separation of the hidden state vector space H into a direct sum S⊕E with s(Φ)∈S and eo(a),s(a)∈E. A third, and perhaps more important effect of anonymization is to provide reference information. Anonymization explicitly marks positions of candidate answers and establishes coreference. A natural question is whether this information can be provided without anonymization by simply adding additional coreference features to the input. Here we evaluate two architectures inspired by this question. This evaluation is done on the Who-did-What dataset which is not anonymized. In each architecture we add features to the input to mark the occurrences of candidate answers. These models are simpler than the Stanford Reader but perform comparably. If an answer choice exists at position t in the passage let it be the index of that choice on the answer choice list. If no answer choice occurs at position t we let it be zero. We define e′(i) to be the zero vector if i=0 and otherwise to be the one-hot vector for i (i.e., the five-dimensional vector with zeroes at all positions except with a one at position i). We define “pointer annotation” to be the result of concatenating e′(it) as additional features to the word embedding e(wt) for token wt in the passage: ¯e(wt)=[e(wt),e′(it)] (17) We feed the new ¯e(wt) to the readers for each token wt. shows results using this reader, showing performance comparable to the Stanford Reader with anonymization. -What dataset.