Variational Question-Answer Pair Generation for Machine Reading Comprehension
Table 2: QA pair modeling capacity measured on the test set. NLL: negative log likelihood (-\log p(q,a|c)). {\rm NLL}_{a}=-\log p(a|c), {\rm NLL}_{q}=-\log p(q|a,c). D_{{\rm KL}_{z}} and D_{{\rm KL}_{y}} are Kullback–Leibler divergence between the approximate posterior and the prior of the latent variable z and y. The lower NLL is, the higher the probability is that the model assigns to the test set. NLL for our models are estimated with importance sampling using 300 samples.
['[EMPTY]', 'NLL', '{\\rm NLL}_{a}', '{\\rm NLL}_{q}', 'D_{{\\rm KL}_{z}}', 'D_{{\\rm KL}_{y}}']
We also observe that this issue happens when implementing our model according to the Ineq. To mitigate this problem, inspired by Prokhorv19, we use modified \beta-VAE beta-vae proposed by Burgess18, which uses two hyperparameters to control the KL terms. Our modified variational lower bound is as follows: \displaystyle\log \displaystyle\ p_{\theta}(q,a|c)\geq\mathbb{E}_{z,y\sim q_{\phi}(z,y|q,a,c)}[% \log p_{\theta}(q|y,a,c) \displaystyle+\log p_{\theta}(a|z,c)] \displaystyle-\beta|D_{\rm KL}(q_{\phi}(z|a,c)||p_{\theta}(z|c))-C| \displaystyle-\beta|D_{\rm KL}(q_{\phi}(y|q,c)||p_{\theta}(y|c))-C|, (3) where \beta>0 and C\geq 0. We use the same \beta and C for the two KL terms for simplicity. First, our models with {\rm C=0} are superior to the pipeline model, which means that introducing latent random variables aid QA pair modeling capacity. However, the KL terms converge to zero with {\rm C=0}. In other tasks, it is shown that our model with {\rm C=0} collapses into a deterministic model. The fact that {\rm NLL_{a}} is consistently lower than {\rm NLL_{q}} is due to the decomposition of probability p(a|c)=p(c_{end}|c_{start},c)p(c_{start}|c) and p(q|a,c)=\prod_{i}p(q_{i}|q_{1: i-1},a,c), which is sensitive to the sequence length. Also, we observe that the hyperparameter C can control the KL values, showing the potential to avoid the posterior collapse issue in our case. When we set C>0, KL values are greater than 0, which implies that latent variables have non-trivial information about questions and answers.