A Discrete Hard EM Approach for Weakly Supervised Question Answering
Figure 3: Varying the size of solution set (|Z|) at training. (Left) Subsets of the train set on WikiSQL varying in the size of solution set (|Z|). All subsets contain 10k training examples (total in the original train set is 55k). All subsets are evaluated on the same, original development set for a fair comparison. (Right) Performance across subsets of the training set with varying |Z|. Our method achieves substantial gains over MML.
['Group', 'Avg | [ITALIC] Z|', 'Median | [ITALIC] Z|', '# train']
To see how our learning method works with respect to the size of a solution set (|Z|) of the training data, particularly with large |Z|, we take 5 subsets of the training set on WikiSQL with |Z|=3,10,30,100,300. We train a model with those subsets and evaluate it on the original development set, both with our training method and MML objective. We observe that (i) our learning method outperforms MML consistently over different values of |Z|, and (ii) the gain is particularly large when |Z|>3.