Distilling Knowledge for Search-based Structured Prediction
Table 4: The ranking performance of parsers’ output distributions evaluated in MAP on “problematic” states.
['[EMPTY]', 'optimal-yet-ambiguous', 'non-optimal']
The dynamic oracle Goldberg and Nivre This allows us to analyze the accuracy of each parser’s individual decisions, in the “problematic” states. In this paper, we evaluate the output distributions of the baseline and ensemble parser against the reference actions suggested by the dynamic oracle. Since dynamic oracle yields more than one reference actions due to ambiguities and previous mistakes and the output distribution can be treated as their scoring, we evaluate them as a ranking problem. Intuitively, when multiple reference actions exist, a good parser should push probability mass to these actions. We draw problematic states by sampling from our baseline parser. This observation indicates the ensemble’s output distribution is more “informative”, thus generalizes well on problematic states and achieves better performance. We also observe that the distillation model perform better than both the baseline and ensemble. We attribute this to the fact that the distillation model is learned from exploration.