Gradient-based Inference for Networks with Output Constraints
Table 5: Evaluation of syntactic parser and SRL system on out-of-domain data. F1 scores are reported on the failure set. SRL model was trained on NW and the syntactic parser was trained on WSJ Section on OntoNote v5.0. Except PT, which is new and old Testament, all failure rate on out-domain data is higher than that of in-domain (11.9% for parsing and 18.1% for SRL) as suspected. The table shows that GBI can be successfully applied to resolve performance degradation on out-of-domain data.
['Genre', '[ITALIC]  [BOLD] Syntactic Parsing  Failure', '[ITALIC]  [BOLD] Syntactic Parsing  Conversion', '[ITALIC]  [BOLD] Syntactic Parsing  F1 on failure set', '[ITALIC]  [BOLD] Syntactic Parsing  F1 on failure set', '[ITALIC]  [BOLD] SRL Failure', '[ITALIC]  [BOLD] SRL Conversion', '[ITALIC]  [BOLD] SRL F1 on failure set', '[ITALIC]  [BOLD] SRL F1 on failure set']
Previously, we saw how GBI performs well even when the underlying network is of lower quality. We now investigate GBI on actual out-of-domain data for which the model quality can suffer. For SRL, we train a SOTA network with ELMo embedding on the NewsWire (NW) section of the OntoNotes v5.0 English PropBank corpus and then test on the other genres provided in the corpus: BC, BN, PT, TC, WB. The failure rate on the within genre data (test set of NW) is 18.10%. Further, by enforcing constraints, we see significant gains on the failure set in terms of F1 score across all genres (ranging from 9.39-16.5 F1), thus, providing additional evidences for answering Q5. As we did for SRL, we train a GMNT seq2seq model on the WSJ NW section in OntoNotes v5.0 The F1 on the within-genre data (test set of WSJ) is 85.03, but the F1 on these genres is much lower, ranging from the mid-forties on BC (46.2–78.5 depending on the subcategory) to the low-eighties on BN (68.3–81.3. depending on the subcategory). Indeed, we find that overall the F1 is lower and in some cases, like WB, the failure rate is much higher (17.6% for WB vs. 11.9% for WSJ). We see that across all genres, the algorithm has high conversion rates (sometimes close to 100%), and that in each case, enforcing the constraints improves the F1. Again, we find support for Q2, Q3 and Q5.