An Exploration of Inference Network Architecturesfor Sequence Labeling
Table 5: NER test results (for BLSTM-CRF+) with more layers in the BLSTM inference network.
['[EMPTY]', 'F1']
While we see strong performance with infnet+, it still lags behind Viterbi in F1. We consider additional experiments in which we increase the number of layers in the inference networks. We use a 2-layer BLSTM as the inference network and also use weight annealing of the local loss hyperparameter λ, setting it to λ=e−0.01t where t is the epoch number. Without this annealing, the 2-layer inference network was difficult to train. The weight annealing was helpful for encouraging the inference network to focus more on the non-local information in the energy function rather than the token-level loss.