Training with Streaming Annotation
Table 3: Performance (%) comparison with different strategies and noise level. Names of strategies (All, Current, Finetune, Proposed) and definition of noise level are introduced in Section 3.2. Note that there is no “previous” batch for the first batch, the results in the first batch are identical.
['Noise', 'Slice Metric', '200303 P', '200303 R', '200303 F1', '200304 P', '200304 R', '200304 F1', '200305 P', '200305 R', '200305 F1', '200306 P', '200306 R', '200306 F1']
we show the performance of the strategies on two noise levels. It is expected that the performance of the Current strategy is the lowest among all strategies because the model does not have enough training data and easily over-fits; hence we observe a drop in the batch of 200305 even this batch has a higher quality of annotations. The Finetune and All strategies perform better, but the noise still influences the performance. The Finetune strategy does not sufficiently alleviate the influence from wrong annotations in the early batches. Our method outperforms all other strategies because the memory embeddings successfully enhance the information with same event occurrence in the sentences in the past batches. Moreover, the improvement is more significant if the noise level is higher.