Learning Intrinsic Sparse Structures within Long Short-Term Memory
Table 4: Remaining ISS components in BiDAF by training from scratch.
['EM', 'F1', 'ModFwd1', 'ModBwd1', 'ModFwd2', 'ModBwd2', 'OutFwd', 'OutBwd', 'weight #', 'Total time*']
At last, instead of fine-tuning baseline, we train BiDAF from scratch with ISS learning. Our approach also works well when training from scratch. Overall, training from scratch balances the sparsity across all layers better than fine-tuning, which results in even better compression of model size and speedup of inference time.