Image-to-Markup Generation with Coarse-to-Fine Attention
Table 4: Average number of coarse (#C) and fine (#F) attention computations for all models throughout the test set. standard and hierarchical provide an upper-bound and coarse-only a lower-board, whereas hard always does the minimal 4×4=16 fine lookups. Test accuracy is shown for ease of comparison.
['Model', 'Attn', '# C', '# F', 'Match']
We next compare the different hierarchical and coarse-to-fine extensions to the system. We first note that the use of the coarse-only system leads to a large drop in accuracy, indicating that fine attention is crucial to performance. On the other hand, the high performance of hierarchical indicates that two layers of soft-attention do not hurt the performance of the model. Both the hard REINFORCE system and sparsemax reduce lookups at a small cost in accuracy. Hard is the most aggressive, selecting a single coarse cell. Sparsemax achieves higher accuracy, at the cost of selecting multiple coarse cells. Depending on the application, these are both reasonable alternatives to reduce the number of lookups in standard attention.