Image-to-Markup Generation with Coarse-to-Fine Attention
Table 3: Image-to-LaTeX ablation experiments. Compares simple LM approaches and versions of the full model on train and test perplexity, and image match accuracy.
['Model', 'Ablation', 'Train', 'Test', 'Match']
The simplest model is a basic (non-conditional) NGram LM on LaTeX which achieves a perplexity of around 8. Simply switching to an LSTM-LM reduces the value to 5, likely due to its ability to count parentheses and nesting-levels. These values are quite low, indicating strong regularity just in the LaTeX alone. Adding back the image data with a CNN further reduces the perplexity down to 1.18. Adding the encoder LSTM adds a small gain to 1.12, but makes a large difference in final accuracy. Adding the positional embeddings (trainable initial states for each row) provides a tiny gain. Hard attention leads to a small increase in perplexity. We also consider the effect of training data on performance. As with many neural systems, the model is quite data hungry. In order for the model to reach ≥50% accuracy, at least 16k training examples are needed.