Multilingual Language Processing From Bytes
Table 4: Macro-averaged (across 4 languages) F1 for the NER task using different model architectures.
['[BOLD] Depth', '[BOLD] Width=320', '[BOLD] Width=640']
This observation holds for both models trained on a single language and models trained on many languages. Because the vocabulary is so small, the total number of parameters is dominated by the size of the recurrent matrices. All the results reported below use the same architecture (unless otherwise noted) and thus have roughly 900k parameters. Increasing the width of the model improves performance less than increasing the depth, and once we use 4 stacked LSTMs, the added benefit of a much wider model has disappeared. This result suggests that rather than learning to partition the space of inputs according to the source language, the model is learning some lanugage-independent representation at the deeper levels.