Adapting Sequence to Sequence models for Text Normalization in Social Media
Table 4: Comparison of our HS2S model with additional baselines.
['[BOLD] Model name', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] F1', '[BOLD] Method highlights']
Dict2 shows that handling ambiguous cases inappropriately results in a dramatic drop in precision. It is therefore necessary for a text normalization model to be able to correctly normalize text in the occurrence of multiple mappings. S2SMulti is a baseline method that firstly normalizes terms that have a unique mapping, based on source and target tokens found in the training data, and later on utilizes a word-level Seq2Seq model that is trained to correct one-to-many mappings (an unnormalized token that can be transformed into several standard words, e.g. “ur” → {“you are”, “your”}). We can see that this method has a better performance, which validates our hypothesis that a naive word replacement would not suffice, however the end-to-end Seq2Seq model (S2S) performs better than S2SMulti, i.e. Seq2Seq models can handle both unique and multiple mappings seamlessly without any additional feature engineering effort apart from tokenization.