Subword-Level Language Identification for Intra-Word Code-Switching
Table 2: Number of tokens classified by language tags seen in the Spanish-Wixarika dataset. We show the total number of Tokens per tag, their proportion (%) with the total tokens, the Unique word types, and the proportion (U. %) of them with the total number of unique word types.
['Tokens', 'All', '%', 'Unique', 'Unique %']
The tag set is parallel to that of German–Turkish: ES (Spanish), WIX (Wixarika), EN (English), AMBIG (ambiguous) OTHER (punctuation, numbers, emoticons, etc), NE.ES, NE.WIX and NE.EN (named entities). Mixed words are segmented and each segment is labeled with its corresponding language (ES, WIX, EN). The percentage of mixed words is higher than in the DE–TR dataset: 3.13% of the tokens and 4.26% of the types. The most common combination is Spanish roots with Wixarika affixes. Furthermore, 16.55% of the sentences contain mixed words.author =Ozlem,color=blue!40,size=,fancyline, caption=,todo: author =Ozlem,color=blue!40,size=,fancyline ,caption=,the URL is empty at the moment! author= Manuel,color=orange!40,size=,fancyline, caption=,todo: author= Manuel,color=orange!40,size=,fancyline, caption=,Thank you! :)