An assessment of orthographic similarity measures for several African languages
Table 5: Tokens with the -fund- root in the Isolezwe articles of August 26 and 27, 2015 (part of the NIC).
['[BOLD] Token', '[ITALIC] n', '[BOLD] Translation']
The lexical diversity is very high for the OOspell file, which is largely due to having combined different word lists, rather than texts, and as such should be disregarded for comparison. Overall, the lexical diversity is high, compared to, say, English. An important reason for this large difference is due to the simplicity of the measure. For instance, amaphoyisa (‘police’, plural) and namaphoyisa (‘and the police’, plural) in the NIC are counted as two different words, but semantically refer to one concept; amaphoyisa occurs only 27 times, yet its root -phoyis- occurs 47 times in just the first news item set (2657 tokens) of the NIC. There are many more such cases in the corpora, such as -mali ‘money’, also in the NIC: imali, kwemali, yimali, onemali, osozimali, kwezimali, ngezimali, which are, respectively of -, and -, that/which/ who has -, of - (pl.), about/by/with/per - (pl.) money. In the simple type-to-token measure, they are all counted as different types. This does not explain the substantial difference in lexical diversity between the UDHR and SAC, however, which are of the same genre. The UDHR has a high lexical diversity due to it being a small document. For instance, the subset of the NIC with Isoleszwe news articles of only August 7, 2015, has a similar lexical diversity of 0.55 on its 2667 tokens. So, this is not unusual. abafundi ‘students’ is the base noun, and bafundi, nabafundi, kubafundi, kwabafundi, wabafundi, and yabafundi are the modified nouns, thus standing in a ratio of 1:6 as types and 10:9 as tokens, and in the singular as umfundi with kunomfundi and mfundi as 1:2 and 3:1, respectively. This could be done likewise for all words, and taking the median over it to obtain λθ and λt, respectively. If there were only these two, then λθ would be 0.50 and λt 3. Filling this into the equation results in a calibrated lexical diversity of TTcal=0.50∗21050.67∗3774=0.42 (5) compared to the original 21053774=0.56.