Stochastic model for the vocabulary growth in natural languages
Table 1: Parameters b∗, γ∗, and C∗=C(γ∗,b∗) obtained from ML-fit of Eq. (1) obtained for the largest database Y(2000) for all considered languages.
['language', '[ITALIC] b∗', '[ITALIC] γ∗', '[ITALIC] C∗= [ITALIC] C( [ITALIC] γ∗, [ITALIC] b∗)']
We now discuss in detail the best two-parameter model we identify from our data: Fdp(r;γ,b)=C{r−1,r≤bbγ−1r−γr>b, (1) characterizing a double power-law (dp), The effect of the threshold n applied to the frequency of words is that, in practice, data of F(r) is limited to F(r)≥n/M (M is the observed number of words). The original Zipf’s law is recovered for high-frequency words and a critical rank r=b determines a transition to a power-law with exponent γ. Double power-laws were proposed as a generalization of Zipf’s law in Ref. S. Naranan, V. Balasubrahmanyan R. Ferrer i Cancho, and R.V. Solé These insightful works used distributions with two power-law exponents γ1,γ2 and were motivated by the visual inspection of double logarithmic plots. Our improved statistical analysis confirm and extend these observations for the simpler distribution Eq. Besides the likelihood analysis and visual inspection given in Fig. Very similar values b∈[7⋅103,12⋅103] and γ∈[1.8,2.5] were obtained for non-overlapping databases (for the English Wikipedia: b=7830, γ=1.68), and the fluctuations become smaller for increasing database size. These observations strongly suggest that the same fixed parameters provide a good description of all English texts (e.g., y(1900) and y(2000)). Therefore, hereafter we do not consider individual fits for each database and instead assume that Eq. =b∗=7873 and γ=γ∗=1.77, values obtained for our largest database Y(2000). Similar findings also apply to the other languages. In Tab. French and Spanish are also best described by Eq. For German and Russian Eq. However, we have strong indications that it provides a better account of the tails (r≫b∗) and therefore we expect that even larger databases will reveal the double power-law as the best fit also in these languages (see SM-Sec. Apart from being the smallest databases among the investigated languages, another feature affecting the fitting in German and especially in Russian is the higher degree of inflection in the morphology of these languages. We recall that no lemmatization was applied in our definition of words and, therefore, inflected words (obtained, e.g., by adding a suffix) are counted as distinct words. This reasoning explains the higher measured values of b∗ (vocabulary in the r−1 regime). From the fitting perspective, however, the large values of b∗ in German and Russian require even larger databases to characterize the deviations from the r−1 regime for r≫b∗.