Word Familiarity and Frequency
Table 4: Coverage and Correlation Coefficients
['label', 'Coverage of the words in the familiarity list', 'Pearson’s correlation coefficient', 'Spearman’s correlation coefficient']
The table shows three values for each corpus: Coverage: the number (and percentage) of the words in the familiarity list found in the corpus, Pearson: Pearson’s product-moment correlation coefficient, and Spearman: Spearman’s rank-order correlation coefficient. Note that missing values —words which are in the familiarity list but not in the corpus— were not considered when calculating Pearson’s coefficient, whereas they were taken into consideration when calculating Spearman’s coefficient. How significant these differences in the correlation coefficients are can be intuitively understood through simulation as follows. For each English and Japanese familiarity list, a copy of a list is generated (which of course perfectly correlates with the original). For this copy, two successively ranked words are randomly selected, and then their orders are reversed and the correlation with the original list is measured. The horizontal axes show the number of reversed pairs, and vertical axes show the correlation. In the English case, starting from 4894 ranked words, it required only 93 reverses to attain the BNC correlation, and 124 reverses to reach the Wikipedia-E level. In Japanese, since the Amano list is about ten times larger, more reverses are required to decrease the correlation from Mainichi to Aozora. For the 68,550 words of the Amano list, only 1340 additional reverses are needed to decrease the Mainichi correlation level to that of Aozora. Correlation thus decreases quickly even with slight inconsistencies with the original. The correlation scores show that the correlation rises as the amount of data increases. This is natural, since, for a given corpus, high frequency is a necessary condition for high familiarity, and a larger corpus would increase the sampling size. Therefore, the data size seems to account for some of the correlation increase. The difference is caused not only because of the size counts, though.