Contents
Table 9: Term frequency-inverse document frequency (tf-idf) for certain key words. The corpus here is the set of all titles for all papers in all five sections of the arXiv. These sections become the five documents of the corpus.
['Word', 'hep-th', 'hep-ph', 'hep-lat', 'gr-qc', 'math-ph']
Recall that words that are extremely common in individual contexts (here, specifically, paper titles), across an entire arXiv section, will have very low values of tf-idf. For example, words such as “theory” and “model” appears in all sections, and would therefore receive a vanishing value of tf-idf. Therefore, it is only illustrative to include words which do not trivially have a score of zero for all sections. Our collection of titles is sufficiently large in all cases that there are no words which appear in all paper titles, even prior to the cleaning step, which removes small words like “a” and “the”.