Topic Discovery in Massive Text Corpora Based on Min-Hashing1footnote 11footnote 1©2019. Licensed under the Creative Commons CC-BY-NC-ND 4.0
Table 5: Time in seconds to discover topics on the 20 Newsgroups, Reuters and Wikipedia corpora with vocabularies of 10000, 100000 and 1000000 words, respectively.
['[BOLD] Corpus', '[BOLD] SMH  [BOLD] 0.04', '[BOLD] SMH  [BOLD] 0.06', '[BOLD] SMH  [BOLD] 0.08', '[BOLD] SMH  [BOLD] 0.10', '[BOLD] Online LDA  [BOLD] 200', '[BOLD] Online LDA  [BOLD] 400', '[BOLD] Online LDA  [BOLD] 600']
In order to evaluate the scalability of SMH, we discovered topics from the 20 Newsgroups, Reuters, and Spanish and English Wikipedia corpora whose sizes range from thousands to millions of documents and whose vocabularies range from thousands of words to as much as one million words. All experiments were performed on a DellTM PowerEdgeTM with 2 Intel® Xeon® CPUs X5650@2.67GHz (12 cores) and 32 GB of RAM. For comparison purposes, each experiment used only a single thread. Note that the time complexity of SMH is linear with respect to both corpus and vocabulary size. Remarkably, SMH took at most 6.4 hours (when η=0.04) and as little as 58 minutes (when η=0.10) to process the entire Wikipedia in English, which contains over 5 million documents with a vocabulary of 1 million words. Although the time required by both SMH and Online LDA to process the 20 Newsgroups corpus was very similar, for the Reuters corpus SMH was significantly faster than Online LDA.