Combinatorial Topic Models using Small-Variance Asymptotics
Figure 1: Left: Running time comparison per iteration (in secs) of CGS to the facility location improved word algorithm (Word) and local refinement (Refine), on data sets of different sizes. Word/CGS and Refine/CGS refer to the ratio of Word and Refine to CGS. For larger datasets, Word takes roughly 2 Gibbs iterations and Refine takes roughly 1 Gibbs iteration. Right: Comparison of topic reconstruction errors of different algorithms with different sizes of SynthB.
['Method', 'Number of Documents 5k', 'Number of Documents 10k', 'Number of Documents 50k', 'Number of Documents 100k', 'Number of Documents 500k', 'Number of Documents 1M']
Topic reconstruction error. Now we look at the reconstruction error between the true topic-word distributions and the learned distributions. We see that, on this data, the Anchor and Word+Refine methods perform the best; see Appendix C for further results and discussion. Running Time. The two most expensive steps of the Word+Refine algorithm are the word assignments via facility location and the local refinement step (the other steps of the algorithm are lower-order). The relative runnings times improve as the data set sizes gets larger and, on large data sets, an iteration of Refine is roughly equivalent to one Gibbs iteration while an iteration of Word is roughly equivalent to two Gibbs iterations. Since one typically runs thousands of Gibbs iterations (while ours runs in 10 iterations even on very large data sets), we can observe several orders of magnitude improvement in speed by our algorithm. Further, running times could be significantly enhanced by noting that the facility location algorithm trivially parallellizes. In addition to these results, we found our per-iteration running times to be consistently faster than VB.