Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence
Table 2: Averaged results over 5 numbers of topics.
['[BOLD] Model', '[BOLD] Avg  [ITALIC] τ', '[BOLD] Avg  [ITALIC] α', '[BOLD] Avg  [ITALIC] ρ']
We compute all the metrics for 25, 50, 75, 100 and 150 topics. The topics generated with our model are more coherent than the ones of the baselines. Occasionally, though, they are less diverse than Neural-ProdLDA or NVDM. In general, our model provides the most coherent topics across all corpora and topic settings. \newcitesrivastava2017avitm reported that NVDM obtains low coherence. We observe that our model suffers the most on non-standard text like tweets, where the representations generated by BERT do not provide enough information to the model; this might also be the results of generating the embedded representations from pre-processed text.