Assessing the Linguistic Productivity of Unsupervised Deep Neural Networks
Table 2: Expected and empirical overlap scores for adult- and autoencoder-generated language with varying levels of dropout. Expected overlap scores were calculated as in Yang (2011). Empirical overlap was calculated as the percent of unique nouns that appeared immediately following both a and the.
['[EMPTY]', '[ITALIC]  [BOLD] N', '[ITALIC]  [BOLD] S', '[BOLD] Exp. Over.', '[BOLD] Emp. Over.']
We note that the expected overlap measure consistently overestimates the productivity of all learners, including the adult-generated text. It is unclear why this should be the case, but could be a result of capping the model vocabularies, resulting in lower N values. In particular, the autoencoders tend to produce a relatively limited set of nouns. Looking at empirical overlap measures, the worst-performing models are the bigram and trigram models with overlap scores below 30%. The AEs fair much better all producing overlap scores over 50%. The 30% dropout AE is actually able to match the overlap score of the original adult-generated corpus (59.4% vs. 59.3%).