Vector representations of text data in deep learning
Table 4.3: Average number of senses per word with marginal probability p≥0.05, learned by Disambiguated Skip-gram models with different values of the entropy cost γ.
['[ITALIC] γ', '0.0', '0.1', '0.25', '0.5', '0.75', '1.0']
AGH University of Science and Technology Faculty of Computer Science, Electronics and Telecommunications Department of Computer Science Doctoral dissertation Vector representations of text data in deep learning Karol Grzegorczyk Advisor: Professor Witold Dzwinel, PhD, DSc Co-advisor: Marcin Kurdziel, PhD Kraków, 2018 A2S1SS3 -Ch4F2sf4 Users of traditional machine learning systems needed to carefully design or select features, and to do that they needed to deeply understand their data. Feature engineering was, and sometimes still is, one of the most time-consuming, daunting and tedious tasks in a machine learning pipeline. Training a state-of-the-art learning algorithm with data represented by a poorly selected set of features most often leads to poor overall performance. Engineering of features is therefore a bottleneck on a way to achieve satisfying results. Some researchers go even further and suggest that we cannot talk about true artificial intelligence (AI) when features are handcrafted. we present histograms of marginal probabilities of senses of all words from the vocabulary. As depicted, in the case there is no entropy cost (black curve) marginal probabilites are distributed quite evenly: there are very few word senses with probabilities close to 0.0 or 1.0. For higher values of the entropy cost, marginal probability distributions are more polarized, i.e. there are more word senses with probabilities close to 0.0 or 1.0. Assuming that we prune senses having probabilities smaller than 0.05, we get on average 4.7 active senses for model without a penalty and as low as 2.5 senses on average for model with a strong γ=1.0 entropy cost ( Tab.