Vector representations of text data in deep learning
Table 3.4: Comparison of performance of different binary units for 32 bit model trained on the 20 Newsgroups dataset.
['Binarization approach', 'MAP', 'NDCG@10']
AGH University of Science and Technology Faculty of Computer Science, Electronics and Telecommunications Department of Computer Science Doctoral dissertation Vector representations of text data in deep learning Karol Grzegorczyk Advisor: Professor Witold Dzwinel, PhD, DSc Co-advisor: Marcin Kurdziel, PhD Kraków, 2018 A2S1SS3 -Ch4F2sf4 Users of traditional machine learning systems needed to carefully design or select features, and to do that they needed to deeply understand their data. Feature engineering was, and sometimes still is, one of the most time-consuming, daunting and tedious tasks in a machine learning pipeline. Training a state-of-the-art learning algorithm with data represented by a poorly selected set of features most often leads to poor overall performance. Engineering of features is therefore a bottleneck on a way to achieve satisfying results. Some researchers go even further and suggest that we cannot talk about true artificial intelligence (AI) when features are handcrafted. In Tab. In case of the former, we tried the approaches proposed by Salakhutdinov and Krizhevsky. In case of the latter, we examined straight-through gradient estimator in both described variants Model hyperparameters were selected separately for each binarization approach using hold-out validation set. Standard deviation of the Gaussian noise added to the pre-activation signal in the Salakhutdinov’s binarization is 0.4. Learning rate for that case is 1.0. Learning rates for BSN have higher values. They are 1.5 for Eq. In contrast to the Krizhevsky’s binarization, dropout does not improve the results of the Salakhutdinov’s binarization. Neither in case of BSNs. In variants that use slope annealing we increase the slope hiperparameter by 0.1 in each epoch. As can be concluded from the table Tab.