All Labels Are Equal, But Some More So Than Others: Fidelity Weighted Learning
Table 3: Performance of FWL using a single GP, a single GP after applying PCA on the input data, and the clustered GP as the teacher.
['Method', 'Document Ranking Robust04', 'Document Ranking Robust04', 'Document Ranking ClueWeb', 'Document Ranking ClueWeb', 'Sentiment Classification Robust04', 'Sentiment Classification ClueWeb']
We suggest using several GP={GPci} to explore the entire data space more effectively. Even though inducing points and stochastic methods make GPs more scalable we still observed poor performance when the entire dataset was modeled by a single GP. : We used Sparse Gaussian Process implemented in GPflow. The algorithm is scalable in the sense that it is not O(N3) as original GP is. It introduces inducing points in the data space and defines a variational lower bound for the marginal likelihood. The variational bound can now be optimized by stochastic methods which make the algorithm applicable in large datasets. However, the tightness of the bound depends on the location of inducing points which are found through the optimization process. We empirically observed that a single GP does not give a satisfactory accuracy on left-out test dataset. We hypothesized that this can be due to the inability of the algorithm to find good inducing points when the number of inducing points is restricted to just a few. Then we increased the number of inducing points M which trades off the scalability of the algorithm because it scales with O(NM2). Moreover, apart from scalability which is partly solved by stochastic methods, we argue that the structure of the entire space may not be explored well by a single GP and its inducing points. We guess this can be due to the observation that our datasets are distributed in a highly sparse way within the high dimensional embedding space. We also tried to cure the problem by means of PCA to reduce input dimensions and give a denser representation, but it did not result in a considerable improvement.