LEAP System for SRE19 CTS Challenge - Improvements and Error Analysis
Table 4: Performance of post-eval systems using improved calibration and adaptive score normalization (AS-Norm).
['Model', 'Train Datasets', 'SRE18 Dev EER (%)', 'SRE18 Dev  [ITALIC] CMin', 'SRE19 Eval EER (%)', 'SRE19 Eval  [ITALIC] CMin', 'SRE19 Eval  [ITALIC] Cprimary']
In our previous work for SRE18 [ramoji2019leap], we proposed an alternative approach to calibration, where the target and non-target scores are modelled as Gaussian distribution with a shared variance. The calibration procedure in this case involved the shifting of scores so that the threshold corresponding to the minimum cost on the development set is the point where the actual cost is computed on the evaluation trials (logβ of [Sadjadi19plan]). This was done with the assumption that the score distributions of the development and evaluation trials match closely. Thus, the threshold where Cmin is achieved in the development set may potentially match with evaluation trials. In the case of SRE18 evaluation [ramoji2019leap], the development and evaluation score distributions were more or less the same, and the threshold that minimized the detection cost were very close. However, in SRE19, there was no exclusive matched development dataset provided. Hence, aforementioned calibration method using the SRE18 development dataset apploed on the SRE19 evaluation trials (as done for our submitted systems) turned out to be ineffective. Given the keys for SRE19 evaluation, we performed a score analysis and this is shown in As seen here, the computation of Cprimary using the distribution of SRE18 resulted in a sub-optimal calibration of the scores. In the post-eval efforts, we have performed score calibration based on the approach described in [brummer2013bosaris]. As seen in the plot, this matched the primary cost metric Cprimary (actDCF) closely with the minimum cost. We experiment with various cohort based normalization techniques [karam2011towards, Matejka2017] using the SRE18 dev unlabelled set as the cohort. The best improvements were observed with the adaptive symmetric normalization (referred to as AS-Norm Type 1 in [Matejka2017]). We achieve 24% relative improvement for the VoxCeleb x-vector system (XV1) and 21% relative improvement for the full x-vector system (XV3) on SRE18 development set. We achieve a comparatively lower but consistent improvements of about 15% on an average for the SRE19 evaluation set for all the systems.