End-to-End Neural Speaker Diarization with Permutation-Free Objectives
Table 4: DERs (%) on different overlapping conditions. For the evaluation on simulated mixtures, the proposed model is trained using 100,000 mixtures generated with β=2. For the evaluation on the CALLHOME dataset, the proposed model is trained with 26,712 telephone recordings. The DER without the domain adaptation is shown in the parenthesis.
['Evaluation set  [ITALIC] β', 'Simulated mixtures 2', 'Simulated mixtures 3', 'Simulated mixtures 5', 'CALLHOME -']
To investigate the robustness to variable conditions, we evaluated different overlap ratio controlled by the average interval β. Larger β values generate less overlapping speech. The baseline systems show better performance on less overlapping speech as expected. However, the proposed method unexpectedly showed degraded performance on less overlapping speech. The result suggests that the network had overfit to the specific overlap ratio: 27.3%. Investigation with various overlap ratio settings of training data is among our future work. In addition, we evaluated the proposed method on real telephone conversations using the CALLHOME dataset. We split two-speaker recordings from the CALLHOME dataset into two subsets: an adaptation set of 155 recordings and a test set of 148 recordings. The overlap ratio of the training data was 5.8%. Then, it was retrained with the adaptation set. For this retraining, we used the Adam optimizer with initial learning rate of 10−6 and ran 5 epochs. For the postprocessing, we adjusted the threshold to 0.6 so that the DER of the adaptation set has the minimum value. Unfortunately, the proposed method produces worse DER than the baseline systems. This is likely because our training set has very different overlap ratio (5.8%) from the CALLHOME test set (11.8%). To reduce this condition mismatch, we tried domain adaptation. The result showed a significant DER reduction. The relative improvement introduced by domain adaptation was 25.6%. Although the DER of the proposed method was still behind those of the baseline systems, we expect it will be much improved by developing better simulation techniques of training data or just by feeding more real data, as was suggested by the result with simulated data. We will address these directions in our future work.