Subword RNNLM Approximations for Out-Of-Vocabulary Keyword Search
Table 1: The table displays the length rates in #subwords per occurrence for frequent words (FW, training set frequency ≥5) on the training and test sets, and for OOV words on the test set from the Arabic and Finnish KWS described in Section 4. OOV Rate on these datasets is 2% approx. Length rates for both single- and multi-character subwords are presented.
['Language', 'FW (Train) Single', 'FW (Train) Multi', 'FW (Test) Single', 'FW (Test) Multi', 'OOV (Test) Single', 'OOV (Test) Multi']
In this paper, we also focus on capturing longer contexts in contrast to previous work on OOV KWS. Additionally, higher-order n-grams can be beneficial for capturing long-term dependencies in an approximated RNNLM. In the experiments, we train our baseline LMs and RNNLMs on single- and multi-character subwords. To evaluate these LMs on their OOV detection efficacy, we setup two KWS tasks on Arabic and Finnish datasets using only such OOV keywords that do not appear in the training data There exist a few approximation techniques for converting RNNLMs to n-gram LMs: variational approximation [Deoras2011], probability-conversion [Adel2014] and iterative conversion [Adel2014, Arisoy:2014:CNN:2583704.2583722]. Prior work [Adel2014] compared these techniques and the best approximating technique – iterative conversion – outperformed other methods using smaller order n-grams in a speech recognition task. However, iterative conversion method’s effectiveness on low-order word n-grams seems sub-optimal for OOVs. Among other methods, the variational approximation method is also not a good fit for our purposes, as it can fail to sample rare subword contexts, which might be present in the subword sequences of OOVs. Hence, we use the probability-conversion method to develop a more efficient method and introduce an n-gram-growing variant of the algorithm for approximating subword RNNLMs to larger context sizes.