End-To-End Visual Speech Recognition With LSTMs
Table 1: Classification Accuracy on the OuluVS2 database. The end-to-end models are evaluated using the protocol suggested in [22] where 40 subjects are used for training and validation and 12 subjects are used for testing. † These models use a leave-one-subject-out cross validation for evaluation.
['Method', 'Classification']
Since this database has been released recently only the baseline results provided by the creators are available. The best provided baseline result, 74.8%, is achieved by HMMs in combination with DCT features. We first test each stream of the end-to-end model individually, i.e., just the encoding layers and the LSTM layer are considered. It is interesting to note that both streams outperform the baseline performance. The best overall result is achieved by the end-to-end 2-stream model, shown in Fig. We should also emphasise that the baseline performance is evaluated using a leave-one-subject-out cross validation approach which means there are 51 subjects for training and validation and only one subject for testing in each iteration. On the other hand, we use much fewer subjects for training and validation, 40, and many more subjects for testing, 12, which makes the problem more challenging. Even in this case, the end-to-end system results in a significant improvement over the baseline performance.