ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA
Table 8: Performance comparison of running LSTM on ESE, CPU and GPU
['[BOLD] Plat.  [BOLD] Matrix', '[BOLD] ESE on FPGA (ours)  [BOLD] Matrix', '[BOLD] ESE on FPGA (ours)  [BOLD] Sparsity', '[BOLD] ESE on FPGA (ours)  [BOLD] Compres.', '[BOLD] ESE on FPGA (ours)  [BOLD] Theoreti.', '[BOLD] ESE on FPGA (ours)  [BOLD] Real', '[BOLD] ESE on FPGA (ours)  [BOLD] Total', '[BOLD] ESE on FPGA (ours)  [BOLD] Real', '[BOLD] ESE on FPGA (ours)  [BOLD] Equ.', '[BOLD] ESE on FPGA (ours)  [BOLD] Equ.', '[BOLD] CPU  [BOLD] Real Comput.', '[BOLD] CPU  [BOLD] Real Comput.', '[BOLD] GPU  [BOLD] Real Comput.', '[BOLD] GPU  [BOLD] Real Comput.']
The CPU implementation used MKL BLAS and MKL SPBLAS for dense/sparse implementation, and the GPU implementation used cuBlas and cuSparse. We optimized the CPU/GPU speed by combining the four matrices of the i, f, o, c gates that have no dependency into one large matrix. Both mklSparse and cuSparse implementation results in significant lower utilization of peak CPU/GPU performance for the interested matrix size (relatively small) and sparsity (around 10% non-zeros). We implemented the whole LSTM on ESE. The model was pruned to 10% non-zeros. There are 11.2% non-zeros taking padding zeros into account. On ESE, the total throughput is 282 GOPS with the sparse LSTM, which corresponds to 2.52 TOPS on the dense LSTM. Processing the LSTM with 1024 hidden elements, ESE takes 82.7 us, CPU takes 6017.3/3569.9 us (dense/sparse), and GPU takes 240.2/287.4 us (dense/sparse). With batch=32, CPU sparse is faster than dense because CPU is good at serial processing, while GPU sparse is slower than dense because GPU is throughput oriented. With no batching, we observed both CPU and GPU are faster for the sparse LSTM because the saving of memory bandwidth is more salient.