LSTM-acceleration-with-singular-value-decomposition

Model size is often limited by the capabilities of the deployed hardware, or in relation to LSTMs the timing requirement. By using the singular value decomposition, the model can be compressed into a smaller size with presumably a shorter forward pass time. The reduced model will be smaller than the starting model, but still be composed of full matrices. Model reduction is done with the use of two types of regularziers: the Hoyer regularizer induces sparsity in the singular values while an orthogonality regularzier preserves orthogonality in the left and right eigenmatrices.