Parameter Reduction for Deep Neural Network Based Acoustic Models Using Sparsity Regularized Factorization Neurons

In this paper, we propose a deep neural network (DNN) model parameter reduction technique for an efficient acoustic model. One of the most common DNN model parameter reduction techniques is to use low-rank matrix approximation. Although it can reduce a significant number of model parameters, there are two problems to be considered; one is the performance degradation, and the other is the appropriate rank selection. To solve these problems, retraining is carried out, and so-called explained variance is used. However, retraining takes additional time, and explained variance is not directly related to classification performance.Therefore, to mitigate these problems, we propose an approach that performs model parameter reduction simultaneously during model training from the aspect of minimizing classification error. The proposed method uses the product of three factorized matrices instead of a dense weight matrix, and applies sparsity constraint to make entries of the center diagonal matrix zero. After finishing training, a parameter-reduced model can be obtained by discarding the left and right vectors corresponding to zero entries within the center diagonal matrix.