Network optimization through learning and pruning in neuromanifold

In this paper, we propose an optimization method of neural networks based on the geometrical structure of neuromanifold. The optimizing process starts from the manifold of sufficiently large network model. In the manifold of the given network structure, we first find an optimal point, which achieves good generalization performance. To do this, we propose an extension of the adaptive natural gradient learning with regularization term. Using hierarchical structure of neuromanifold, we then try to optimize the network structure. To do this, we apply the natural pruning method starting from the current optimal parameter point. The whole optimization process can be explained from the geometrical point of view. We confirm the generalization performance of the optimized network by the proposed method through experiments on benchmark data sets.