TY - GEN
T1 - Reconsideration to pruning and regularization for complexity optimization in neural networks
AU - Park, Hyeyoung
AU - Lee, Hyunjin
N1 - Publisher Copyright:
© 2002 Nanyang Technological University.
PY - 2002
Y1 - 2002
N2 - The ultimate purpose of neural network design is to find an optimal network that can give good generalization performance with compact structure. To achieve this, it is necessary to control complexities of networks so as to avoid its overfitting to noisy learning data. The most popular methods for complexity control are the pruning method and the regularization method. Even though there have been many variations in the methods, the peculiar properties of each method compared to others has not been so clear. We reconsider the pruning strategy from a geometrical and statistical viewpoint, and show that the natural pruning method is in accordance with the geometrical and statistical intuition in choosing connections to be pruned. In addition, we also suggest that the regularization method should be used in combination with natural pruning in order to improve the optimization performance. We also show some experimental results supporting our suggestions.
AB - The ultimate purpose of neural network design is to find an optimal network that can give good generalization performance with compact structure. To achieve this, it is necessary to control complexities of networks so as to avoid its overfitting to noisy learning data. The most popular methods for complexity control are the pruning method and the regularization method. Even though there have been many variations in the methods, the peculiar properties of each method compared to others has not been so clear. We reconsider the pruning strategy from a geometrical and statistical viewpoint, and show that the natural pruning method is in accordance with the geometrical and statistical intuition in choosing connections to be pruned. In addition, we also suggest that the regularization method should be used in combination with natural pruning in order to improve the optimization performance. We also show some experimental results supporting our suggestions.
UR - http://www.scopus.com/inward/record.url?scp=84966327389&partnerID=8YFLogxK
U2 - 10.1109/ICONIP.2002.1198955
DO - 10.1109/ICONIP.2002.1198955
M3 - Conference contribution
AN - SCOPUS:84966327389
T3 - ICONIP 2002 - Proceedings of the 9th International Conference on Neural Information Processing: Computational Intelligence for the E-Age
SP - 1649
EP - 1653
BT - ICONIP 2002 - Proceedings of the 9th International Conference on Neural Information Processing
A2 - Fukushima, Kunihiko
A2 - Wang, Lipo
A2 - Rajapakse, Jagath C.
A2 - Lee, Soo-Young
A2 - Yao, Xin
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 9th International Conference on Neural Information Processing, ICONIP 2002
Y2 - 18 November 2002 through 22 November 2002
ER -