Generalization error and training error at singularities of multilayer perceptions

Shim Ichi Amari; Toinoko Ozeki; Hyeyoung Park

doi:10.1007/3-540-45720-8_37

Generalization error and training error at singularities of multilayer perceptions

Shim Ichi Amari, Toinoko Ozeki, Hyeyoung Park

RIKEN

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The neuromanifold or the parameter space of multilayer per- ceptrons includes complex singularities at which the Fisher information matrix degenerates. The parameters are unidentifiable at singularities, and t his causes serious difficulties in learning, known as plateaus in the cost function. The natural or adaptive natural gradient method is proposed for overcoming this difficulty. It. is important to study the relation bet w een t he generalization error and and the t raining error at t he singularities. because t he generalization error is estimated in terms of the training error. The generalization error is studied both for the maximum likelihood estimator (mle) and the Baycsian predictive distribution est imator in terms of the Gaussian random field, by using a simple model. This elucidates the strange behaviors of learning dynamics around singularities.

Original language	English
Title of host publication	Connectionist Models of Neurons, Learning Processes, and Artificial Intelligence - 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001, Proceedings
Publisher	Springer Verlag
Pages	325-332
Number of pages	8
Edition	PART 1
ISBN (Print)	3540422358, 9783540422358
DOIs	https://doi.org/10.1007/3-540-45720-8_37
State	Published - 2001
Event	6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001 - Granada, Spain Duration: 13 Jun 2001 → 15 Jun 2001

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	PART 1
Volume	2084 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001
Country/Territory	Spain
City	Granada
Period	13/06/01 → 15/06/01

Access to Document

10.1007/3-540-45720-8_37

Cite this

Amari, S. I., Ozeki, T., & Park, H. (2001). Generalization error and training error at singularities of multilayer perceptions. In Connectionist Models of Neurons, Learning Processes, and Artificial Intelligence - 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001, Proceedings (PART 1 ed., pp. 325-332). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 2084 LNCS, No. PART 1). Springer Verlag. https://doi.org/10.1007/3-540-45720-8_37

Amari, Shim Ichi ; Ozeki, Toinoko ; Park, Hyeyoung. / Generalization error and training error at singularities of multilayer perceptions. Connectionist Models of Neurons, Learning Processes, and Artificial Intelligence - 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001, Proceedings. PART 1. ed. Springer Verlag, 2001. pp. 325-332 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1).

@inproceedings{1eff8381a113466e9ebd8dba4aaeb81a,

title = "Generalization error and training error at singularities of multilayer perceptions",

abstract = "The neuromanifold or the parameter space of multilayer per- ceptrons includes complex singularities at which the Fisher information matrix degenerates. The parameters are unidentifiable at singularities, and t his causes serious difficulties in learning, known as plateaus in the cost function. The natural or adaptive natural gradient method is proposed for overcoming this difficulty. It. is important to study the relation bet w een t he generalization error and and the t raining error at t he singularities. because t he generalization error is estimated in terms of the training error. The generalization error is studied both for the maximum likelihood estimator (mle) and the Baycsian predictive distribution est imator in terms of the Gaussian random field, by using a simple model. This elucidates the strange behaviors of learning dynamics around singularities.",

author = "Amari, {Shim Ichi} and Toinoko Ozeki and Hyeyoung Park",

year = "2001",

doi = "10.1007/3-540-45720-8_37",

language = "English",

isbn = "3540422358",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

number = "PART 1",

pages = "325--332",

booktitle = "Connectionist Models of Neurons, Learning Processes, and Artificial Intelligence - 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001, Proceedings",

address = "Germany",

edition = "PART 1",

note = "6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001 ; Conference date: 13-06-2001 Through 15-06-2001",

}

Amari, SI, Ozeki, T & Park, H 2001, Generalization error and training error at singularities of multilayer perceptions. in Connectionist Models of Neurons, Learning Processes, and Artificial Intelligence - 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001, Proceedings. PART 1 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 1, vol. 2084 LNCS, Springer Verlag, pp. 325-332, 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001, Granada, Spain, 13/06/01. https://doi.org/10.1007/3-540-45720-8_37

Generalization error and training error at singularities of multilayer perceptions. / Amari, Shim Ichi; Ozeki, Toinoko; Park, Hyeyoung.
Connectionist Models of Neurons, Learning Processes, and Artificial Intelligence - 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001, Proceedings. PART 1. ed. Springer Verlag, 2001. p. 325-332 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 2084 LNCS, No. PART 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Generalization error and training error at singularities of multilayer perceptions

AU - Amari, Shim Ichi

AU - Ozeki, Toinoko

AU - Park, Hyeyoung

PY - 2001

Y1 - 2001

N2 - The neuromanifold or the parameter space of multilayer per- ceptrons includes complex singularities at which the Fisher information matrix degenerates. The parameters are unidentifiable at singularities, and t his causes serious difficulties in learning, known as plateaus in the cost function. The natural or adaptive natural gradient method is proposed for overcoming this difficulty. It. is important to study the relation bet w een t he generalization error and and the t raining error at t he singularities. because t he generalization error is estimated in terms of the training error. The generalization error is studied both for the maximum likelihood estimator (mle) and the Baycsian predictive distribution est imator in terms of the Gaussian random field, by using a simple model. This elucidates the strange behaviors of learning dynamics around singularities.

AB - The neuromanifold or the parameter space of multilayer per- ceptrons includes complex singularities at which the Fisher information matrix degenerates. The parameters are unidentifiable at singularities, and t his causes serious difficulties in learning, known as plateaus in the cost function. The natural or adaptive natural gradient method is proposed for overcoming this difficulty. It. is important to study the relation bet w een t he generalization error and and the t raining error at t he singularities. because t he generalization error is estimated in terms of the training error. The generalization error is studied both for the maximum likelihood estimator (mle) and the Baycsian predictive distribution est imator in terms of the Gaussian random field, by using a simple model. This elucidates the strange behaviors of learning dynamics around singularities.

UR - http://www.scopus.com/inward/record.url?scp=84902189129&partnerID=8YFLogxK

U2 - 10.1007/3-540-45720-8_37

DO - 10.1007/3-540-45720-8_37

M3 - Conference contribution

AN - SCOPUS:84902189129

SN - 3540422358

SN - 9783540422358

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 325

EP - 332

BT - Connectionist Models of Neurons, Learning Processes, and Artificial Intelligence - 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001, Proceedings

PB - Springer Verlag

T2 - 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001

Y2 - 13 June 2001 through 15 June 2001

ER -

Amari SI, Ozeki T, Park H. Generalization error and training error at singularities of multilayer perceptions. In Connectionist Models of Neurons, Learning Processes, and Artificial Intelligence - 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001, Proceedings. PART 1 ed. Springer Verlag. 2001. p. 325-332. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1). doi: 10.1007/3-540-45720-8_37

Generalization error and training error at singularities of multilayer perceptions

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this