hh.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A Simple trick for estimating the weight decay parameter
Halmstad University, School of Information Technology, Halmstad Embedded and Intelligent Systems Research (EIS).ORCID iD: 0000-0001-5163-2997
2012 (English)In: Lecture Notes in Computer Science, ISSN 0302-9743, E-ISSN 1611-3349, Vol. 7700 LECTURE NO, p. 69-89Article in journal (Refereed) Published
Abstract [en]

We present a simple trick to get an approximate estimate of the weight decay parameter λ. The method combines early stopping and weight decay, into the estimate λ̂ = ||∇E(Wes)||/||2W es||, where Wes is the set of weights at the early stopping point, and E(W) is the training data fit error. The estimate is demonstrated and compared to the standard cross-validation procedure for λ selection on one synthetic and four real life data sets. The result is that is as good an estimator for the optimal weight decay parameter value as the standard search estimate, but orders of magnitude quicker to compute. The results also show that weight decay can produce solutions that are significantly superior to committees of networks trained with early stopping. © Springer-Verlag Berlin Heidelberg 2012.

Place, publisher, year, edition, pages
Berlin: Springer Berlin/Heidelberg, 2012. Vol. 7700 LECTURE NO, p. 69-89
Keywords [en]
Cross validation, Early stopping, Optimal weight, Orders of magnitude, Real life datasets, Training data, Weight decay, Artificial intelligence, Estimation
National Category
Signal Processing
Identifiers
URN: urn:nbn:se:hh:diva-37450DOI: 10.1007/978-3-642-35289-8-6Scopus ID: 2-s2.0-84872568634OAI: oai:DiVA.org:hh-37450DiVA, id: diva2:1242363
Available from: 2018-08-28 Created: 2018-08-28 Last updated: 2018-08-28Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records BETA

Rögnvaldsson, Thorsteinn

Search in DiVA

By author/editor
Rögnvaldsson, Thorsteinn
By organisation
Halmstad Embedded and Intelligent Systems Research (EIS)
In the same journal
Lecture Notes in Computer Science
Signal Processing

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 37 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf