Constructive Approximation

, volume 26 , issue 2 , pages 289-315

On Early Stopping in Gradient Descent Learning

Yuan Yao ¹

Lorenzo Rosasco ^{2, 3}

ANDREA CAPONNETTO ^{2, 3}

Hide authors affiliations Show authors affiliations: 3 affiliations

Department of Mathematics, University of California, Berkeley, CA 94720, USA |

C.B.C.L., Massachusetts Institute of Technology, Bldg. E25-201, 45 Carleton St., Cambridge, MA 02142, USA |

DISI, Universita di Genova, Via Dodecaneso 35, 16146 Genova, Italy |

Publication type: Journal Article

Publication date: 2007-04-04

Springer Nature

Constructive Approximation

scimago Q1

wos Q1

SJR: 1.871

CiteScore: 5.0

Impact factor: 1.2

ISSN: 01764276, 14320940

DOI: 10.1007/s00365-006-0663-2

Copy DOI

General Mathematics

Computational Mathematics

Analysis

Abstract

In this paper we study a family of gradient descent algorithms to approximate the regression function from reproducing kernel Hilbert spaces (RKHSs), the family being characterized by a polynomial decreasing rate of step sizes (or learning rate). By solving a bias-variance trade-off we obtain an early stopping rule and some probabilistic upper bounds for the convergence of the algorithms. We also discuss the implication of these results in the context of classification where some fast convergence rates can be achieved for plug-in classifiers. Some connections are addressed with Boosting, Landweber iterations, and the online learning algorithms as stochastic approximations of the gradient descent method.

Found

1 citation

Lazarev Mikhail

🥼 🤝

PhD in Physics and Mathematics

6 publications, 26 citations, 1 review

h-index: 3

National Research University Higher School of Economics

Research interests

2D Materials

Computer vision

Inteligencia artificial

Machine learning

Optics

Physics of semiconductors

1 citation

Singh Amninder

14 publications, 109 citations

h-index: 7

University of California, Riverside

1 citation

Nepal Subigya

37 publications, 1 117 citations

h-index: 16

Dartmouth College

1 citation

Rudi Alessandro

15 publications, 114 citations, 1 review

h-index: 6

National Institute for Research in Digital Science and Technology

Paris Sciences et Lettres

1 citation

Xiao Xusheng

65 publications, 2 173 citations

h-index: 22

Arizona State University

Case Western Reserve University

Research interests

Software Engineering

1 citation

Truong Vinh

🥼 🤝

PhD in Engineering, associate professor

128 publications, 991 citations, 20 reviews

h-index: 17

Ho Chi Minh City Open University

Suan Sunandha Rajabhat University

University of Montpellier

University of the Littoral Opal Coast

Top-30

Journals

	2 4 6 8 10 12 14
Lecture Notes in Computer Science	Lecture Notes in Computer Science, 14, 2.18% Lecture Notes in Computer Science 14 publications, 2.18%
IEEE Access	IEEE Access, 14, 2.18% IEEE Access 14 publications, 2.18%
Applied Sciences (Switzerland)	Applied Sciences (Switzerland), 10, 1.56% Applied Sciences (Switzerland) 10 publications, 1.56%
Remote Sensing	Remote Sensing, 8, 1.24% Remote Sensing 8 publications, 1.24%
Scientific Reports	Scientific Reports, 8, 1.24% Scientific Reports 8 publications, 1.24%
Applied and Computational Harmonic Analysis	Applied and Computational Harmonic Analysis, 8, 1.24% Applied and Computational Harmonic Analysis 8 publications, 1.24%
Mathematics	Mathematics, 7, 1.09% Mathematics 7 publications, 1.09%
IEEE Transactions on Neural Networks and Learning Systems	IEEE Transactions on Neural Networks and Learning Systems, 7, 1.09% IEEE Transactions on Neural Networks and Learning Systems 7 publications, 1.09%
Analysis and Applications	Analysis and Applications, 6, 0.93% Analysis and Applications 6 publications, 0.93%
Electronics (Switzerland)	Electronics (Switzerland), 5, 0.78% Electronics (Switzerland) 5 publications, 0.78%
Sensors	Sensors, 5, 0.78% Sensors 5 publications, 0.78%
Multimedia Tools and Applications	Multimedia Tools and Applications, 5, 0.78% Multimedia Tools and Applications 5 publications, 0.78%
Neural Networks	Neural Networks, 5, 0.78% Neural Networks 5 publications, 0.78%
Inverse Problems	Inverse Problems, 5, 0.78% Inverse Problems 5 publications, 0.78%
Lecture Notes in Networks and Systems	Lecture Notes in Networks and Systems, 5, 0.78% Lecture Notes in Networks and Systems 5 publications, 0.78%
Constructive Approximation	Constructive Approximation, 4, 0.62% Constructive Approximation 4 publications, 0.62%
Mechanical Systems and Signal Processing	Mechanical Systems and Signal Processing, 4, 0.62% Mechanical Systems and Signal Processing 4 publications, 0.62%
Neurocomputing	Neurocomputing, 4, 0.62% Neurocomputing 4 publications, 0.62%
Applied Energy	Applied Energy, 4, 0.62% Applied Energy 4 publications, 0.62%
Applied Soft Computing Journal	Applied Soft Computing Journal, 4, 0.62% Applied Soft Computing Journal 4 publications, 0.62%
Communications in Computer and Information Science	Communications in Computer and Information Science, 4, 0.62% Communications in Computer and Information Science 4 publications, 0.62%
Physical Review D	Physical Review D, 3, 0.47% Physical Review D 3 publications, 0.47%
Neural Computation	Neural Computation, 3, 0.47% Neural Computation 3 publications, 0.47%
IEEE Transactions on Geoscience and Remote Sensing	IEEE Transactions on Geoscience and Remote Sensing, 3, 0.47% IEEE Transactions on Geoscience and Remote Sensing 3 publications, 0.47%
SIAM Journal on Numerical Analysis	SIAM Journal on Numerical Analysis, 3, 0.47% SIAM Journal on Numerical Analysis 3 publications, 0.47%
Annals of Statistics	Annals of Statistics, 3, 0.47% Annals of Statistics 3 publications, 0.47%
Entropy	Entropy, 3, 0.47% Entropy 3 publications, 0.47%
Neural Processing Letters	Neural Processing Letters, 3, 0.47% Neural Processing Letters 3 publications, 0.47%
Applied Intelligence	Applied Intelligence, 3, 0.47% Applied Intelligence 3 publications, 0.47%
	2 4 6 8 10 12 14

Publishers

	20 40 60 80 100 120 140
Institute of Electrical and Electronics Engineers (IEEE)	Institute of Electrical and Electronics Engineers (IEEE), 137, 21.31% Institute of Electrical and Electronics Engineers (IEEE) 137 publications, 21.31%
Springer Nature	Springer Nature, 119, 18.51% Springer Nature 119 publications, 18.51%
Elsevier	Elsevier, 114, 17.73% Elsevier 114 publications, 17.73%
MDPI	MDPI, 64, 9.95% MDPI 64 publications, 9.95%
Association for Computing Machinery (ACM)	Association for Computing Machinery (ACM), 24, 3.73% Association for Computing Machinery (ACM) 24 publications, 3.73%
Wiley	Wiley, 19, 2.95% Wiley 19 publications, 2.95%
Oxford University Press	Oxford University Press, 12, 1.87% Oxford University Press 12 publications, 1.87%
World Scientific	World Scientific, 10, 1.56% World Scientific 10 publications, 1.56%
Cold Spring Harbor Laboratory	Cold Spring Harbor Laboratory, 10, 1.56% Cold Spring Harbor Laboratory 10 publications, 1.56%
American Physical Society (APS)	American Physical Society (APS), 8, 1.24% American Physical Society (APS) 8 publications, 1.24%
IOP Publishing	IOP Publishing, 8, 1.24% IOP Publishing 8 publications, 1.24%
American Chemical Society (ACS)	American Chemical Society (ACS), 8, 1.24% American Chemical Society (ACS) 8 publications, 1.24%
Taylor & Francis	Taylor & Francis, 8, 1.24% Taylor & Francis 8 publications, 1.24%
Institute of Mathematical Statistics	Institute of Mathematical Statistics, 7, 1.09% Institute of Mathematical Statistics 7 publications, 1.09%
Frontiers Media S.A.	Frontiers Media S.A., 7, 1.09% Frontiers Media S.A. 7 publications, 1.09%
Copernicus	Copernicus, 6, 0.93% Copernicus 6 publications, 0.93%
AIP Publishing	AIP Publishing, 5, 0.78% AIP Publishing 5 publications, 0.78%
Hindawi Limited	Hindawi Limited, 5, 0.78% Hindawi Limited 5 publications, 0.78%
SAGE	SAGE, 4, 0.62% SAGE 4 publications, 0.62%
MIT Press	MIT Press, 4, 0.62% MIT Press 4 publications, 0.62%
Society for Industrial and Applied Mathematics (SIAM)	Society for Industrial and Applied Mathematics (SIAM), 3, 0.47% Society for Industrial and Applied Mathematics (SIAM) 3 publications, 0.47%
Public Library of Science (PLoS)	Public Library of Science (PLoS), 3, 0.47% Public Library of Science (PLoS) 3 publications, 0.47%
Walter de Gruyter	Walter de Gruyter, 3, 0.47% Walter de Gruyter 3 publications, 0.47%
SPIE-Intl Soc Optical Eng	SPIE-Intl Soc Optical Eng, 2, 0.31% SPIE-Intl Soc Optical Eng 2 publications, 0.31%
Institute for Operations Research and the Management Sciences (INFORMS)	Institute for Operations Research and the Management Sciences (INFORMS), 2, 0.31% Institute for Operations Research and the Management Sciences (INFORMS) 2 publications, 0.31%
Tech Science Press	Tech Science Press, 2, 0.31% Tech Science Press 2 publications, 0.31%
Social Science Electronic Publishing	Social Science Electronic Publishing, 2, 0.31% Social Science Electronic Publishing 2 publications, 0.31%
Royal Society of Chemistry (RSC)	Royal Society of Chemistry (RSC), 2, 0.31% Royal Society of Chemistry (RSC) 2 publications, 0.31%
Society of Exploration Geophysicists	Society of Exploration Geophysicists, 2, 0.31% Society of Exploration Geophysicists 2 publications, 0.31%
	20 40 60 80 100 120 140

We do not take into account publications without a DOI.
Statistics recalculated weekly.

Are you a researcher?

Create a profile to get free access to personal recommendations for colleagues and new articles.

Metrics

643

Cite this

GOST |

Cite this

GOST Copy

Yao Y. et al. On Early Stopping in Gradient Descent Learning // Constructive Approximation. 2007. Vol. 26. No. 2. pp. 289-315.

GOST all authors (up to 50) Copy

Yao Y., Rosasco L., CAPONNETTO A. On Early Stopping in Gradient Descent Learning // Constructive Approximation. 2007. Vol. 26. No. 2. pp. 289-315.

RIS |

Cite this

RIS Copy

TY - JOUR

DO - 10.1007/s00365-006-0663-2

UR - https://doi.org/10.1007/s00365-006-0663-2

TI - On Early Stopping in Gradient Descent Learning

T2 - Constructive Approximation

AU - Yao, Yuan

AU - Rosasco, Lorenzo

AU - CAPONNETTO, ANDREA

PY - 2007

DA - 2007/04/04

PB - Springer Nature

SP - 289-315

IS - 2

VL - 26

SN - 0176-4276

SN - 1432-0940

ER -

BibTex |

Cite this

BibTex (up to 50 authors) Copy

@article{2007_Yao,

author = {Yuan Yao and Lorenzo Rosasco and ANDREA CAPONNETTO},

title = {On Early Stopping in Gradient Descent Learning},

journal = {Constructive Approximation},

year = {2007},

volume = {26},

publisher = {Springer Nature},

month = {apr},

url = {https://doi.org/10.1007/s00365-006-0663-2},

number = {2},

pages = {289--315},

doi = {10.1007/s00365-006-0663-2}

}

MLA

Cite this

MLA Copy

Yao, Yuan, et al. “On Early Stopping in Gradient Descent Learning.” Constructive Approximation, vol. 26, no. 2, Apr. 2007, pp. 289-315. https://doi.org/10.1007/s00365-006-0663-2.

Publisher

Springer Nature

Journal

Constructive Approximation

scimago Q1

wos Q1

SJR

1.871

CiteScore

5.0

Impact factor

1.2

ISSN

01764276 (Print)

14320940 (Electronic)