Incremental Cubic Regularized SR1 Quasi-Newton Method and the Applications in Large-Scale Problems

Abstract

This paper delves into an innovative incremental cubic regularized symmetric rank-1 (SR1) method (ICuREGSR1). By incorporating the cubic regularization technique into SR1, we successfully address the issue of indefinite resulting matrix in SR1. Our core strategy is to adopt an incremental optimization scheme, gradually updating the information of the objective function, which typically involves a sum of multiple independent functions, and is very common in large-scale machine learning tasks. Through numerical experiments on multiple machine learning problems, we find that compared with other traditional algorithms, our proposed algorithm exhibits superior performance in terms of gradient magnitude.