A comprehensive survey of fractional gradient descent methods and their convergence analysis

Fractional Gradient Descent (FGD) methods extend classical optimization algorithms by integrating fractional calculus, leading to notable improvements in convergence speed, stability, and accuracy. However, recent studies indicate that engineering challenges—such as tensor-based differentiation in deep neural networks—remain partially unresolved, prompting further investigation into the scalability and computational feasibility of FGD. This paper provides a comprehensive review of recent advancements in FGD techniques, focusing on their approximation methods and convergence properties. These methods are systematically categorized based on their strategies to overcome convergence challenges inherent in fractional-order calculations, such as non-locality and long-memory effects. Key techniques examined include modified fractional-order gradients designed to avoid singularities and ensure convergence to the true extremum. Adaptive step-size strategies and variable fractional-order schemes are analyzed, balancing rapid convergence with precise parameter estimation. Additionally, the application of truncation methods is explored to mitigate oscillatory behavior associated with fractional derivatives. By synthesizing convergence analyses from multiple studies, insights are offered into the theoretical foundations of these methods, including proofs of linear convergence. Ultimately, this paper highlights the effectiveness of various FGD approaches in accelerating convergence and enhancing stability. While also acknowledging significant gaps in practical implementations for large-scale engineering tasks, including deep learning. The presented review serves as a resource for researchers and practitioners in the selection of appropriate FGD techniques for different optimization problems.