A guide to neural ordinary differential equations: Machine learning for data-driven digital engineering

Advances in deep learning have impacted all areas of business, government and academia, and deep learning is expanding into domains that are beyond the scope of the standard Computer Science curriculum. One such domain is that of dynamical systems, optimal estimation and control which are studied in the neighboring disciplines of Engineering and Physics. A new focus on deep dynamical learning has emerged at this intersection and has begun to produce effective solutions which rely heavily on techniques from both Engineering and Computer Science. At the core of many of these solutions is the neural ordinary differential equation (ODE) which can be applied to learn the evolution of a system that is continuous in time. While this model can be viewed as analogous to the residual network, the implementation and application can seem out-of-reach due to patterns that are uncommon in other deep learning models. This paper offers a deep learning perspective on neural ODEs, explores a novel derivation of backpropagation with the adjoint sensitivity method, outlines design patterns for use and provides a survey on state-of-the-art research in neural ODEs. Oftentimes, those who are training neural networks do not possess the background needed to understand the adjoint gradient thus limiting their ability to fine-tune neural ODEs. This novel derivation of the adjoint gradient does not rely on this background allowing computer scientists and engineers to better understand the adjoint and make informed decisions when training. Several toy dynamical systems are used as examples throughout this paper. The guide contained within is intended to help experienced deep learning practitioners understand neural ODEs when working with continuous-time problems.