  Scientific machine learning (SciML) is a relatively new field that aims to
solve problems from different fields of natural sciences using machine learning
tools. It is well-documented that the optimizers commonly used in other areas
of machine learning perform poorly on many SciML problems. We provide an
infinite-dimensional view on optimization problems encountered in scientific
machine learning and advocate for the paradigm first optimize, then discretize
for their solution. This amounts to first choosing an appropriate
infinite-dimensional algorithm which is then discretized in a second step. To
illustrate this point, we show that recently proposed state-of-the-art
algorithms for SciML applications can be derived within this framework. As the
infinite-dimensional viewpoint is presently underdeveloped in scientific
machine learning, we formalize it here and advocate for its use in SciML in the
development of efficient optimization algorithms.
