  Much has been said about the fusion of bio-inspired optimization algorithms
and Deep Learning models for several purposes: from the discovery of network
topologies and hyper-parametric configurations with improved performance for a
given task, to the optimization of the model's parameters as a replacement for
gradient-based solvers. Indeed, the literature is rich in proposals showcasing
the application of assorted nature-inspired approaches for these tasks. In this
work we comprehensively review and critically examine contributions made so far
based on three axes, each addressing a fundamental question in this research
avenue: a) optimization and taxonomy (Why?), including a historical
perspective, definitions of optimization problems in Deep Learning, and a
taxonomy associated with an in-depth analysis of the literature, b) critical
methodological analysis (How?), which together with two case studies, allows us
to address learned lessons and recommendations for good practices following the
analysis of the literature, and c) challenges and new directions of research
(What can be done, and what for?). In summary, three axes - optimization and
taxonomy, critical analysis, and challenges - which outline a complete vision
of a merger of two technologies drawing up an exciting future for this area of
fusion research.
