  Challenging optimization problems, which elude acceptable solution via
conventional calculus methods, arise commonly in different areas of industrial
design and practice. Hard optimization problems are those who manifest the
following behavior: a) high number of independent input variables; b) very
complex or irregular multi-modal fitness; c) computational expensive fitness
evaluation. This paper will focus on some theoretical issues that have strong
implications for practice. I will stress how an interpretation of the No Free
Lunch theorem leads naturally to a general Bayesian optimization framework. The
choice of a prior over the space of functions is a critical and inevitable step
in every black-box optimization.
