  The automated machine learning (AutoML) process can require searching through
complex configuration spaces of not only machine learning (ML) components and
their hyperparameters but also ways of composing them together, i.e. forming ML
pipelines. Optimisation efficiency and the model accuracy attainable for a
fixed time budget suffer if this pipeline configuration space is excessively
large. A key research question is whether it is both possible and practical to
preemptively avoid costly evaluations of poorly performing ML pipelines by
leveraging their historical performance for various ML tasks, i.e.
meta-knowledge. The previous experience comes in the form of
classifier/regressor accuracy rankings derived from either (1) a substantial
but non-exhaustive number of pipeline evaluations made during historical AutoML
runs, i.e. 'opportunistic' meta-knowledge, or (2) comprehensive cross-validated
evaluations of classifiers/regressors with default hyperparameters, i.e.
'systematic' meta-knowledge. Numerous experiments with the AutoWeka4MCPS
package suggest that (1) opportunistic/systematic meta-knowledge can improve ML
outcomes, typically in line with how relevant that meta-knowledge is, and (2)
configuration-space culling is optimal when it is neither too conservative nor
too radical. However, the utility and impact of meta-knowledge depend
critically on numerous facets of its generation and exploitation, warranting
extensive analysis; these are often overlooked/underappreciated within AutoML
and meta-learning literature. In particular, we observe strong sensitivity to
the `challenge' of a dataset, i.e. whether specificity in choosing a predictor
leads to significantly better performance. Ultimately, identifying `difficult'
datasets, thus defined, is crucial to both generating informative
meta-knowledge bases and understanding optimal search-space reduction
strategies.
