  Bayesian optimization (BO) is a powerful approach for optimizing complex and
expensive-to-evaluate black-box functions. Its importance is underscored in
many applications, notably including hyperparameter tuning, but its efficacy
depends on efficiently balancing exploration and exploitation. While there has
been substantial progress in BO methods, striking this balance remains a
delicate process. In this light, we present LLAMBO, a novel approach that
integrates the capabilities of Large Language Models (LLM) within BO. At a high
level, we frame the BO problem in natural language, enabling LLMs to
iteratively propose and evaluate promising solutions conditioned on historical
evaluations. More specifically, we explore how combining contextual
understanding, few-shot learning proficiency, and domain knowledge of LLMs can
improve model-based BO. Our findings illustrate that LLAMBO is effective at
zero-shot warmstarting, and enhances surrogate modeling and candidate sampling,
especially in the early stages of search when observations are sparse. Our
approach is performed in context and does not require LLM finetuning.
Additionally, it is modular by design, allowing individual components to be
integrated into existing BO frameworks, or function cohesively as an end-to-end
method. We empirically validate LLAMBO's efficacy on the problem of
hyperparameter tuning, highlighting strong empirical performance across a range
of diverse benchmarks, proprietary, and synthetic tasks.
