  We introduce a cooperative Bayesian optimization problem for optimizing
black-box functions of two variables where two agents choose together at which
points to query the function but have only control over one variable each. This
setting is inspired by human-AI teamwork, where an AI-assistant helps its human
user solve a problem, in this simplest case, collaborative optimization. We
formulate the solution as sequential decision-making, where the agent we
control models the user as a computationally rational agent with prior
knowledge about the function. We show that strategic planning of the queries
enables better identification of the global maximum of the function as long as
the user avoids excessive exploration. This planning is made possible by using
Bayes Adaptive Monte Carlo planning and by endowing the agent with a user model
that accounts for conservative belief updates and exploratory sampling of the
points to query.
