  Functionally constrained stochastic optimization problems, where neither the
objective function nor the constraint functions are analytically available,
arise frequently in machine learning applications. In this work, assuming we
only have access to the noisy evaluations of the objective and constraint
functions, we propose and analyze stochastic zeroth-order algorithms for
solving the above class of stochastic optimization problem. When the domain of
the functions is $\mathbb{R}^n$, assuming there are $m$ constraint functions,
we establish oracle complexities of order $\mathcal{O}((m+1)n/\epsilon^2)$ and
$\mathcal{O}((m+1)n/\epsilon^3)$ respectively in the convex and nonconvex
setting, where $\epsilon$ represents the accuracy of the solutions required in
appropriately defined metrics. The established oracle complexities are, to our
knowledge, the first such results in the literature for functionally
constrained stochastic zeroth-order optimization problems. We demonstrate the
applicability of our algorithms by illustrating its superior performance on the
problem of hyperparameter tuning for sampling algorithms and neural network
training.
