  Multi-agent simulation is commonly used across multiple disciplines,
specifically in artificial intelligence in recent years, which creates an
environment for downstream machine learning or reinforcement learning tasks. In
many practical scenarios, however, only the output series that result from the
interactions of simulation agents are observable. Therefore, simulators need to
be calibrated so that the simulated output series resemble historical -- which
amounts to solving a complex simulation optimization problem. In this paper, we
propose a simple and efficient framework for calibrating simulator parameters
from historical output series observations. First, we consider a novel concept
of eligibility set to bypass the potential non-identifiability issue. Second,
we generalize the two-sample Kolmogorov-Smirnov (K-S) test with Bonferroni
correction to test the similarity between two high-dimensional distributions,
which gives a simple yet effective distance metric between the output series
sample sets. Third, we suggest using Bayesian optimization (BO) and
trust-region BO (TuRBO) to minimize the aforementioned distance metric.
Finally, we demonstrate the efficiency of our framework using numerical
experiments both on a multi-agent financial market simulator.
