  Machine learning is a prevalent approach to tame the complexity of design
space exploration for domain-specific architectures. Using ML for design space
exploration poses challenges. First, it's not straightforward to identify the
suitable algorithm from an increasing pool of ML methods. Second, assessing the
trade-offs between performance and sample efficiency across these methods is
inconclusive. Finally, lack of a holistic framework for fair, reproducible, and
objective comparison across these methods hinders progress of adopting ML-aided
architecture design space exploration and impedes creating repeatable
artifacts. To mitigate these challenges, we introduce ArchGym, an open-source
gym and easy-to-extend framework that connects diverse search algorithms to
architecture simulators. To demonstrate utility, we evaluate ArchGym across
multiple vanilla and domain-specific search algorithms in designing custom
memory controller, deep neural network accelerators, and custom SoC for AR/VR
workloads, encompassing over 21K experiments. Results suggest that with
unlimited samples, ML algorithms are equally favorable to meet user-defined
target specification if hyperparameters are tuned; no solution is necessarily
better than another (e.g., reinforcement learning vs. Bayesian methods). We
coin the term hyperparameter lottery to describe the chance for a search
algorithm to find an optimal design provided meticulously selected
hyperparameters. The ease of data collection and aggregation in ArchGym
facilitates research in ML-aided architecture design space exploration. As a
case study, we show this advantage by developing a proxy cost model with an
RMSE of 0.61% that offers a 2,000-fold reduction in simulation time. Code and
data for ArchGym is available at https://bit.ly/ArchGym.
