  As machine learning (ML) is more tightly woven into society, it is imperative
that we better characterize ML's strengths and limitations if we are to employ
it responsibly. Existing benchmark environments for ML, such as board and video
games, offer well-defined benchmarks for progress, but constituent tasks are
often complex, and it is frequently unclear how task characteristics contribute
to overall difficulty for the machine learner. Likewise, without a systematic
assessment of how task characteristics influence difficulty, it is challenging
to draw meaningful connections between performance in different benchmark
environments. We introduce a novel benchmark environment that offers an
enormous range of ML challenges and enables precise examination of how task
elements influence practical difficulty. The tool frames learning tasks as a
"board-clearing game," which we call the Game of Hidden Rules (GOHR). The
environment comprises an expressive rule language and a captive server
environment that can be installed locally. We propose a set of benchmark
rule-learning tasks and plan to support a performance leader-board for
researchers interested in attempting to learn our rules. GOHR complements
existing environments by allowing fine, controlled modifications to tasks,
enabling experimenters to better understand how each facet of a given learning
task contributes to its practical difficulty for an arbitrary ML algorithm.
