  Powerful artificial intelligence systems are often used in settings where
they must interact with agents that are computationally much weaker, for
example when they work alongside humans or operate in complex environments
where some tasks are handled by algorithms, heuristics, or other entities of
varying computational power. For AI agents to successfully interact in these
settings, however, achieving superhuman performance alone is not sufficient;
they also need to account for suboptimal actions or idiosyncratic style from
their less-skilled counterparts. We propose a formal evaluation framework for
assessing the compatibility of near-optimal AI with interaction partners who
may have much lower levels of skill; we use popular collaborative chess
variants as model systems to study and develop AI agents that can successfully
interact with lower-skill entities. Traditional chess engines designed to
output near-optimal moves prove to be inadequate partners when paired with
engines of various lower skill levels in this domain, as they are not designed
to consider the presence of other agents. We contribute three methodologies to
explicitly create skill-compatible AI agents in complex decision-making
settings, and two chess game frameworks designed to foster collaboration
between powerful AI agents and less-skilled partners. On these frameworks, our
agents outperform state-of-the-art chess AI (based on AlphaZero) despite being
weaker in conventional chess, demonstrating that skill-compatibility is a
tangible trait that is qualitatively and measurably distinct from raw
performance. Our evaluations further explore and clarify the mechanisms by
which our agents achieve skill-compatibility.
