  Humans are well-versed in reasoning about the behaviors of physical objects
and choosing actions accordingly to accomplish tasks, while it remains a major
challenge for AI. To facilitate research addressing this problem, we propose a
new testbed that requires an agent to reason about physical scenarios and take
an action appropriately. Inspired by the physical knowledge acquired in infancy
and the capabilities required for robots to operate in real-world environments,
we identify 15 essential physical scenarios. We create a wide variety of
distinct task templates, and we ensure all the task templates within the same
scenario can be solved by using one specific strategic physical rule. By having
such a design, we evaluate two distinct levels of generalization, namely the
local generalization and the broad generalization. We conduct an extensive
evaluation with human players, learning agents with varying input types and
architectures, and heuristic agents with different strategies. Inspired by how
human IQ is calculated, we define the physical reasoning quotient (Phy-Q score)
that reflects the physical reasoning intelligence of an agent using the
physical scenarios we considered. Our evaluation shows that 1) all agents are
far below human performance, and 2) learning agents, even with good local
generalization ability, struggle to learn the underlying physical reasoning
rules and fail to generalize broadly. We encourage the development of
intelligent agents that can reach the human level Phy-Q score. Website:
https://github.com/phy-q/benchmark
