  One of the fundamental cognitive abilities of humans is to quickly resolve
uncertainty by generating hypotheses and testing them via active trials.
Encountering a novel phenomenon accompanied by ambiguous cause-effect
relationships, humans make hypotheses against data, conduct inferences from
observation, test their theory via experimentation, and correct the proposition
if inconsistency arises. These iterative processes persist until the underlying
mechanism becomes clear. In this work, we devise the IVRE (pronounced as
"ivory") environment for evaluating artificial agents' reasoning ability under
uncertainty. IVRE is an interactive environment featuring rich scenarios
centered around Blicket detection. Agents in IVRE are placed into environments
with various ambiguous action-effect pairs and asked to determine each object's
role. They are encouraged to propose effective and efficient experiments to
validate their hypotheses based on observations and actively gather new
information. The game ends when all uncertainties are resolved or the maximum
number of trials is consumed. By evaluating modern artificial agents in IVRE,
we notice a clear failure of today's learning methods compared to humans. Such
inefficacy in interactive reasoning ability under uncertainty calls for future
research in building human-like intelligence.
