  Detecting and responding to novel situations in open-world environments is a
key capability of human cognition and is a persistent problem for AI systems.
In an open-world, novelties can appear in many different forms and may be easy
or hard to detect. Therefore, to accurately evaluate the novelty detection
capability of AI systems, it is necessary to investigate how difficult it may
be to detect different types of novelty. In this paper, we propose a
qualitative physics-based method to quantify the difficulty of novelty
detection focusing on open-world physical domains. We apply our method in the
popular physics simulation game Angry Birds, and conduct a user study across
different novelties to validate our method. Results indicate that our
calculated detection difficulties are in line with those of human users.
