  In recent years, the field of computer vision has seen significant
advancements thanks to the development of large language models (LLMs). These
models have enabled more effective and sophisticated interactions between
humans and machines, paving the way for novel techniques that blur the lines
between human and machine intelligence. In this paper, we introduce a new
paradigm for object detection that we call reasoning-based object detection.
Unlike conventional object detection methods that rely on specific object
names, our approach enables users to interact with the system using natural
language instructions, allowing for a higher level of interactivity. Our
proposed method, called DetGPT, leverages state-of-the-art multi-modal models
and open-vocabulary object detectors to perform reasoning within the context of
the user's instructions and the visual scene. This enables DetGPT to
automatically locate the object of interest based on the user's expressed
desires, even if the object is not explicitly mentioned. For instance, if a
user expresses a desire for a cold beverage, DetGPT can analyze the image,
identify a fridge, and use its knowledge of typical fridge contents to locate
the beverage. This flexibility makes our system applicable across a wide range
of fields, from robotics and automation to autonomous driving. Overall, our
proposed paradigm and DetGPT demonstrate the potential for more sophisticated
and intuitive interactions between humans and machines. We hope that our
proposed paradigm and approach will provide inspiration to the community and
open the door to more interative and versatile object detection systems. Our
project page is launched at detgpt.github.io.
