  Seamless interaction between AI agents and humans using natural language
remains a key goal in AI research. This paper addresses the challenges of
developing interactive agents capable of understanding and executing grounded
natural language instructions through the IGLU competition at NeurIPS. Despite
advancements, challenges such as a scarcity of appropriate datasets and the
need for effective evaluation platforms persist. We introduce a scalable data
collection tool for gathering interactive grounded language instructions within
a Minecraft-like environment, resulting in a Multi-Modal dataset with around
9,000 utterances and over 1,000 clarification questions. Additionally, we
present a Human-in-the-Loop interactive evaluation platform for qualitative
analysis and comparison of agent performance through multi-turn communication
with human annotators. We offer to the community these assets referred to as
IDAT (IGLU Dataset And Toolkit) which aim to advance the development of
intelligent, interactive AI agents and provide essential resources for further
research.
