  The Internet of Things (IoT) network integrating billions of smart physical
devices embedded with sensors, software, and communication technologies is a
critical and rapidly expanding component of our modern world. The IoT ecosystem
provides a rich source of real-world modalities such as motion, thermal,
geolocation, imaging, depth, sensors, and audio to recognize the states of
humans and physical objects. Machine learning presents a rich opportunity to
automatically process IoT data at scale, enabling efficient inference for
understanding human wellbeing, controlling physical devices, and
interconnecting smart cities. To realize this potential, we introduce IoT-LM,
an open-source large multisensory language model tailored for the IoT
ecosystem. IoT-LM is enabled by two technical contributions: the first is
MultiIoT, the most expansive unified IoT dataset to date, encompassing over
1.15 million samples from 12 modalities and 8 tasks prepared for multisensory
pre-training and instruction-tuning. The second is a new multisensory multitask
adapter layer to condition pre-trained large language models on multisensory
IoT data. Not only does IoT-LM yield substantial improvements on 8 supervised
IoT classification tasks, but it also demonstrates new interactive
question-answering, reasoning, and dialog capabilities conditioned on IoT
sensors. We release IoT-LM's data sources and new multisensory language
modeling framework.
