  Developing an operating system (OS) for low-end embedded devices requires
continuous adaptation to new hardware architectures and components, while
serviceability of features needs to be assured for each individual platform
under tight resource constraints. It is challenging to design a versatile and
accurate heterogeneous test environment that is agile enough to cover a
continuous evolution of the code base and platforms. This mission is even
morehallenging when organized in an agile open-source community process with
many contributors such as for the RIOT OS. Hardware in the Loop (HiL) testing
and Continuous Integration (CI) are automatable approaches to verify
functionality, prevent regressions, and improve the overall quality at
development speed in large community projects. In this paper, we present PHiLIP
(Primitive Hardware in the Loop Integration Product), an open-source external
reference device together with tools that validate the system software while it
controls hardware and interprets physical signals. Instead of focusing on a
specific test setting, PHiLIP takes the approach of a tool-assisted agile HiL
test process, designed for continuous evolution and deployment cycles. We
explain its design, describe how it supports HiL tests, evaluate performance
metrics, and report on practical experiences of employing PHiLIP in an
automated CI test infrastructure. Our initial deployment comprises 22 unique
platforms, each of which executes 98 peripheral tests every night. PHiLIP
allows for easy extension of low-cost, adaptive testing infrastructures but
serves testing techniques and tools to a much wider range of applications.
