  Context: Internet of Things (IoT) has become an important kind of distributed
systems thanks to the wide-spread of cheap embedded devices equipped with
different networking technologies. Although ubiquitous, developing IoT systems
remains challenging.
  Inquiry: A recent field study with 194 IoT developers identifies debugging as
one of the main challenges faced when developing IoT systems. This comes from
the lack of debugging tools taking into account the unique properties of IoT
systems such as non-deterministic data, and hardware restricted devices. On the
one hand, offline debuggers allow developers to analyse post-failure recorded
program information, but impose too much overhead on the devices while
generating such information.
  Furthermore, the analysis process is also time-consuming and might miss
contextual information relevant to find the root cause of bugs. On the other
hand, online debuggers do allow debugging a program upon a failure while
providing contextual information (e.g., stack trace). In particular, remote
online debuggers enable debugging of devices without physical access to them.
However, they experience debugging interference due to network delays which
complicates bug reproducibility, and have limited support for dynamic software
updates on remote devices.
  Approach: This paper proposes out-of-things debugging, an online debugging
approach especially designed for IoT systems. The debugger is always-on as it
ensures constant availability to for instance debug post-deployment situations.
Upon a failure or breakpoint, out-of-things debugging moves the state of a
deployed application to the developer's machine. Developers can then debug the
application locally by applying operations (e.g., step commands) to the
retrieved state. Once debugging is finished, developers can commit bug fixes to
the device through live update capabilities. Finally, by means of a
fine-grained flexible interface for accessing remote resources, developers have
full control over the debugging overhead imposed on the device, and the access
to device hardware resources (e.g., sensors) needed during local debugging.
  Knowledge: Out-of-things debugging maintains good properties of remote
debugging as it does not require physical access to the device to debug it,
while reducing debugging interference since there are no network delays on
operations (e.g., stepping) issued on the debugger since those happen locally.
Furthermore, device resources are only accessed when requested by the user
which further mitigates overhead and opens avenues for mocking or simulation of
non-accessed resources.
  Grounding: We implemented an out-of-things debugger as an extension to a
WebAssembly Virtual Machine and benchmarked its suitability for IoT. In
particular, we compared our solution to remote debugging alternatives based on
metrics such as network overhead, memory usage, scalability, and usability in
production settings. From the benchmarks, we conclude that our debugger
exhibits competitive performance in addition to confining overhead without
sacrificing debugging convenience and flexibility.
  Importance: Out-of-things debugging enables debugging of IoT systems by means
of classical online operations (e.g., stepwise execution) while addressing
IoT-specific concerns (e.g., hardware limitations). We show that having the
debugger always-on does not have to come at cost of performance loss or
increased overhead but instead can enforce a smooth-going and flexible
debugging experience of IoT systems.
