  Progressing beyond centralized AI is of paramount importance, yet,
distributed AI solutions, in particular various federated learning (FL)
algorithms, are often not comprehensively assessed, which prevents the research
community from identifying the most promising approaches and practitioners from
being convinced that a certain solution is deployment-ready. The largest hurdle
towards FL algorithm evaluation is the difficulty of conducting real-world
experiments over a variety of FL client devices and different platforms, with
different datasets and data distribution, all while assessing various
dimensions of algorithm performance, such as inference accuracy, energy
consumption, and time to convergence, to name a few. In this paper, we present
CoLExT, a real-world testbed for FL research. CoLExT is designed to streamline
experimentation with custom FL algorithms in a rich testbed configuration
space, with a large number of heterogeneous edge devices, ranging from
single-board computers to smartphones, and provides real-time collection and
visualization of a variety of metrics through automatic instrumentation.
According to our evaluation, porting FL algorithms to CoLExT requires minimal
involvement from the developer, and the instrumentation introduces minimal
resource usage overhead. Furthermore, through an initial investigation
involving popular FL algorithms running on CoLExT, we reveal previously unknown
trade-offs, inefficiencies, and programming bugs.
