  Distributing the inference of convolutional neural network (CNN) to multiple
mobile devices has been studied in recent years to achieve real-time inference
without losing accuracy. However, how to map CNN to devices remains a
challenge. On the one hand, scheduling the workload of state-of-the-art CNNs
with multiple devices is NP-Hard because the structures of CNNs are directed
acyclic graphs (DAG) rather than simple chains. On the other hand, distributing
the inference workload suffers from expensive communication and unbalanced
computation due to the wireless environment and heterogeneous devices. This
paper presents PICO, a pipeline cooperation framework to accelerate the
inference of versatile CNNs on diverse mobile devices. At its core, PICO
features: (1) a generic graph partition algorithm that considers the
characteristics of any given CNN and orchestrates it into a list of model
pieces with suitable granularity, and (2) a many-to-many mapping algorithm that
produces the best pipeline configuration for heterogeneous devices. In our
experiment with 2 ~ 8 Raspberry-Pi devices, the throughput can be improved by
1.8 ~ 6.8x under different CPU frequencies.
