  While deep neural networks have achieved state-of-the-art performance across
a large number of complex tasks, it remains a big challenge to deploy such
networks for practical, on-device edge scenarios such as on mobile devices,
consumer devices, drones, and vehicles. In this study, we take a deeper
exploration into a human-machine collaborative design approach for creating
highly efficient deep neural networks through a synergy between principled
network design prototyping and machine-driven design exploration. The efficacy
of human-machine collaborative design is demonstrated through the creation of
AttoNets, a family of highly efficient deep neural networks for on-device edge
deep learning. Each AttoNet possesses a human-specified network-level
macro-architecture comprising of custom modules with unique machine-designed
module-level macro-architecture and micro-architecture designs, all driven by
human-specified design requirements. Experimental results for the task of
object recognition showed that the AttoNets created via human-machine
collaborative design has significantly fewer parameters and computational costs
than state-of-the-art networks designed for efficiency while achieving
noticeably higher accuracy (with the smallest AttoNet achieving ~1.8% higher
accuracy while requiring ~10x fewer multiply-add operations and parameters than
MobileNet-V1). Furthermore, the efficacy of the AttoNets is demonstrated for
the task of instance-level object segmentation and object detection, where an
AttoNet-based Mask R-CNN network was constructed with significantly fewer
parameters and computational costs (~5x fewer multiply-add operations and ~2x
fewer parameters) than a ResNet-50 based Mask R-CNN network.
