  The deep neural network (DNN) based AI applications on the edge require both
low-cost computing platforms and high-quality services. However, the limited
memory, computing resources, and power budget of the edge devices constrain the
effectiveness of the DNN algorithms. Developing edge-oriented AI algorithms and
implementations (e.g., accelerators) is challenging. In this paper, we
summarize our recent efforts for efficient on-device AI development from three
aspects, including both training and inference. First, we present on-device
training with ultra-low memory usage. We propose a novel rank-adaptive
tensor-based tensorized neural network model, which offers orders-of-magnitude
memory reduction during training. Second, we introduce an ultra-low bitwidth
quantization method for DNN model compression, achieving the state-of-the-art
accuracy under the same compression ratio. Third, we introduce an ultra-low
latency DNN accelerator design, practicing the software/hardware co-design
methodology. This paper emphasizes the importance and efficacy of training,
quantization and accelerator design, and calls for more research breakthroughs
in the area for AI on the edge.
