  Low power deep learning accelerators on the speech processing enable
real-time applications on edge devices. However, most of the existing
accelerators suffer from high power consumption and focus on image applications
only. This paper presents a low power accelerator for speech separation through
algorithm and hardware optimizations. At the algorithm level, the model is
compressed with structured sensitivity as well as unstructured pruning, and
further quantized to the shifted 8-bit floating-point format instead of the
32-bit floating-point format. The computations with the zero kernel and zero
activation values are skipped by decomposition of the dilated and transposed
convolutions. At the hardware level, the compressed model is then supported by
an architecture with eight independent multipliers and accumulators (MACs) with
a simple zero-skipping hardware to take advantage of the activation sparsity
and low power processing. The proposed approach reduces the model size by
95.44\% and computation complexity by 93.88\%. The final implementation with
the TSMC 40 $nm$ process can achieve real-time speech separation and consumes
1.6 mW power when operated at 150 MHz. The normalized energy efficiency and
area efficiency are 2.344 TOPS/W and 14.42 GOPS/mm$^2$, respectively.
