  Neural vocoders model the raw audio waveform and synthesize high-quality
audio, but even the highly efficient ones, like MB-MelGAN and LPCNet, fail to
run real-time on a low-end device like a smartglass. A pure digital signal
processing (DSP) based vocoder can be implemented via lightweight fast Fourier
transforms (FFT), and therefore, is a magnitude faster than any neural vocoder.
A DSP vocoder often gets a lower audio quality due to consuming over-smoothed
acoustic model predictions of approximate representations for the vocal tract.
In this paper, we propose an ultra-lightweight differential DSP (DDSP) vocoder
that uses a jointly optimized acoustic model with a DSP vocoder, and learns
without an extracted spectral feature for the vocal tract. The model achieves
audio quality comparable to neural vocoders with a high average MOS of 4.36
while being efficient as a DSP vocoder. Our C++ implementation, without any
hardware-specific optimization, is at 15 MFLOPS, surpasses MB-MelGAN by 340
times in terms of FLOPS, and achieves a vocoder-only RTF of 0.003 and overall
RTF of 0.044 while running single-threaded on a 2GHz Intel Xeon CPU.
