  The cloud-based speech recognition/API provides developers or enterprises an
easy way to create speech-enabled features in their applications. However,
sending audios about personal or company internal information to the cloud,
raises concerns about the privacy and security issues. The recognition results
generated in cloud may also reveal some sensitive information. This paper
proposes a deep polynomial network (DPN) that can be applied to the encrypted
speech as an acoustic model. It allows clients to send their data in an
encrypted form to the cloud to ensure that their data remains confidential, at
mean while the DPN can still make frame-level predictions over the encrypted
speech and return them in encrypted form. One good property of the DPN is that
it can be trained on unencrypted speech features in the traditional way. To
keep the cloud away from the raw audio and recognition results, a cloud-local
joint decoding framework is also proposed. We demonstrate the effectiveness of
model and framework on the Switchboard and Cortana voice assistant tasks with
small performance degradation and latency increased comparing with the
traditional cloud-based DNNs.
