  Machine recognition of an atypical speech like whispered speech, is a
challenging task. We introduce whisper-to-natural-speech conversion using
sequence-to-sequence approach by proposing enhanced transformer architecture,
which uses both parallel and non-parallel data. We investigate different
features like Mel frequency cepstral coefficients and smoothed spectral
features. The proposed networks are trained end-to-end using supervised
approach for feature-to-feature transformation. Further, we also investigate
the effectiveness of embedded auxillary decoder used after N encoder
sub-layers, trained with the frame-level objective function for identifying
source phoneme labels. We show results on opensource wTIMIT and CHAINS datasets
by measuring word error rate using end-to-end ASR and also BLEU scores for the
generated speech. Alternatively, we also propose a novel method to measure
spectral shape of it by measuring formant distributions w.r.t. reference
speech, as formant divergence metric. We have found whisper-to-natural
converted speech formants probability distribution is similar to the
groundtruth distribution. To the authors' best knowledge, this is the first
time enhanced transformer has been proposed, both with and without auxiliary
decoder for whisper-to-natural-speech conversion and vice versa.
