  Acoustic-to-articulatory inversion (AAI) is to convert audio into articulator
movements, such as ultrasound tongue imaging (UTI) data. An issue of existing
AAI methods is only using the personalized acoustic information to derive the
general patterns of tongue motions, and thus the quality of generated UTI data
is limited. To address this issue, this paper proposes an audio-textual
diffusion model for the UTI data generation task. In this model, the inherent
acoustic characteristics of individuals related to the tongue motion details
are encoded by using wav2vec 2.0, while the ASR transcriptions related to the
universality of tongue motions are encoded by using BERT. UTI data are then
generated by using a diffusion module. Experimental results showed that the
proposed diffusion model could generate high-quality UTI data with clear tongue
contour that is crucial for the linguistic analysis and clinical assessment.
The project can be found on the
website\footnote{https://yangyudong2020.github.io/wav2uti/
