  Voice conversion (VC) research traditionally depends on scripted or acted
speech, which lacks the natural spontaneity of real-life conversations. While
natural speech data is limited for VC, our study focuses on filling in this
gap. We introduce a novel data-sourcing pipeline that makes the release of a
natural speech dataset for VC, named NaturalVoices. The pipeline extracts rich
information in speech such as emotion and signal-to-noise ratio (SNR) from raw
podcast data, utilizing recent deep learning methods and providing flexibility
and ease of use. NaturalVoices marks a large-scale, spontaneous, expressive,
and emotional speech dataset, comprising over 3,800 hours speech sourced from
the original podcasts in the MSP-Podcast dataset. Objective and subjective
evaluations demonstrate the effectiveness of using our pipeline for providing
natural and expressive data for VC, suggesting the potential of NaturalVoices
for broader speech generation tasks.
