  In this project, we demonstrate that phoneme-based models for speech
processing can achieve strong crosslinguistic generalizability to unseen
languages. We curated the IPAPACK, a massively multilingual speech corpora with
phonemic transcriptions, encompassing more than 115 languages from diverse
language families, selectively checked by linguists. Based on the IPAPACK, we
propose CLAP-IPA, a multi-lingual phoneme-speech contrastive embedding model
capable of open-vocabulary matching between arbitrary speech signals and
phonemic sequences. The proposed model was tested on 95 unseen languages,
showing strong generalizability across languages. Temporal alignments between
phonemes and speech signals also emerged from contrastive training, enabling
zeroshot forced alignment in unseen languages. We further introduced a neural
forced aligner IPA-ALIGNER by finetuning CLAP-IPA with the Forward-Sum loss to
learn better phone-to-audio alignment. Evaluation results suggest that
IPA-ALIGNER can generalize to unseen languages without adaptation.
