  We investigate unsupervised learning of correspondences between sound events
and textual phrases through aligning audio clips with textual captions
describing the content of a whole audio clip. We align originally unaligned and
unannotated audio clips and their captions by scoring the similarities between
audio frames and words, as encoded by modality-specific encoders and using a
ranking-loss criterion to optimize the model. After training, we obtain
clip-caption similarity by averaging frame-word similarities and estimate
event-phrase correspondences by calculating frame-phrase similarities. We
evaluate the method with two cross-modal tasks: audio-caption retrieval, and
phrase-based sound event detection (SED). Experimental results show that the
proposed method can globally associate audio clips with captions as well as
locally learn correspondences between individual sound events and textual
phrases in an unsupervised manner.
