  The aim of this research is to refine knowledge transfer on audio-image
temporal agreement for audio-text cross retrieval. To address the limited
availability of paired non-speech audio-text data, learning methods for
transferring the knowledge acquired from a large amount of paired audio-image
data to shared audio-text representation have been investigated, suggesting the
importance of how audio-image co-occurrence is learned. Conventional approaches
in audio-image learning assign a single image randomly selected from the
corresponding video stream to the entire audio clip, assuming their
co-occurrence. However, this method may not accurately capture the temporal
agreement between the target audio and image because a single image can only
represent a snapshot of a scene, though the target audio changes from moment to
moment. To address this problem, we propose two methods for audio and image
matching that effectively capture the temporal information: (i) Nearest Match
wherein an image is selected from multiple time frames based on similarity with
audio, and (ii) Multiframe Match wherein audio and image pairs of multiple time
frames are used. Experimental results show that method (i) improves the
audio-text retrieval performance by selecting the nearest image that aligns
with the audio information and transferring the learned knowledge. Conversely,
method (ii) improves the performance of audio-image retrieval while not showing
significant improvements in audio-text retrieval performance. These results
indicate that refining audio-image temporal agreement may contribute to better
knowledge transfer to audio-text retrieval.
