  In traditional audio captioning methods, a model is usually trained in a
fully supervised manner using a human-annotated dataset containing audio-text
pairs and then evaluated on the test sets from the same dataset. Such methods
have two limitations. First, these methods are often data-hungry and require
time-consuming and expensive human annotations to obtain audio-text pairs.
Second, these models often suffer from performance degradation in cross-domain
scenarios, i.e., when the input audio comes from a different domain than the
training set, which, however, has received little attention. We propose an
effective audio captioning method based on the contrastive language-audio
pre-training (CLAP) model to address these issues. Our proposed method requires
only textual data for training, enabling the model to generate text from the
textual feature in the cross-modal semantic space.In the inference stage, the
model generates the descriptive text for the given audio from the audio feature
by leveraging the audio-text alignment from CLAP.We devise two strategies to
mitigate the discrepancy between text and audio embeddings: a
mixed-augmentation-based soft prompt and a retrieval-based acoustic-aware hard
prompt. These approaches are designed to enhance the generalization performance
of our proposed model, facilitating the model to generate captions more
robustly and accurately. Extensive experiments on AudioCaps and Clotho
benchmarks show the effectiveness of our proposed method, which outperforms
other zero-shot audio captioning approaches for in-domain scenarios and
outperforms the compared methods for cross-domain scenarios, underscoring the
generalization ability of our method.
