  We present a novel framework, Localized Image Stylization with Audio (LISA)
which performs audio-driven localized image stylization. Sound often provides
information about the specific context of the scene and is closely related to a
certain part of the scene or object. However, existing image stylization works
have focused on stylizing the entire image using an image or text input.
Stylizing a particular part of the image based on audio input is natural but
challenging. In this work, we propose a framework that a user provides an audio
input to localize the sound source in the input image and another for locally
stylizing the target object or scene. LISA first produces a delicate
localization map with an audio-visual localization network by leveraging CLIP
embedding space. We then utilize implicit neural representation (INR) along
with the predicted localization map to stylize the target object or scene based
on sound information. The proposed INR can manipulate the localized pixel
values to be semantically consistent with the provided audio input. Through a
series of experiments, we show that the proposed framework outperforms the
other audio-guided stylization methods. Moreover, LISA constructs concise
localization maps and naturally manipulates the target object or scene in
accordance with the given audio input.
