  Recent video generative models primarily rely on carefully written text
prompts for specific tasks, like inpainting or style editing. They require
labor-intensive textual descriptions for input videos, hindering their
flexibility to adapt personal/raw videos to user specifications. This paper
proposes RACCooN, a versatile and user-friendly video-to-paragraph-to-video
generative framework that supports multiple video editing capabilities such as
removal, addition, and modification, through a unified pipeline. RACCooN
consists of two principal stages: Video-to-Paragraph (V2P) and
Paragraph-to-Video (P2V). In the V2P stage, we automatically describe video
scenes in well-structured natural language, capturing both the holistic context
and focused object details. Subsequently, in the P2V stage, users can
optionally refine these descriptions to guide the video diffusion model,
enabling various modifications to the input video, such as removing, changing
subjects, and/or adding new objects. The proposed approach stands out from
other methods through several significant contributions: (1) RACCooN suggests a
multi-granular spatiotemporal pooling strategy to generate well-structured
video descriptions, capturing both the broad context and object details without
requiring complex human annotations, simplifying precise video content editing
based on text for users. (2) Our video generative model incorporates
auto-generated narratives or instructions to enhance the quality and accuracy
of the generated content. It supports the addition of video objects,
inpainting, and attribute modification within a unified framework, surpassing
existing video editing and inpainting benchmarks. The proposed framework
demonstrates impressive versatile capabilities in video-to-paragraph
generation, video content editing, and can be incorporated into other SoTA
video generative models for further enhancement.
