  Visual storytelling is an emerging field that combines images and narratives
to create engaging and contextually rich stories. Despite its potential,
generating coherent and emotionally resonant visual stories remains challenging
due to the complexity of aligning visual and textual information. This paper
presents a novel approach leveraging large language models (LLMs) and large
vision-language models (LVLMs) combined with instruction tuning to address
these challenges. We introduce a new dataset comprising diverse visual stories,
annotated with detailed captions and multimodal elements. Our method employs a
combination of supervised and reinforcement learning to fine-tune the model,
enhancing its narrative generation capabilities. Quantitative evaluations using
GPT-4 and qualitative human assessments demonstrate that our approach
significantly outperforms existing models, achieving higher scores in narrative
coherence, relevance, emotional depth, and overall quality. The results
underscore the effectiveness of instruction tuning and the potential of
LLMs/LVLMs in advancing visual storytelling.
