  In the rapidly evolving digital content landscape, media firms and news
publishers require automated and efficient methods to enhance user engagement.
This paper introduces the LLM-Assisted Online Learning Algorithm (LOLA), a
novel framework that integrates Large Language Models (LLMs) with adaptive
experimentation to optimize content delivery. Leveraging a large-scale dataset
from Upworthy, which includes 17,681 headline A/B tests, we first investigate
three pure-LLM approaches: prompt-based methods, embedding-based classification
models, and fine-tuned open-source LLMs. We find that prompt-based approaches
perform poorly, achieving no more than 65\% accuracy in identifying the
catchier headline. In contrast, both OpenAI-embedding-based classification
models and fine-tuned Llama-3 with 8 billion parameters achieve an accuracy of
around 82-84\%. We then introduce LOLA, which combines the best pure-LLM
approach with the Upper Confidence Bound algorithm to allocate traffic and
maximize clicks adaptively. Our numerical experiments on Upworthy data show
that LOLA outperforms the standard A/B test method (the current status quo at
Upworthy), pure bandit algorithms, and pure-LLM approaches, particularly in
scenarios with limited experimental traffic. Our approach is scalable and
applicable to content experiments across various settings where firms seek to
optimize user engagement, including digital advertising and social media
recommendations.
