  With their exceptional natural language processing capabilities, tools based
on Large Language Models (LLMs) like ChatGPT and Co-Pilot have swiftly become
indispensable resources in the software developer's toolkit. While recent
studies suggest the potential productivity gains these tools can unlock, users
still encounter drawbacks, such as generic or incorrect answers. Additionally,
the pursuit of improved responses often leads to extensive prompt engineering
efforts, diverting valuable time from writing code that delivers actual value.
To address these challenges, a new breed of tools, built atop LLMs, is
emerging. These tools aim to mitigate drawbacks by employing techniques like
fine-tuning or enriching user prompts with contextualized information.
  In this paper, we delve into the lessons learned by a software development
team venturing into the creation of such a contextualized LLM-based
application, using retrieval-based techniques, called CodeBuddy. Over a
four-month period, the team, despite lacking prior professional experience in
LLM-based applications, built the product from scratch. Following the initial
product release, we engaged with the development team responsible for the code
generative components. Through interviews and analysis of the application's
issue tracker, we uncover various intriguing challenges that teams working on
LLM-based applications might encounter. For instance, we found three main group
of lessons: LLM-based lessons, User-based lessons, and Technical lessons. By
understanding these lessons, software development teams could become better
prepared to build LLM-based applications.
