  Large Language Models (LLMs) are shifting how scientific research is done. It
is imperative to understand how researchers interact with these models and how
scientific sub-communities like astronomy might benefit from them. However,
there is currently no standard for evaluating the use of LLMs in astronomy.
Therefore, we present the experimental design for an evaluation study on how
astronomy researchers interact with LLMs. We deploy a Slack chatbot that can
answer queries from users via Retrieval-Augmented Generation (RAG); these
responses are grounded in astronomy papers from arXiv. We record and anonymize
user questions and chatbot answers, user upvotes and downvotes to LLM
responses, user feedback to the LLM, and retrieved documents and similarity
scores with the query. Our data collection method will enable future dynamic
evaluations of LLM tools for astronomy.
