  Users can discuss a wide range of topics with large language models (LLMs),
but they do not always prefer solving problems or getting information through
lengthy conversations. This raises an intriguing HCI question: How does
instructing LLMs to engage in longer or shorter conversations affect
conversation quality? In this paper, we developed two Slack chatbots using
GPT-4 with the ability to vary conversation lengths and conducted a user study.
Participants asked the chatbots both highly and less conversable questions,
engaging in dialogues with 0, 3, 5, and 7 conversational turns. We found that
the conversation quality does not differ drastically across different
conditions, while participants had mixed reactions. Our study demonstrates
LLMs' ability to change conversation length and the potential benefits for
users resulting from such changes, but we caution that changes in text form may
not necessarily imply changes in quality or content.
