  Large language models (LLMs) provide a new way to build chatbots by accepting
natural language prompts. Yet, it is unclear how to design prompts to power
chatbots to carry on naturalistic conversations while pursuing a given goal,
such as collecting self-report data from users. We explore what design factors
of prompts can help steer chatbots to talk naturally and collect data reliably.
To this aim, we formulated four prompt designs with different structures and
personas. Through an online study (N = 48) where participants conversed with
chatbots driven by different designs of prompts, we assessed how prompt designs
and conversation topics affected the conversation flows and users' perceptions
of chatbots. Our chatbots covered 79% of the desired information slots during
conversations, and the designs of prompts and topics significantly influenced
the conversation flows and the data collection performance. We discuss the
opportunities and challenges of building chatbots with LLMs.
