  LLM-powered tools like ChatGPT Data Analysis, have the potential to help
users tackle the challenging task of data analysis programming, which requires
expertise in data processing, programming, and statistics. However, our
formative study (n=15) uncovered serious challenges in verifying AI-generated
results and steering the AI (i.e., guiding the AI system to produce the desired
output). We developed two contrasting approaches to address these challenges.
The first (Stepwise) decomposes the problem into step-by-step subgoals with
pairs of editable assumptions and code until task completion, while the second
(Phasewise) decomposes the entire problem into three editable, logical phases:
structured input/output assumptions, execution plan, and code. A controlled,
within-subjects experiment (n=18) compared these systems against a
conversational baseline. Users reported significantly greater control with the
Stepwise and Phasewise systems, and found intervention, correction, and
verification easier, compared to the baseline. The results suggest design
guidelines and trade-offs for AI-assisted data analysis tools.
