  Non-programming users should be able to create their own customized scripts
to perform computer-based tasks for them, just by demonstrating to the machine
how it's done. To that end, we develop a system prototype which
learns-by-demonstration called HILC (Help, It Looks Confusing). Users train
HILC to synthesize a task script by demonstrating the task, which produces the
needed screenshots and their corresponding mouse-keyboard signals. After the
demonstration, the user answers follow-up questions.
  We propose a user-in-the-loop framework that learns to generate scripts of
actions performed on visible elements of graphical applications. While pure
programming-by-demonstration is still unrealistic, we use quantitative and
qualitative experiments to show that non-programming users are willing and
effective at answering follow-up queries posed by our system. Our models of
events and appearance are surprisingly simple, but are combined effectively to
cope with varying amounts of supervision.
  The best available baseline, Sikuli Slides, struggled with the majority of
the tests in our user study experiments. The prototype with our proposed
approach successfully helped users accomplish simple linear tasks, complicated
tasks (monitoring, looping, and mixed), and tasks that span across multiple
executables. Even when both systems could ultimately perform a task, ours was
trained and refined by the user in less time.
