  Natural language programming is a promising approach to enable end users to
instruct new tasks for intelligent agents. However, our formative study found
that end users would often use unclear, ambiguous or vague concepts when
naturally instructing tasks in natural language, especially when specifying
conditionals. Existing systems have limited support for letting the user teach
agents new concepts or explaining unclear concepts. In this paper, we describe
a new multi-modal domain-independent approach that combines natural language
programming and programming-by-demonstration to allow users to first naturally
describe tasks and associated conditions at a high level, and then collaborate
with the agent to recursively resolve any ambiguities or vagueness through
conversations and demonstrations. Users can also define new procedures and
concepts by demonstrating and referring to contents within GUIs of existing
mobile apps. We demonstrate this approach in PUMICE, an end-user programmable
agent that implements this approach. A lab study with 10 users showed its
usability.
