  Although virtual agents are increasingly situated in environments where
natural language is the most effective mode of interaction with humans, these
exchanges are rarely used as an opportunity for learning. Leveraging language
interactions effectively requires addressing limitations in the two most common
approaches to language grounding: semantic parsers built on top of fixed object
categories are precise but inflexible and end-to-end models are maximally
expressive, but fickle and opaque. Our goal is to develop a system that
balances the strengths of each approach so that users can teach agents new
instructions that generalize broadly from a single example. We introduce the
idea of neural abstructions: a set of constraints on the inference procedure of
a label-conditioned generative model that can affect the meaning of the label
in context. Starting from a core programming language that operates over
abstructions, users can define increasingly complex mappings from natural
language to actions. We show that with this method a user population is able to
build a semantic parser for an open-ended house modification task in Minecraft.
The semantic parser that results is both flexible and expressive: the
percentage of utterances sourced from redefinitions increases steadily over the
course of 191 total exchanges, achieving a final value of 28%.
