  We present a novel framework for learning to interpret and generate language
using only perceptual context as supervision. We demonstrate its capabilities
by developing a system that learns to sportscast simulated robot soccer games
in both English and Korean without any language-specific prior knowledge.
Training employs only ambiguous supervision consisting of a stream of
descriptive textual comments and a sequence of events extracted from the
simulation trace. The system simultaneously establishes correspondences between
individual comments and the events that they describe while building a
translation model that supports both parsing and generation. We also present a
novel algorithm for learning which events are worth describing. Human
evaluations of the generated commentaries indicate they are of reasonable
quality and in some cases even on par with those produced by humans for our
limited domain.
