  Automatic data abstraction is an important capability for both benchmarking
machine intelligence and supporting summarization applications. In the former
one asks whether a machine can `understand' enough about the meaning of input
data to produce a meaningful but more compact abstraction. In the latter this
capability is exploited for saving space or human time by summarizing the
essence of input data. In this paper we study a general reinforcement learning
based framework for learning to abstract sequential data in a goal-driven way.
The ability to define different abstraction goals uniquely allows different
aspects of the input data to be preserved according to the ultimate purpose of
the abstraction. Our reinforcement learning objective does not require
human-defined examples of ideal abstraction. Importantly our model processes
the input sequence holistically without being constrained by the original input
order. Our framework is also domain agnostic -- we demonstrate applications to
sketch, video and text data and achieve promising results in all domains.
