  In this paper, we explore the task of automatically generating natural
language descriptions of salient patterns in a time series, such as stock
prices of a company over a week. A model for this task should be able to
extract high-level patterns such as presence of a peak or a dip. While typical
contemporary neural models with attention mechanisms can generate fluent output
descriptions for this task, they often generate factually incorrect
descriptions. We propose a computational model with a truth-conditional
architecture which first runs small learned programs on the input time series,
then identifies the programs/patterns which hold true for the given input, and
finally conditions on only the chosen valid program (rather than the input time
series) to generate the output text description. A program in our model is
constructed from modules, which are small neural networks that are designed to
capture numerical patterns and temporal information. The modules are shared
across multiple programs, enabling compositionality as well as efficient
learning of module parameters. The modules, as well as the composition of the
modules, are unobserved in data, and we learn them in an end-to-end fashion
with the only training signal coming from the accompanying natural language
text descriptions. We find that the proposed model is able to generate
high-precision captions even though we consider a small and simple space of
module types.
