  Events across a timeline are a common data representation, seen in different
temporal modalities. Individual atomic events can occur in a certain temporal
ordering to compose higher level composite events. Examples of a composite
event are a patient's medical symptom or a baseball player hitting a home run,
caused distinct temporal orderings of patient vitals and player movements
respectively. Such salient composite events are provided as labels in temporal
datasets and most works optimize models to predict these composite event labels
directly. We focus on uncovering the underlying atomic events and their
relations that lead to the composite events within a noisy temporal data
setting. We propose Neural Temporal Logic Programming (Neural TLP) which first
learns implicit temporal relations between atomic events and then lifts logic
rules for composite events, given only the composite events labels for
supervision. This is done through efficiently searching through the
combinatorial space of all temporal logic rules in an end-to-end differentiable
manner. We evaluate our method on video and healthcare datasets where it
outperforms the baseline methods for rule discovery.
