  Any human activity can be represented as a temporal sequence of actions
performed to achieve a certain goal. Unlike machine-made time series, these
action sequences are highly disparate as the time taken to finish a similar
action might vary between different persons. Therefore, understanding the
dynamics of these sequences is essential for many downstream tasks such as
activity length prediction, goal prediction, etc. Existing neural approaches
that model an activity sequence are either limited to visual data or are task
specific, i.e., limited to next action or goal prediction. In this paper, we
present ProActive, a neural marked temporal point process (MTPP) framework for
modeling the continuous-time distribution of actions in an activity sequence
while simultaneously addressing three high-impact problems -- next action
prediction, sequence-goal prediction, and end-to-end sequence generation.
Specifically, we utilize a self-attention module with temporal normalizing
flows to model the influence and the inter-arrival times between actions in a
sequence. Moreover, for time-sensitive prediction, we perform an early
detection of sequence goal via a constrained margin-based optimization
procedure. This in-turn allows ProActive to predict the sequence goal using a
limited number of actions. Extensive experiments on sequences derived from
three activity recognition datasets show the significant accuracy boost of
ProActive over the state-of-the-art in terms of action and goal prediction, and
the first-ever application of end-to-end action sequence generation.
