  In this paper, we propose sequence-based pretraining methods to enhance
procedural understanding in natural language processing. Procedural text,
containing sequential instructions to accomplish a task, is difficult to
understand due to the changing attributes of entities in the context. We focus
on recipes, which are commonly represented as ordered instructions, and use
this order as a supervision signal. Our work is one of the first to compare
several 'order as-supervision' transformer pre-training methods, including
Permutation Classification, Embedding Regression, and Skip-Clip, and shows that
these methods give improved results compared to the baselines and SoTA LLMs on
two downstream Entity-Tracking datasets: NPN-Cooking dataset in recipe domain
and ProPara dataset in open domain. Our proposed methods address the
non-trivial Entity Tracking Task that requires prediction of entity states
across procedure steps, which requires understanding the order of steps. These
methods show an improvement over the best baseline by 1.6% and 7-9% on
NPN-Cooking and ProPara Datasets respectively across metrics.
