  We investigate contract element extraction. We show that LSTM-based encoders
perform better than dilated CNNs, Transformers, and BERT in this task. We also
find that domain-specific WORD2VEC embeddings outperform generic pre-trained
GLOVE embeddings. Morpho-syntactic features in the form of POS tag and token
shape embeddings, as well as context-aware ELMO embeddings do not improve
performance. Several of these observations contradict choices or findings of
previous work on contract element extraction and generic sequence labeling
tasks, indicating that contract element extraction requires careful
task-specific choices. Analyzing the results of (i) plain TRANSFORMER-based and
(ii) BERT-based models, we find that in the examined task, where the entities
are highly context-sensitive, the lack of recurrency in TRANSFORMERs greatly
affects their performance.
