  Predicting the judgment of a legal case from its unannotated case facts is a
challenging task. The lengthy and non-uniform document structure poses an even
greater challenge in extracting information for decision prediction. In this
work, we explore and propose a two-level classification mechanism; both
supervised and unsupervised; by using domain-specific pre-trained BERT to
extract information from long documents in terms of sentence embeddings further
processing with transformer encoder layer and use unsupervised clustering to
extract hidden labels from these embeddings to better predict a judgment of a
legal case. We conduct several experiments with this mechanism and see higher
performance gains than the previously proposed methods on the ILDC dataset. Our
experimental results also show the importance of domain-specific pre-training
of Transformer Encoders in legal information processing.
