  We consider Large-Scale Multi-Label Text Classification (LMTC) in the legal
domain. We release a new dataset of 57k legislative documents from EURLEX,
annotated with ~4.3k EUROVOC labels, which is suitable for LMTC, few- and
zero-shot learning. Experimenting with several neural classifiers, we show that
BIGRUs with label-wise attention perform better than other current state of the
art methods. Domain-specific WORD2VEC and context-sensitive ELMO embeddings
further improve performance. We also find that considering only particular
zones of the documents is sufficient. This allows us to bypass BERT's maximum
text length limit and fine-tune BERT, obtaining the best results in all but
zero-shot learning cases.
