  The evolution of Generative Pre-trained Transformer (GPT) models has led to
significant advancements in various natural language processing applications,
particularly in legal textual entailment. We present an analysis of GPT-3.5
(ChatGPT) and GPT-4 performances on COLIEE Task 4 dataset, a prominent
benchmark in this domain. The study encompasses data from Heisei 18 (2006) to
Reiwa 3 (2021), exploring the models' abilities to discern entailment
relationships within Japanese statute law across different periods. Our
preliminary experimental results unveil intriguing insights into the models'
strengths and weaknesses in handling legal textual entailment tasks, as well as
the patterns observed in model performance. In the context of proprietary
models with undisclosed architectures and weights, black-box analysis becomes
crucial for evaluating their capabilities. We discuss the influence of training
data distribution and the implications on the models' generalizability. This
analysis serves as a foundation for future research, aiming to optimize
GPT-based models and enable their successful adoption in legal information
extraction and entailment applications.
