  This paper presents the first study for temporal relation extraction in a
zero-shot setting focusing on biomedical text. We employ two types of prompts
and five LLMs (GPT-3.5, Mixtral, Llama 2, Gemma, and PMC-LLaMA) to obtain
responses about the temporal relations between two events. Our experiments
demonstrate that LLMs struggle in the zero-shot setting performing worse than
fine-tuned specialized models in terms of F1 score, showing that this is a
challenging task for LLMs. We further contribute a novel comprehensive temporal
analysis by calculating consistency scores for each LLM. Our findings reveal
that LLMs face challenges in providing responses consistent to the temporal
properties of uniqueness and transitivity. Moreover, we study the relation
between the temporal consistency of an LLM and its accuracy and whether the
latter can be improved by solving temporal inconsistencies. Our analysis shows
that even when temporal consistency is achieved, the predictions can remain
inaccurate.
