  Critical Infrastructure Facilities (CIFs), such as healthcare and
transportation facilities, are vital for the functioning of a community,
especially during large-scale emergencies. In this paper, we explore a
potential application of Large Language Models (LLMs) to monitor the status of
CIFs affected by natural disasters through information disseminated in social
media networks. To this end, we analyze social media data from two disaster
events in two different countries to identify reported impacts to CIFs as well
as their impact severity and operational status. We employ state-of-the-art
open-source LLMs to perform computational tasks including retrieval,
classification, and inference, all in a zero-shot setting. Through extensive
experimentation, we report the results of these tasks using standard evaluation
metrics and reveal insights into the strengths and weaknesses of LLMs. We note
that although LLMs perform well in classification tasks, they encounter
challenges with inference tasks, especially when the context/prompt is complex
and lengthy. Additionally, we outline various potential directions for future
exploration that can be beneficial during the initial adoption phase of LLMs
for disaster response tasks.
