  Recent advancements in Artificial Intelligence, and particularly Large
Language Models (LLMs), offer promising prospects for aiding system
administrators in managing the complexity of modern networks. However, despite
this potential, a significant gap exists in the literature regarding the extent
to which LLMs can understand computer networks. Without empirical evidence,
system administrators might rely on these models without assurance of their
efficacy in performing network-related tasks accurately.
  In this paper, we are the first to conduct an exhaustive study on LLMs'
comprehension of computer networks. We formulate several research questions to
determine whether LLMs can provide correct answers when supplied with a network
topology and questions on it. To assess them, we developed a thorough framework
for evaluating LLMs' capabilities in various network-related tasks. We evaluate
our framework on multiple computer networks employing proprietary (e.g., GPT4)
and open-source (e.g., Llama2) models. Our findings in general purpose LLMs
using a zero-shot scenario demonstrate promising results, with the best model
achieving an average accuracy of 79.3%. Proprietary LLMs achieve noteworthy
results in small and medium networks, while challenges persist in comprehending
complex network topologies, particularly for open-source models. Moreover, we
provide insight into how prompt engineering can enhance the accuracy of some
tasks.
