  Symbolic knowledge graphs (KGs) play a pivotal role in knowledge-centric
applications such as search, question answering and recommendation. As
contemporary language models (LMs) trained on extensive textual data have
gained prominence, researchers have extensively explored whether the parametric
knowledge within these models can match up to that present in knowledge graphs.
Various methodologies have indicated that enhancing the size of the model or
the volume of training data enhances its capacity to retrieve symbolic
knowledge, often with minimal or no human supervision. Despite these
advancements, there is a void in comprehensively evaluating whether LMs can
encompass the intricate topological and semantic attributes of KGs, attributes
crucial for reasoning processes. In this work, we provide an exhaustive
evaluation of language models of varying sizes and capabilities. We construct
nine qualitative benchmarks that encompass a spectrum of attributes including
symmetry, asymmetry, hierarchy, bidirectionality, compositionality, paths,
entity-centricity, bias and ambiguity. Additionally, we propose novel
evaluation metrics tailored for each of these attributes. Our extensive
evaluation of various LMs shows that while these models exhibit considerable
potential in recalling factual information, their ability to capture intricate
topological and semantic traits of KGs remains significantly constrained. We
note that our proposed evaluation metrics are more reliable in evaluating these
abilities than the existing metrics. Lastly, some of our benchmarks challenge
the common notion that larger LMs (e.g., GPT-4) universally outshine their
smaller counterparts (e.g., BERT).
