  In this work, we systematically expose and measure the inconsistency and
knowledge gaps of Large Language Models (LLMs). Specifically, we propose an
automated testing framework (called KONTEST) which leverages a knowledge graph
to construct test cases. KONTEST probes and measures the inconsistencies in the
LLM's knowledge of the world via a combination of semantically-equivalent
queries and test oracles (metamorphic or ontological oracle). KONTEST further
mitigates knowledge gaps via a weighted LLM model ensemble. Using four
state-of-the-art LLMs (Falcon, Gemini, GPT3.5, and Llama2), we show that
KONTEST generates 19.2% error inducing inputs (1917 errors from 9983 test
inputs). It also reveals a 16.5% knowledge gap across all tested LLMs.
KONTEST's mitigation method reduces LLM knowledge gap by 32.48%. Our ablation
study further shows that GPT3.5 is not suitable for knowledge-based consistency
testing because it is only 60%-68% effective in knowledge construction.
