  Despite efforts to expand the knowledge of large language models (LLMs),
knowledge gaps -- missing or outdated information in LLMs -- might always
persist given the evolving nature of knowledge. In this work, we study
approaches to identify LLM knowledge gaps and abstain from answering questions
when knowledge gaps are present. We first adapt existing approaches to model
calibration or adaptation through fine-tuning/prompting and analyze their
ability to abstain from generating low-confidence outputs. Motivated by their
failures in self-reflection and over-reliance on held-out sets, we propose two
novel approaches that are based on model collaboration, i.e., LLMs probing
other LLMs for knowledge gaps, either cooperatively or competitively. Extensive
experiments with three LLMs on four QA tasks featuring diverse knowledge
domains demonstrate that both cooperative and competitive approaches to
unveiling LLM knowledge gaps achieve up to 19.3% improvements on abstain
accuracy against the strongest baseline. Further analysis reveals that our
proposed mechanisms could help identify failure cases in retrieval augmentation
and pinpoint knowledge gaps in multi-hop reasoning.
