  Populating Commonsense Knowledge Bases (CSKB) is an important yet hard task
in NLP, as it tackles knowledge from external sources with unseen events and
entities. Fang et al. (2021a) proposed a CSKB Population benchmark with an
evaluation set CKBP v1. However, CKBP v1 adopts crowdsourced annotations that
suffer from a substantial fraction of incorrect answers, and the evaluation set
is not well-aligned with the external knowledge source as a result of random
sampling. In this paper, we introduce CKBP v2, a new high-quality CSKB
Population benchmark, which addresses the two mentioned problems by using
experts instead of crowd-sourced annotation and by adding diversified
adversarial samples to make the evaluation set more representative. We conduct
extensive experiments comparing state-of-the-art methods for CSKB Population on
the new evaluation set for future research comparisons. Empirical results show
that the population task is still challenging, even for large language models
(LLM) such as ChatGPT. Codes and data are available at
https://github.com/HKUST-KnowComp/CSKB-Population.
