  Recently, the community has achieved substantial progress on many commonsense
reasoning benchmarks. However, it is still unclear what is learned from the
training process: the knowledge, inference capability, or both? We argue that
due to the large scale of commonsense knowledge, it is infeasible to annotate a
large enough training set for each task to cover all commonsense for learning.
Thus we should separate the commonsense knowledge acquisition and inference
over commonsense knowledge as two separate tasks. In this work, we focus on
investigating models' commonsense inference capabilities from two perspectives:
(1) Whether models can know if the knowledge they have is enough to solve the
task; (2) Whether models can develop commonsense inference capabilities that
generalize across commonsense tasks. We first align commonsense tasks with
relevant knowledge from commonsense knowledge bases and ask humans to annotate
whether the knowledge is enough or not. Then, we convert different commonsense
tasks into a unified question answering format to evaluate models'
generalization capabilities. We name the benchmark as Commonsense Inference
with Knowledge-in-the-loop Question Answering (CIKQA).
