  We present InferWiki, a Knowledge Graph Completion (KGC) dataset that
improves upon existing benchmarks in inferential ability, assumptions, and
patterns. First, each testing sample is predictable with supportive data in the
training set. To ensure it, we propose to utilize rule-guided train/test
generation, instead of conventional random split. Second, InferWiki initiates
the evaluation following the open-world assumption and improves the inferential
difficulty of the closed-world assumption, by providing manually annotated
negative and unknown triples. Third, we include various inference patterns
(e.g., reasoning path length and types) for comprehensive evaluation. In
experiments, we curate two settings of InferWiki varying in sizes and
structures, and apply the construction process on CoDEx as comparative
datasets. The results and empirical analyses demonstrate the necessity and
high-quality of InferWiki. Nevertheless, the performance gap among various
inferential assumptions and patterns presents the difficulty and inspires
future research direction. Our datasets can be found in
https://github.com/TaoMiner/inferwiki
