  We propose to use thought-provoking children's questions (TPCQs), namely
Highlights BrainPlay questions, as a new method to drive artificial
intelligence research and to evaluate the capabilities of general-purpose AI
systems. These questions are designed to stimulate thought and learning in
children, and they can be used to do the same thing in AI systems, while
demonstrating the system's reasoning capabilities to the evaluator. We
introduce the TPCQ task, which which takes a TPCQ question as input and
produces as output (1) answers to the question and (2) learned generalizations.
We discuss how BrainPlay questions stimulate learning. We analyze 244 BrainPlay
questions, and we report statistics on question type, question class, answer
cardinality, answer class, types of knowledge needed, and types of reasoning
needed. We find that BrainPlay questions span many aspects of intelligence.
Because the answers to BrainPlay questions and the generalizations learned from
them are often highly open-ended, we suggest using human judges for evaluation.
