  Large language models (LLMs) have received increasing attention. However, due
to the complexity of its capabilities, how to rationally evaluate the
capabilities of LLMs is still a task to be solved. We propose the RoCar method,
which utilizes the defined basic schemas to randomly construct a task graph and
generates natural language evaluation tasks based on the task graph to evaluate
the reasoning and memory abilities of LLMs respectively. Due to the very large
randomness of the task construction process, it is possible to ensure that none
of the LLMs to be tested has directly learned the evaluation tasks,
guaranteeing the fairness of the evaluation method.
