  Code generation with Large Language Models (LLMs) has been extensively
studied and achieved remarkable progress. As a complementary aspect to code
generation, test case generation is of crucial importance in ensuring the
quality and reliability of code. However, using LLMs as test case generators
has been much less explored. Current research along this line primarily focuses
on enhancing code generation with assistance from test cases generated by LLMs,
while the performance of LLMs in test case generation alone has not been
comprehensively examined. To bridge this gap, we conduct extensive experiments
to study how well LLMs can generate high-quality test cases. We find that as
the problem difficulty increases, state-of-the-art LLMs struggle to generate
correct test cases, largely due to their inherent limitations in computation
and reasoning. To mitigate this issue, we further propose a multi-agent
framework called \emph{TestChain} that decouples the generation of test inputs
and test outputs. Notably, TestChain uses a ReAct format conversation chain for
LLMs to interact with a Python interpreter in order to provide more accurate
test outputs. Our results indicate that TestChain outperforms the baseline by a
large margin. Particularly, in terms of the accuracy of test cases, TestChain
using GPT-4 as the backbone achieves a 13.84\% improvement over the baseline on
the LeetCode-hard dataset.
