  Recent efforts have evaluated large language models (LLMs) in areas such as
commonsense reasoning, mathematical reasoning, and code generation. However, to
the best of our knowledge, no work has specifically investigated the
performance of LLMs in natural language generation (NLG) tasks, a pivotal
criterion for determining model excellence. Thus, this paper conducts a
comprehensive evaluation of well-known and high-performing LLMs, namely
ChatGPT, ChatGLM, T5-based models, LLaMA-based models, and Pythia-based models,
in the context of NLG tasks. We select English and Chinese datasets
encompassing Dialogue Generation and Text Summarization. Moreover, we propose a
common evaluation setting that incorporates input templates and post-processing
strategies. Our study reports both automatic results, accompanied by a detailed
analysis.
