  Natural language (NL) has long been the predominant format for human
cognition and communication, and by extension, has been similarly pivotal in
the development and application of Large Language Models (LLMs). Yet, besides
NL, LLMs have seen various non-NL formats during pre-training, such as code and
logical expression. NL's status as the optimal format for LLMs, particularly in
single-LLM reasoning and multi-agent communication, has not been thoroughly
examined. In this work, we challenge the default use of NL by exploring the
utility of non-NL formats in these contexts. We show that allowing LLMs to
autonomously select the most suitable format before reasoning or communicating
leads to a 3.3 to 5.7\% improvement in reasoning efficiency for different LLMs,
and up to a 72.7\% reduction in token usage in multi-agent communication, all
while maintaining communicative effectiveness. Our comprehensive analysis
further reveals that LLMs can devise a format from limited task instructions
and that the devised format is effectively transferable across different LLMs.
Intriguingly, the structured communication format decided by LLMs exhibits
notable parallels with established agent communication languages, suggesting a
natural evolution towards efficient, structured communication in agent
communication. Our code is released at
\url{https://github.com/thunlp/AutoForm}.
