  Large language models (LLMs) have demonstrated remarkable performances on a
wide range of natural language tasks. Yet, LLMs' successes have been largely
restricted to tasks concerning words, sentences, or documents, and it remains
questionable how much they understand the minimal units of text, namely
characters. In this paper, we examine contemporary LLMs regarding their ability
to understand character composition of words, and show that most of them fail
to reliably carry out even the simple tasks that can be handled by humans with
perfection. We analyze their behaviors with comparison to token level
performances, and discuss the potential directions for future research.
