  In this paper, we explain the inference logic of large language models (LLMs)
as a set of symbolic concepts. Many recent studies have discovered that
traditional DNNs usually encode sparse symbolic concepts. However, because an
LLM has much more parameters than traditional DNNs, whether the LLM also
encodes sparse symbolic concepts is still an open problem. Therefore, in this
paper, we propose to disentangle the inference score of LLMs for dialogue tasks
into a small number of symbolic concepts. We verify that we can use those
sparse concepts to well estimate all inference scores of the LLM on all
arbitrarily masking states of the input sentence. We also evaluate the
transferability of concepts encoded by an LLM and verify that symbolic concepts
usually exhibit high transferability across similar input sentences. More
crucially, those symbolic concepts can be used to explain the exact reasons
accountable for the LLM's prediction errors.
