  Large language models based on self-attention mechanisms have achieved
astonishing performances not only in natural language itself, but also in a
variety of tasks of different nature. However, regarding processing language,
our human brain may not operate using the same principle. Then, a debate is
established on the connection between brain computation and artificial
self-supervision adopted in large language models. One of most influential
hypothesis in brain computation is the predictive coding framework, which
proposes to minimize the prediction error by local learning. However, the role
of predictive coding and the associated credit assignment in language
processing remains unknown. Here, we propose a mean-field learning model within
the predictive coding framework, assuming that the synaptic weight of each
connection follows a spike and slab distribution, and only the distribution,
rather than specific weights, is trained. This meta predictive learning is
successfully validated on classifying handwritten digits where pixels are input
to the network in sequence, and moreover on the toy and real language corpus.
Our model reveals that most of the connections become deterministic after
learning, while the output connections have a higher level of variability. The
performance of the resulting network ensemble changes continuously with data
load, further improving with more training data, in analogy with the emergent
behavior of large language models. Therefore, our model provides a starting
point to investigate the connection among brain computation, next-token
prediction and general intelligence.
