  In this report, we present ChuXin, an entirely open-source language model
with a size of 1.6 billion parameters. Unlike the majority of works that only
open-sourced the model weights and architecture, we have made everything needed
to train a model available, including the training data, the training process,
and the evaluation code. Our goal is to empower and strengthen the open
research community, fostering transparency and enabling a new wave of
innovation in the field of language modeling. Furthermore, we extend the
context length to 1M tokens through lightweight continual pretraining and
demonstrate strong needle-in-a-haystack retrieval performance. The weights for
both models are available at Hugging Face to download and use.
