  We introduce Falcon2-11B, a foundation model trained on over five trillion
tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a
vision-to-text model. We report our findings during the training of the
Falcon2-11B which follows a multi-stage approach where the early stages are
distinguished by their context length and a final stage where we use a curated,
high-quality dataset. Additionally, we report the effect of doubling the batch
size mid-training and how training loss spikes are affected by the learning
rate. The downstream performance of the foundation model is evaluated on
established benchmarks, including multilingual and code datasets. The
foundation model shows strong generalization across all the tasks which makes
it suitable for downstream finetuning use cases. For the vision language model,
we report the performance on several benchmarks and show that our model
achieves a higher average score compared to open-source models of similar size.
The model weights and code of both Falcon2-11B and Falcon2-11B-vlm are made
available under a permissive license.
