Upload folder using huggingface_hub

LICENSE

@@ -0,0 +1,31 @@
+The following applies to all files in this repository, unless otherwise noted:
+
+Copyright (c) 2025 Aleph Alpha Research GmbH. All rights reserved.
+
+This project is licensed under the terms of the Open Aleph License 1.0, available at
+https://github.com/Aleph-Alpha/.github/blob/main/oal.pdf
+
+---
+Excerpt from the license text:
+
+Subject to the terms and conditions of this License, the Licensor grants you a non-exclusive, worldwide,
+non-transferable, non-sublicensable, and royalty-free limited right to use, copy, modify, distribute, make
+otherwise publicly available, and reproduce the Works and Derivative Works under Licensor’s copyright,
+for any Non-Commercial and Non-Administrative purpose.
+You may not use, copy, modify, distribute, make otherwise publicly available, reproduce, or sublicense the
+Works or Derivative Works except as expressly provided under and in accordance with this License.
+Your rights granted under this License will automatically terminate if you fail to comply with any of the
+terms of this License.
+
+EXCEPT FOR DAMAGES CAUSED BY INTENT OR FRAUDULENTLY CONCEALED
+DEFECTS, AND EXCEPT FOR DAMAGES RESULTING FROM BREACH OF ANY
+WARRANTY OR GUARANTEE EXPRESSLY GIVEN BY LICENSOR IN THE OPEN ALEPH LICENSE,
+IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY
+DAMAGES ARISING OUT OF THE OPEN ALEPH LICENSE OR THE USE OF THE WORK. ANY
+MANDATORY STATUTORY LIABILITY UNDER APPLICABLE LAW REMAINS
+UNAFFECTED.
+
+EXCEPT AS EXPRESSLY STATED IN THIS LICENSE OR REQUIRED BY APPLICABLE
+LAW, THE WORKS ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES
+OF ANY KIND INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES REGARDING
+THE CONTENTS, ACCURACY, OR FITNESS FOR A PARTICULAR PURPOSE.

README.md

@@ -0,0 +1,448 @@
+---
+language:
+- en
+- de
+license: other
+thumbnail: https://huggingface.co/Aleph-Alpha/Llama-TFree-HAT-Pretrained-8B-DPO/raw/main/source/aleph_alpha_logo_thumbnail.png
+license_name: open-aleph-license
+license_link: LICENSE
+tags:
+- Aleph Alpha Research
+- pytorch
+- Hierarchical Autoregressive Transformer
+- HAT
+model-index:
+- name: TFree-HAT-Pretrained-8B-Base
+  results: []
+---
+
+<div align="center">
+  <img src="source/aleph_alpha_logo.svg" width="60%" alt="Aleph Alpha Research Logo" />
+</div>
+ 
+<div align="center" style="line-height: 1;">
+  <a href="https://aleph-alpha.com/research/" target="_blank" style="margin: 2px;">
+    <img alt="Homepage" src="source/aleph_alpha_homepage_badge.svg" style="display: inline-block; vertical-align: middle;" />
+  </a>
+  <a href="https://huggingface.co/Aleph-Alpha" target="_blank" style="margin: 2px;">
+    <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-AlephAlpha%20Research-e3ff00?color=e3ff00&amp;logoColor=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+</div>
+ 
+<div align="center" style="line-height: 1;">
+  <a href="https://twitter.com/Aleph__Alpha" target="_blank" style="margin: 2px;">
+    <img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-AlephAlpha_Research-white?logo=x&amp;logoColor=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+  <a href="https://huggingface.co/Aleph-Alpha/TFree-HAT-Pretrained-8B-Base/blob/main/LICENSE" style="margin: 2px;">
+    <img alt="License" src="https://img.shields.io/badge/License-Open Aleph License-white?&amp;color=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+</div>
+
+<hr>
+
+# TFree-HAT-Pretrained-8B-Base
+<!-- markdownlint-disable first-line-h1 -->
+<!-- markdownlint-disable html -->
+<!-- markdownlint-disable no-duplicate-header -->
+
+This model card provides an overview of our **TFree-HAT-Pretrained-8B-Base** model , which is a foundation model developed by Aleph Alpha Research* and publicly available under the Open Aleph License, a license explicitly allowing for non-commercial research and educational use.
+
+The model is based on our Hierarchical Autoregressive Transformer (HAT) architecture which is described originally in our [paper](https://arxiv.org/abs/2501.10322). This novel architecture integrates character-level encoding and decoding with the word-level backbone, allowing for improved text compression (less sequence positions) and performance in the languages it has been trained on, and potentially higher robustness to prompt changes, as well as improved adaptability to new languages & domains via fine-tuning.
+
+The model was pre-trained in English & German on carefully curated data in compliance with applicable EU and national regulations, including copyright and data privacy laws. It shows strong proficiency in German, while also beating Llama 3.1 on many benchmarks in English. 
+
+You can find model weights and their corresponding safetensors conversions at the following links:
+
+| Model Name | Description |
+| --- | --- |
+| `TFree-HAT-Pretrained-8B-Base` | [Link](https://huggingface.co/Aleph-Alpha/TFree-HAT-Pretrained-8B-Base) - pre-trained for English and German, adapted to a maximum context length of 32900 words  |
+
+# Model Access
+
+We provide access to our models through the channels listed below.
+
+- **HuggingFace**: The model’s weights as well as basic inference implementation are available on HuggingFace under the [Open Aleph License](https://github.com/Aleph-Alpha/.github/blob/main/oal.pdf), a license explicitly allowing for non-commercial research and educational use.
+
+We do not collect PII (personally identifiable information) for any of these channels. We do not log user inputs to the models. We do not train on user data.
+
+**Note**: The same models are made available to users regardless of their geographic location and their input language but subject to sanction regimes, technology export regulations, and other restrictions that may apply. The same offering is provided to all countries within and external to the European Union if no legal restrictions apply.
+
+# How to use
+
+## Inference
+
+We provide an inference module compatible with HuggingFace Transformers for running model inference. We recommend pinning the transformers library to version 4.46.3. Before executing the inference example below, make sure the [hat-splitter package](https://pypi.org/project/hat-splitter/) is installed in your environment.
+
+```shell
+pip install 'hat-splitter>=0.1.9' 'transformers==4.46.3' torch
+pip install flash_attn
+```
+
+Download model weights and run inference using the following example:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM
+INPUT ="When was Rome founded?"
+MODEL_ID = "Aleph-Alpha/TFree-HAT-Pretrained-8B-Base"
+model = AutoModelForCausalLM.from_pretrained(
+    trust_remote_code=True,
+    pretrained_model_name_or_path=MODEL_ID,
+    attn_implementation="flash_attention_2",
+).to("cuda", torch.bfloat16)
+input_ids, cumulative_word_lengths = model._prepare_input(INPUT)
+model_output = model.generate(
+    input_ids,
+    cumulative_seq_lengths_per_word=cumulative_word_lengths,
+    max_new_tokens=300,
+    use_cache=False,
+)
+print("Prompt: ", INPUT)
+print("Completion: ", model_output.completion_text)
+```
+
+Please note that the realized inference speed strongly depends on the maturity of the inference implementation beyond the intrinsic text compression of any model. Besides this huggingface transformers-based inference solution, we are also releasing a [vLLM-based inference solution](https://github.com/Aleph-Alpha/vllm) for our models that is optimized for batched inference. Please note that this vLLM inference for HAT is still under active development.
+
+
+# Evaluation
+
+**Performance**: Our T-Free models deliver performance on par with strong tokenizer-based models such as [Llama 3.1 8B Base](https://huggingface.co/meta-llama/Llama-3.1-8B). Respective benchmarks and results can be found in the tables below.
+
+**Efficiency**: Our tokenizer-free approach results in improved text compression, providing a foundation for improved efficiency in inference speed. We measure in terms of words processed across all languages and domains. We define the metric as **tokenizer fertility** or **bytes per sequence position**, where a higher value indicates better performance. Latency and throughput are currently out of scope for research-centric evaluations and will be addressed in the future. Currently, our evaluation framework automatically measures **bytes per sequence position** across datasets, allowing us to derive text compression scores and analyze variations across different dataset distributions. The end to end resulting efficiency is depends on the inference implementation beyond the scope of the here provided inference implementation and reported compression scores.
+
+**Disclaimer**: The results presented below were generated using our internal inference implementation, not the inference module mentioned above. As a sanity check, we reproduced some of the benchmarks using our evaluation framework with the huggingface inference code, but other results might still deviate slightly. We will also make source-available both our evaluation framework and a [high-performance vLLM integration](https://github.com/Aleph-Alpha/vllm) for this model to ensure reproducibility. 
+
+**Metric Glossary**
+
+`log_acc`: Average Accuracy Loglikelihood<br>
+`norm_log_acc`: Average Normalized Loglikelihood Accuracy<br>
+`comp_acc`: Average Completion Accuracy<br>
+`norm_prob_mass`: Average Probability Mass Normalized<br>
+`bleu`: Average BLEU Score<br>
+`rouge_gm`: Average ROUGE-Geometric-Mean<br>
+`F1`: Average F1<br>
+`CS`: Chatbot Style<br>
+`IF`: Instruction Following<br>
+`LC`: Language Consistency<br>
+`CI`: Concordance Index<br>
+`ES`: Exponential Similarity
+
+## Pre-training Benchmarks
+
+|   Group   |   Task   |   Metric Name   |   Num Fewshot   |   [TFree-HAT-Pretrained-8B-Base](https://huggingface.co/Aleph-Alpha/TFree-HAT-Pretrained-8B-Base)  |   [Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B)   |   [TFree-HAT-Pretrained-8B-Base](https://huggingface.co/Aleph-Alpha/TFree-HAT-Pretrained-8B-Base) Compression   |   [Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B) Compression   |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+|   Knowledge   |   MMLU   |   `norm_log_acc`   |   5   |  **0.669**   |   0.668   |  **5.184**  |   4.278   |
+|   Knowledge   |   MMLU Pro   |   `norm_log_acc`   |   5   |  **0.387** |   0.367   |  **4.734**   |   3.731   |
+|   Knowledge   |   OpenBookQA   |   `norm_log_acc`   |   10   | 0.360  |  **0.366**  |  **4.982** |   4.724   |
+|   Knowledge   |   TriviaQA   |   `comp_acc`   |   10   |  0.657 |  **0.695**   |  **5.317**  |   4.221   |
+|   Knowledge   |   TruthfulQA   |   `norm_prob_mass`   |   6   | **0.304**  |  0.279   |  **4.945** |   4.197   |
+|   Reasoning   |   ARC Challenge   |   `norm_log_acc`   |   25   | **0.592** |   0.538   |  **5.514**  |   4.924   |
+|   Reasoning   |   Winogrande   |   `norm_log_acc`   |   5   |  **0.754**  |   0.747   | **5.158**  |   4.909   |
+|   German   |   MMMLU   |   `norm_log_acc`   |   5   |  **0.618**   |   0.576   |  **6.056** |   3.410   |
+|   German   |   WMT16   |   `bleu`  |   5   |  34.465   |   **34.998**   | **5.968**   |   4.210   |
+|   German   |   WMT20   |   `bleu`  |   5   |  **33.212**   |   32.892   | **6.269**   |   4.222   |
+|   Math   |   GSM8K   |   `comp_acc`   |   8   |  0.508 |   **0.528**   |  **3.840** |   3.332   |
+|   Long context   |   GSM8K   |   `comp_acc`   |   16   |  0.520  |  ---   |  3.837 |   ---   |
+|   Long context   |   Long Bench v2   |   `norm_log_acc`   |   10   |  0.299  |  ---   |  5.125 |   ---   |
+|   Long context German  |   Long Bench v2   |   `norm_log_acc`   |   10   |  0.282  |  ---   |  5.872 |   ---   |
+|   Safety   |   Winogender   |   `norm_log_acc`   |   5   | **0.662**    |   0.636   | **5.232**  |   4.799   |
+
+# Training Details
+
+## Model Architecture
+
+The model uses a hierarchical autoregressive transformer (HAT) architecture consisting of three components: encoder, backbone, and decoder, together with connector layers between components. Encoder, backbone, and decoder are all instances of autoregressive transformers with pre-norm residual blocks in the style of Llama, using a SwiGLU unit as a feed-forward block, with all model parameters active during training and inference. The backbone model uses standard causal attention, while the encoder and decoder use local causal attention with a finite look-back window. The architecture of the backbone largely follows the design of LLama 3.1 8B (with embedding and language modeling head removed and weights randomly initialized). In addition, we added per-head QK-norm in the backbone, which we found important for training stability.
+
+The encoder processes input text as a sequence of UTF-8 bytes and produces a sequence of activations of the same length. This sequence is then split into chunks corresponding to words or other semantic units in the text (this is further explained below). In the encoder-backbone connector layer, for each word, a learned latent vector cross-attends to its corresponding chunk of encoder activations. The resulting sequence of latent vectors then serves as input to the backbone. The backbone processes this latent sequence and produces a sequence of word-level representations. Finally, the decoder module is another transformer that acts on the byte-level activations and has an LM head that produces next-byte probabilities. To make use of the higher level information stored in the word-level embeddings during decoding, another cross-attention mechanism is used. In each transformer block of the decoder, every byte-level position cross-attends to the backbone’s word-level representations that correspond to the words preceding this byte.
+
+## Encoder module
+
+|    |   **8B**   |
+| --- | --- |
+|   Number of layers   |   6   |
+|   Number of attention heads   |   8   |
+|   Head size   |   128   |
+|   Number of Key-Value heads   |   8   |
+|   Hidden size   |   1024   |
+|   Cross-attention hidden size   |   4096   |
+|   MLP expansion factor   |   2.75   |
+|   MLP type   |   SwiGLU   |
+|   Sequence length   |   262144   |
+|   Position embeddings   |   RoPE with base 1e5   |
+|   Attention type   |   causal, local with window size 768   |
+|   QK-norm  |   disabled  |
+
+## Backbone module
+
+|    |   **8B**   |
+| --- | --- |
+|   Number of layers   |   32   |
+|   Number of attention heads   |   32   |
+|   Head size   |   128   |
+|   Number of Key-Value heads   |   8   |
+|   Hidden size   |   4096   |
+|   MLP expansion factor   |   3.5   |
+|   MLP type   |   SwiGLU   |
+|   Sequence length   |   32900   |
+|   Position embeddings   |   RoPE with base 5e5   |
+|   Attention type   |   causal   |
+|   QK-norm  |   per head   |
+
+## Decoder module
+
+|    |   **8B**   |
+| --- | --- |
+|   Number of layers   |   4   |
+|   Number of attention heads   |   8   |
+|   Head size   |   128   |
+|   Number of Key-Value heads   |   8   |
+|   Hidden size   |   1024   |
+|   Cross-attention hidden size   |   4096   |
+|   MLP expansion factor   |   2.75   |
+|   MLP type   |   SwiGLU   |
+|   Sequence length   |   262144   |
+|   Position embeddings   |   RoPE with base 1e5   |
+|   Attention type   |   causal, local with window size 768   |
+|   QK-norm  |   disabled   |
+
+**Parameter count**
+
+Total: `7,192,507,136`
+Encoder: `119,293,696`
+Backbone: `6,979,592,192`
+Decoder: `93,621,248`
+
+We note that one distinctive property of our tokenizer-free architectures is that encoder and decoder are substantially smaller than typical embedding and language model head layers of tokenizer-based models. Because of this, while our models share the architecture with Llama 3.1 8B (plus the added QK-norm), they are closer to 7B than 8B parameters in total.
+
+**Word splitter**
+
+To split arbitrary byte sequences, we adopted the guidelines from [UAX #29](https://unicode.org/reports/tr29/), which splits text into words for common Western languages but also produces meaningful semantic units for other types of languages (e.g. Chinese, Japanese, Korean). From now on, we refer to these splits as words.
+
+We also merged leading whitespace and trailing punctuation into the words to reduce sequence length at the word level.
+
+To improve the processing of code and math documents, we made additional adjustments to the Unicode splitter. First, we split instances of camel cases like FooBar into Foo and Bar. Second, we treated math symbols (again by Unicode standard) as separate words.
+
+## Pre-Training
+
+**Approach**
+
+We randomly initialized all model parameters. The model was then trained on the next-byte-prediction objective on a large and diverse document corpus (see below). Initially, we trained on sequences up to 3500 words for a total amount of nearly 4T words. We used global batch-size of 1024 (2.5M words) and followed a warmup-stable-decay schedule with a warmup of 5000 steps, a phase of stable learning rate 2e-3 for 945000 steps and inverse-square-root cooldown to learning rate 0 over the last 50000 steps. We employed weight decay of 0.05 for all parameters except for the embedding and normalization parameters. We employed QK-norm per head and attention logit softcapping at 100, which we found to be important for training stability during pretraining.
+
+We then continued training on sequences of up to 32900 words for another 2500 steps with global batch size 128, totaling to 10.5B words, upweighting longer documents to make use of the extended context. We used warmup-stable-decay learning rate schedule with 500 steps warmup, a phase of stable learning 2e-4, and a final decay to 0 over the last 500 steps. We disabled attention logit softcapping during this long-context adaptation such that it is not required during inference.
+
+The training was conducted in our [Scaling framework](https://github.com/Aleph-Alpha/scaling).
+
+**Data sources**
+
+The model was trained on a filtered subset of diverse corpora of text data including proprietary curated datasets, high-quality web content, public domain sources, German texts, mathematical texts, and programming code. The proportions and sources of data we used in the pre-training were:
+
+English Language Data (70%)
+
+- curated web and synthetic data (63%)
+
+- high quality curated sources such as Wikipedia and public domain books (7%)
+
+German Language Data (7%)
+
+- curated web and synthetic data (6.3%)
+
+- high quality curated sources such as Wikipedia and public domain books (0.7%)
+
+Mathematical Content (5%)
+
+- mathematical code and proofs (2%)
+
+- mathematical word problems and equations (3%)
+
+Programming Code (18%)
+
+- general programming code (11%)
+
+- high-quality and synthetic Python code (7%)
+
+## Data curation
+
+We applied a range of curation techniques, e.g., for German as described in [Aleph-Alpha-GermanWeb](https://huggingface.co/datasets/Aleph-Alpha/Aleph-Alpha-GermanWeb). These include but are not limited to: 
+
+- URL filtering. We used a URL filter developed to filter out fraudulent, harmful, and illegal content from an explicit blocklist, e.g., adult websites, or URLs containing words associated with fraudulent, harmful, or adult content.
+
+- Text extraction. Natural language texts which were embedded HTML and other web programming languages were extracted using the [Resiliparse](https://github.com/chatnoir-eu/chatnoir-resiliparse) text extractor.
+
+- Language identification. We used a [fastText language classifier](https://fasttext.cc/docs/en/language-identification.html) trained on character n-grams from Wikipedia to identify, retain, and sort texts into English and German.
+
+- Repetition removal. We applied heuristic methods for detection and removal of repetitions on the line, paragraph, and character level.
+
+- Document- and line-level filtering. We utilized additional document-level heuristics to ensure documents had reasonable numbers and quality of words, naturalistic symbols-to-words and numbers-to-words ratios, not predominantly made up of bullet points, and a sufficient quantity of real words.
+
+- Deduplication. Using exact and fuzzy deduplication to remove duplicate documents.
+
+## Synthetic data
+
+We also generated synthetic data by using permissively-licensed LLMs.
+
+
+## Legal Compliance
+
+We acknowledge and abide by applicable national and international regulations, including copyright, data privacy, and other related legislation. Any text and data mining by us is performed in compliance with Directive (EU) 2019/790 and its respective national transposition. During the training and fine-tuning of our models, we comply with applicable data privacy laws, including Regulation (EU) 2016/679 (GDPR) and national data privacy regulations. To the extent possible and foreseeable, we also took legislation with forthcoming obligations into account, such as the obligations for General Purpose AI Models under Regulation (EU) 2024/1689 (EU AI Act), and will constantly monitor such developments and adapt our products and this model card accordingly.
+
+# Resource Usage
+
+## Compute & Training Efficiency 
+
+The following table shows the compute resources used in the training stages for the 8B models.
+
+|   **Model**   |   **Training phase**   |   **GPUs**   |   **Approximate average power consumption per GPU**   |   **Approximate GPU hours**   |
+| --- | --- | --- | --- | --- |
+|   8B   |   Pre-training  (part 1) |   256 x H200   |   460W   |   111,822   |
+|   8B   |   Pre-training  (part 2) |   256 x H100   |   460W   |   151,289   |
+|   8B   |   Long context adaptation   |   256 x H100   |   190W   |   5,328   |
+
+## Environmental Impact
+
+Our H200 and A100 infrastructure runs entirely on 100% renewable energy, ensuring that no CO₂ emissions are directly incurred from training. In addition to this, the H200 data center boasts a power usage effectiveness (PUE) of ≤1.2. Its operation also maintains a net-zero water footprint. Specific number on renewable energy usage for the H100 GPUs is not yet available to us.
+
+To estimate the carbon footprint of inference, we base our calculations on publicly available data from the infrastructure provider and, where applicable, standard emissions accounting methodology. We report:
+
+- **Carbon emitted**: GPU runtime emissions
+
+- **Carbon emitted accounting for PUE**: GPU runtime emissions scaled by the data center's PUE
+
+Because the data centers operate fully on renewable energy, both metrics for its operation (excluding infrastructure-related emissions, e.g., initial chip manufacturing) are effectively zero. For H100 GPU infrastructure no information has been made available to us.
+
+|   Metric   |   H200 GPU   |   H100 GPU   |   A100 GPU   |
+| --- | --- | --- | --- |
+|   Carbon emitted   |   0 kg CO₂   |   no information available   |   0 kg CO₂   |
+|   Carbon emitted accounting for PUE   |   0 kg CO₂   |   no information available   |   0 kg CO₂   |
+
+## Power Consumption
+
+|   GPU Model   |   Max Power (W)   |
+| --- | --- |
+|   A100   |   400 W   |
+|   H100   |   700 W   |
+|   H200   |   700 W   |
+
+Numbers may be contextualized with reference to publicly available studies, such as the carbon footprint of language model training.
+
+# Intended Use
+
+These models are intended to be deployed as components of AI systems or applications. Use-cases and the model's capabilities include but are not limited to: text generation, classification, summarization, question answering, and labeling. Note that applications might require additional model adaptations or components for guarding against unwanted application behavior or model output.
+
+## Non-Permitted Use
+
+Our models shall not be used for illegal or unlawful actions of any kind and with any illegal or unlawful content. This includes in particular prohibited practices according to Article 5 of Regulation (EU) 2024/1689 (EU AI Act) and other activities such as engaging in terrorism, violence, human trafficking, illegal distribution of materials to minors, sexual solicitation, any other criminal activities, harassment, discrimination, creating or promoting malicious code or activities risking death or harm, including those related to military or nuclear applications, and activities not in compliance with sanction regimes, technology export regulations, and other restrictions that may apply. The models are to be used following ethical standards. The utilization of our technology is always governed by, and may be limited in accordance with, our Terms and Conditions, the Open Aleph License, or any specific agreement we might have established with you.
+
+Although we do not inspect the requests sent to our API, we regularly review and monitor potential violations that may be related to our models and depending on the circumstances of the specific case take legal action against them. This includes but is not limited to, enforcement to remove published model content, requesting compensation for damages caused, and account termination or removal of credits.
+
+For non-anonymous reports, we also provide an appeals mechanism for usage policy violations via our dedicated contact address [[email protected]](mailto:[email protected]) to communicate with us.
+
+Customers and partners are enabled to use our [ticketing system](https://servicedesk.aleph-alpha.de/external) for appeals, claims, and feedback.
+
+
+# Risks and Limitations
+
+**Note:** Language models are **not agents** and not optimized for prescriptive actions. The use of language models in high-stake environments, for critical decisions or to support a user's wellbeing should be performed with additional guardrails in place.
+
+## Risk Categories
+
+In the following sections, we describe risk categories and provide examples of completions we would consider inappropriate or harmful. We then describe steps to minimize these risks.
+
+**Harmful Language**
+
+Large language models can sometimes generate undesired outputs that are unsuitable for certain applications. This includes producing content with harmful language, discriminative content, inappropriate tone and style, systemic biases, or suggestions that might encourage illegal actions. Such outputs can also include incorrect, outdated information, or material that is not suitable for all ages. While we constantly take efforts to reduce the likelihood of such undesired outputs, this possibility can never be fully ruled out. To minimize these issues, the following strategies can be employed:
+
+- Abide by the guidance on illegal use provided for in this Model Card.
+    
+- Crafting prompts carefully to guide the model's output more effectively.
+    
+- Utilizing a finetuned model (often referred to as a control or instruct model) that prioritizes using explicitly provided information.
+    
+- Employing a finetuned model designed to maintain an appropriate tone and style, including avoiding offensive language.
+    
+- Conducting additional validations at the application level to ensure output quality and appropriateness.
+    
+
+### Systemic Biases
+
+Language models obtain world-knowledge from their pre-training data and may therefore exhibit the same systematic biases that are present in the data. Differing deployment scenarios (including differing cultural contexts) can expose systematic biases in different ways. We acknowledge the cultural diversity of communities and users inside and outside the EU. For larger deployments, we encourage users to track systematic biases relevant to their use-case, and we are happy to consult on bespoke fine-tunings to alleviate such biases.
+
+### Outdated World Knowledge
+
+|   **Prompt**   |   **Completion**   |
+| --- | --- |
+|   What was the warmest year in human history?   | The warmest year in human history, based on global average temperatures, is 2016. However, it's important to note that the ranking of the warmest years can vary slightly depending on the dataset used and the methodology applied.  |
+
+Pre-training was performed using a fixed dataset, created at a fixed date in the past. Accordingly, the world knowledge of foundation models is limited to the information contained in its training data. More recent information may not be known to the model or misunderstood when presented as input during live usage. In the above example, the model is convinced that 2016 was the warmest year in human history, however, 2023 ended up breaking that record. This information can not be known by the model due to its knowledge cutoff.
+
+Risks include:
+
+- Generation of personally identifiable information. Models are not explicitly trained to provide such information, but may seem to provide personally identifiable information. This does not necessarily imply the presence of such information in training data, as hallucination is possible.
+    
+- Generation of unintended, irrelevant, or repetitive outputs. This includes the production of incorrect or outdated information.
+    
+
+Risks may be mitigated by:
+
+- Injecting context, where relevant.
+    
+- Crafting prompts carefully to guide the model's output more effectively.
+    
+- Performing validations on the application layer, e.g., classifying the output.
+    
+- Using the repetition penalty, especially in the case of repetition, or other parameters available in the API (see [documentation](https://docs.aleph-alpha.com/api/complete/)).
+    
+- Avoiding of use cases targeted at retrieval of personally identifiable information.
+    
+
+### Political Bias
+
+Our models have not been optimized to represent a political opinion or take a specific point of view. They may generate outputs that contradict a user's opinion or expectation, e.g., produce hateful, violent or inappropriate, biased, or discriminatory content. Such behavior may be addressed by:
+
+- Crafting prompts carefully to guide the model's output more effectively.
+    
+- Performing validations on the application layer, e.g., via Red-Teaming or classifying the output.
+    
+
+### Mistaken for a Human
+
+|   **Prompt**   |   **Completion**   |
+| --- | --- |
+|   Good morning, today is such a beautiful day!   |  Hello! It sounds like you're starting your day on a positive note. I'm glad to hear that the weather is beautiful. How are you feeling today? Is there anything specific you'd like to discuss or any tasks you'd like some help with? Whether it's planning your day, sharing some thoughts, or getting advice on something, I'm here to assist! Enjoy your day! |
+
+Users may attribute human traits to AI models. This also includes the fact that content generated by the model is not explicitly detectable at this point. It is therefore required to:
+
+- Inform end users that they are interacting with or reading output of an AI.
+    
+- Design the system in a way that mitigates the impact of unintended interpretation of the output.
+    
+
+### Other Errors
+
+Any AI module can produce errors, even after implementing all the recommended measures. When integrating foundation language models into an application, users should:
+
+- be aware of the risk of (harmful) failure cases and implement the use case in a way that mitigates such risks.
+    
+- be aware that foundation models do not contain application logic, e.g., content filters. Enforcement policies relevant to the use case need to be implemented in the application layer.
+    
+- avoid unsupervised use in high-stakes environments.
+    
+- validate output with adequate measures.
+    
+
+### Mitigation Approach
+
+We specifically tailor model alignment and risk mitigation techniques to each user-facing application built on top of our models, working closely with our customers to refine them according to their unique requirements. Our intention is for these models to undergo further fine-tuning by us and our customers, utilizing their own datasets alongside our support and datasets to ensure suitability for end-user applications, including harm mitigation efforts. Our customers are responsible for adhering to the terms and conditions when aligning the models in their downstream applications.
+
+### Reproducibility
+
+Some inference parameters, e.g., temperature, lead to the random sampling of outputs, which precludes the reproducibility of outputs. Even when such parameters are not in use, outputs may diverge slightly on a numeric level for technical reasons. One may implement the following measures if needed:
+
+- Logging of past model outputs on the application layer (Aleph Alpha Research is not storing any data and/or using any data provided in prompts for the training of its LLMs).
+    
+
+This list of risks, biases, and limitations may not be complete, as improving the understanding and behavior of language models is an ongoing research topic in the AI science community.
+
+
+\*Aleph Alpha Research refers to Aleph Alpha Research GmbH
+
+[hat-paper]: https://arxiv.org/abs/2501.10322

config.json

@@ -0,0 +1,100 @@
+{
+  "architectures": [
+    "HATForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "config.HATArchitectureConfig",
+    "AutoModelForCausalLM": "model.HATForCausalLM"
+  },
+  "backbone_config": {
+    "hidden_size": 4096,
+    "intermediate_size": 14336,
+    "is_neox_style": true,
+    "key_query_norm": true,
+    "key_query_norm_per_head": true,
+    "max_position_embeddings": 3500,
+    "mlp_bias": false,
+    "num_attention_heads": 32,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": {
+      "rope_type": "default"
+    },
+    "rope_theta": 500000,
+    "sliding_window": null,
+    "transformers_version": null,
+    "use_cache": true,
+    "vocab_size": 0
+  },
+  "decoder_config": {
+    "cross_attention_config": {
+      "attention_num_kv_heads": 8,
+      "hidden_size": 1024,
+      "hidden_size_kv": 4096,
+      "hidden_size_q": 1024,
+      "key_query_norm": true,
+      "key_query_norm_per_head": true,
+      "num_attention_heads": 8,
+      "word_window_size": 1
+    },
+    "cross_attn_every_layer": true,
+    "hidden_size": 1024,
+    "intermediate_size": 2816,
+    "is_neox_style": true,
+    "key_query_norm": true,
+    "key_query_norm_per_head": true,
+    "max_position_embeddings": 28000,
+    "mlp_bias": false,
+    "num_attention_heads": 8,
+    "num_hidden_layers": 4,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": {
+      "rope_type": "default"
+    },
+    "rope_theta": 100000,
+    "sliding_window": 768,
+    "transformers_version": null,
+    "use_cache": true,
+    "vocab_size": 256
+  },
+  "encoder_config": {
+    "cross_attention_config": {
+      "attention_num_kv_heads": 32,
+      "hidden_size": 4096,
+      "hidden_size_kv": 1024,
+      "hidden_size_q": 4096,
+      "key_query_norm": true,
+      "key_query_norm_per_head": true,
+      "num_attention_heads": 32,
+      "word_window_size": 1
+    },
+    "hidden_size": 1024,
+    "intermediate_size": 2816,
+    "is_neox_style": true,
+    "key_query_norm": true,
+    "key_query_norm_per_head": true,
+    "max_position_embeddings": 28000,
+    "mlp_bias": false,
+    "num_attention_heads": 8,
+    "num_hidden_layers": 6,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": {
+      "rope_type": "default"
+    },
+    "rope_theta": 100000,
+    "sliding_window": 768,
+    "transformers_version": null,
+    "use_cache": true,
+    "vocab_size": 256
+  },
+  "max_position_embeddings": 28000,
+  "max_word_size": 100,
+  "model_type": "hierarchical_autoregressive_transformer",
+  "sliding_window": 768,
+  "special_token_dict": {},
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.46.3"
+}

config.py

@@ -0,0 +1,262 @@
+from dataclasses import dataclass
+
+import torch.nn as nn
+from transformers.configuration_utils import PretrainedConfig
+from transformers.models.llama.configuration_llama import LlamaConfig
+
+
+@dataclass
+class TransformerHATModelConfig(LlamaConfig):
+    def __init__(
+        self,
+        hidden_size: int,
+        num_hidden_layers: int,
+        num_attention_heads: int,
+        num_key_value_heads: int,
+        rms_norm_eps: float,
+        intermediate_size: int,
+        max_position_embeddings: int,
+        rope_scaling: dict,
+        rope_theta: float,
+        mlp_bias: bool,
+        use_cache: bool = True,
+        sliding_window: int | None = None,
+        vocab_size: int = 0,
+        hidden_act: str = "silu",
+        key_query_norm: bool = False,
+        key_query_norm_per_head: bool = False,
+        is_neox_style: bool = True,
+        **kwargs,
+    ):
+        super().__init__(
+            vocab_size=vocab_size,
+            hidden_size=hidden_size,
+            num_hidden_layers=num_hidden_layers,
+            num_attention_heads=num_attention_heads,
+            num_key_value_heads=num_key_value_heads,
+            hidden_act=hidden_act,
+            rms_norm_eps=rms_norm_eps,
+            intermediate_size=intermediate_size,
+            max_position_embeddings=max_position_embeddings,
+            rope_scaling=rope_scaling,
+            rope_theta=rope_theta,
+            mlp_bias=mlp_bias,
+            use_cache=use_cache,
+            **kwargs,
+        )
+
+        self.sliding_window = sliding_window
+        self.key_query_norm = key_query_norm
+        self.key_query_norm_per_head = key_query_norm_per_head
+        self.is_neox_style = is_neox_style
+
+    def to_dict(self):
+        config_dict = {
+            "vocab_size": self.vocab_size,
+            "hidden_size": self.hidden_size,
+            "num_hidden_layers": self.num_hidden_layers,
+            "num_attention_heads": self.num_attention_heads,
+            "num_key_value_heads": self.num_key_value_heads,
+            "rms_norm_eps": self.rms_norm_eps,
+            "intermediate_size": self.intermediate_size,
+            "max_position_embeddings": self.max_position_embeddings,
+            "rope_scaling": self.rope_scaling,
+            "rope_theta": self.rope_theta,
+            "mlp_bias": self.mlp_bias,
+            "use_cache": self.use_cache,
+            "sliding_window": self.sliding_window,
+            "transformers_version": self.transformers_version,
+            "key_query_norm": self.key_query_norm,
+            "key_query_norm_per_head": self.key_query_norm_per_head,
+            "is_neox_style": self.is_neox_style,
+        }
+        return config_dict
+
+
+@dataclass
+class CrossAttentionConfig:
+    def __init__(
+        self,
+        hidden_size: int,
+        hidden_size_q: int,
+        hidden_size_kv: int,
+        num_attention_heads: int,
+        attention_num_kv_heads: int,
+        word_window_size: int,
+        key_query_norm: bool, 
+        key_query_norm_per_head: bool,
+    ):
+        self.hidden_size = hidden_size
+        self.hidden_size_q = hidden_size_q
+        self.hidden_size_kv = hidden_size_kv
+        self.num_attention_heads = num_attention_heads
+        self.attention_num_kv_heads = attention_num_kv_heads
+        self.word_window_size = word_window_size
+        self.key_query_norm = key_query_norm
+        self.key_query_norm_per_head = key_query_norm_per_head
+
+    def to_dict(self):
+        return {
+            "hidden_size_q": self.hidden_size_q,
+            "hidden_size_kv": self.hidden_size_kv,
+            "hidden_size": self.hidden_size,
+            "num_attention_heads": self.num_attention_heads,
+            "attention_num_kv_heads": self.attention_num_kv_heads,
+            "word_window_size": self.word_window_size,
+            "key_query_norm": self.key_query_norm,
+            "key_query_norm_per_head": self.key_query_norm_per_head,
+        }
+
+
+@dataclass
+class DecoderHATModelConfig(TransformerHATModelConfig):
+    def __init__(
+        self,
+        num_attention_heads: int,
+        num_key_value_heads: int,
+        sliding_window: int,
+        cross_attention_config: CrossAttentionConfig,
+        cross_attn_every_layer: bool,
+        **kwargs,
+    ):
+        super().__init__(
+            num_attention_heads=num_attention_heads,
+            num_key_value_heads=num_key_value_heads,
+            sliding_window=sliding_window,
+            **kwargs,
+        )
+        self.cross_attn_every_layer = cross_attn_every_layer
+        self.cross_attention_config = cross_attention_config
+
+    def to_dict(self):
+        config_dict = super().to_dict()
+        config_dict["cross_attn_every_layer"] = self.cross_attn_every_layer
+        config_dict["cross_attention_config"] = self.cross_attention_config.to_dict()
+        return config_dict
+
+    @classmethod
+    def from_dict(cls, config_dict, **kwargs):
+        config_dict = config_dict.copy()  # Avoid modifying the original dict
+        config_dict.update(kwargs)  # Apply overrides
+        dict_config = config_dict.pop("cross_attention_config", {})
+        cross_attention_config = CrossAttentionConfig(**dict_config)
+        config_dict["cross_attention_config"] = cross_attention_config
+        return cls(**config_dict)
+
+
+@dataclass
+class EncoderHATModelConfig(TransformerHATModelConfig):
+    def __init__(
+        self,
+        cross_attention_config: CrossAttentionConfig,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.cross_attention_config = cross_attention_config
+
+    @classmethod
+    def from_dict(cls, config_dict, **kwargs):
+        config_dict = config_dict.copy()  # Avoid modifying the original dict
+        config_dict.update(kwargs)  # Apply overrides
+        dict_config = config_dict.pop("cross_attention_config", {})
+        cross_attention_config = CrossAttentionConfig(**dict_config)
+        config_dict["cross_attention_config"] = cross_attention_config
+
+        return cls(**config_dict)
+
+    def to_dict(self):
+        config_dict = super().to_dict()
+        if self.cross_attention_config:
+            config_dict["cross_attention_config"] = self.cross_attention_config.to_dict()
+        return config_dict
+
+
+@dataclass
+class HATArchitectureConfig(PretrainedConfig):
+    model_type: str = "hierarchical_autoregressive_transformer"
+
+    def __init__(
+        self,
+        special_token_dict: dict | None = None,
+        encoder_config: EncoderHATModelConfig | None = None,
+        backbone_config: TransformerHATModelConfig | None = None,
+        decoder_config: DecoderHATModelConfig | None = None,
+        model_type: str = "hierarchical_autoregressive_transformer",
+        eos_token_id: int = 192,
+        max_word_size: int = 100,
+        sliding_window: int = 768,
+        max_position_embeddings: int = 262144,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.encoder_config = encoder_config
+        self.backbone_config = backbone_config
+        self.decoder_config = decoder_config
+        self.model_type = model_type
+        self.eos_token_id = eos_token_id
+        self.max_word_size = max_word_size
+        self.special_token_dict = special_token_dict
+        self.transformers_version = "4.46.3"
+
+        # set these for out of the box vllm inference
+        self.architectures = ["HATDecoderForCausalLM"]
+        self.sliding_window = sliding_window
+        self.max_position_embeddings = max_position_embeddings
+        self.torch_dtype = "bfloat16"
+
+    @classmethod
+    def from_dict(cls, config_dict, **kwargs):
+        """
+        Instantiates a HATArchitectureConfig from a Python dictionary of parameters.
+
+        Overrides the base `from_dict` to correctly handle nested config objects.
+        """
+        config_dict = config_dict.copy()  # Avoid modifying the original dict
+        config_dict.update(kwargs)  # Apply overrides
+
+        # Pop and instantiate nested config dictionaries
+        encoder_dict = config_dict.pop("encoder_config", {})
+        backbone_dict = config_dict.pop("backbone_config", {})
+        decoder_dict = config_dict.pop("decoder_config", {})
+
+        # Instantiate nested configs
+        encoder_config = EncoderHATModelConfig.from_dict(encoder_dict) if encoder_dict else None
+        backbone_config = TransformerHATModelConfig.from_dict(backbone_dict) if backbone_dict else None
+        decoder_config = DecoderHATModelConfig.from_dict(decoder_dict) if decoder_dict else None
+        special_token_dict = config_dict.pop("special_token_dict", {"<|eot_id|>": 192})
+        max_word_size = config_dict.pop("max_word_size", 100)
+        return cls(
+            encoder_config=encoder_config,
+            backbone_config=backbone_config,
+            decoder_config=decoder_config,
+            special_token_dict=special_token_dict,
+            max_word_size=max_word_size,
+            **config_dict,
+        ), {}
+
+    def to_dict(self):
+        config_dict = {}
+        if self.encoder_config:
+            config_dict["encoder_config"] = self.encoder_config.to_dict()
+        if self.backbone_config:
+            config_dict["backbone_config"] = self.backbone_config.to_dict()
+        if self.decoder_config:
+            config_dict["decoder_config"] = self.decoder_config.to_dict()
+        config_dict["model_type"] = self.model_type
+        config_dict["transformers_version"] = self.transformers_version
+        config_dict["auto_map"] = {"AutoConfig": "config.HATArchitectureConfig", "AutoModelForCausalLM": "model.HATForCausalLM"}
+        config_dict["special_token_dict"] = self.special_token_dict
+        
+        # print these out to the config for vllm
+        config_dict["max_word_size"] = self.max_word_size
+        config_dict["sliding_window"] = self.sliding_window
+        config_dict["max_position_embeddings"] = self.max_position_embeddings
+        config_dict["torch_dtype"] = self.torch_dtype
+        config_dict["architectures"] = self.architectures
+        return config_dict
+
+
+class EncoderHATModel(nn.Module):
+    def __init__(self, config: HATArchitectureConfig, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.config = config

config.yaml

@@ -0,0 +1,88 @@
+encoder_config:
+  vocab_size: 256
+  hidden_size: 1024
+  num_hidden_layers: 6
+  num_attention_heads: 8
+  num_key_value_heads: 8
+  rms_norm_eps: 1.0e-05
+  intermediate_size: 2816
+  max_position_embeddings: 28000
+  rope_scaling:
+    rope_type: default
+  rope_theta: 100000
+  mlp_bias: false
+  use_cache: true
+  sliding_window: 768
+  transformers_version: null
+  key_query_norm: true
+  key_query_norm_per_head: true
+  is_neox_style: true
+  cross_attention_config:
+    hidden_size_q: 4096
+    hidden_size_kv: 1024
+    hidden_size: 4096
+    num_attention_heads: 32
+    attention_num_kv_heads: 32
+    word_window_size: 1
+    key_query_norm: true
+    key_query_norm_per_head: true
+backbone_config:
+  vocab_size: 0
+  hidden_size: 4096
+  num_hidden_layers: 32
+  num_attention_heads: 32
+  num_key_value_heads: 8
+  rms_norm_eps: 1.0e-05
+  intermediate_size: 14336
+  max_position_embeddings: 3500
+  rope_scaling:
+    rope_type: default
+  rope_theta: 500000
+  mlp_bias: false
+  use_cache: true
+  sliding_window: null
+  transformers_version: null
+  key_query_norm: true
+  key_query_norm_per_head: true
+  is_neox_style: true
+decoder_config:
+  vocab_size: 256
+  hidden_size: 1024
+  num_hidden_layers: 4
+  num_attention_heads: 8
+  num_key_value_heads: 8
+  rms_norm_eps: 1.0e-05
+  intermediate_size: 2816
+  max_position_embeddings: 28000
+  rope_scaling:
+    rope_type: default
+  rope_theta: 100000
+  mlp_bias: false
+  use_cache: true
+  sliding_window: 768
+  transformers_version: null
+  key_query_norm: true
+  key_query_norm_per_head: true
+  is_neox_style: true
+  cross_attn_every_layer: true
+  cross_attention_config:
+    hidden_size_q: 1024
+    hidden_size_kv: 4096
+    hidden_size: 1024
+    num_attention_heads: 8
+    attention_num_kv_heads: 8
+    word_window_size: 1
+    key_query_norm: true
+    key_query_norm_per_head: true
+model_type: hierarchical_autoregressive_transformer
+transformers_version: 4.46.3
+auto_map:
+  AutoConfig: config.HATArchitectureConfig
+  AutoModelForCausalLM: model.HATForCausalLM
+special_token_dict: {}
+max_word_size: 100
+sliding_window: 768
+max_position_embeddings: 28000
+torch_dtype: bfloat16
+architectures:
+- HATDecoderForCausalLM

generation_config.json

@@ -0,0 +1,4 @@
+{
+  "_from_model_config": true,
+  "transformers_version": "4.46.3"
+}

model-00001-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b63eea8783a24fd114863084cb643599940cb7a53324b980f6f111ab264e2280
+size 4922789928

model-00002-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02e25bef21a7ba415e7e2c35c7c1f1f7e9813f6ad477bf41e4bbf6e34dab8b2c
+size 4832013880

model-00003-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4437c2f7f5145df1c4ba34c0652c3a8d84a42b31fff03edbf475d05b60b4cbec
+size 4999820824

model-00004-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2b92e4eaba076d7d18f6776983d4f2b89c2232a60136ec820a35374a0eb3777
+size 4999820832

model-00005-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e49586a2b7a12f36a76567739a430498ade7cb61be567df4047e73e1bc48e51
+size 4832013928

model-00006-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2582415526c586f8815750c56f5bec08198ab142c31dd7c175d5868b180a8245
+size 4014747208

model.py

@@ -0,0 +1,951 @@
+import itertools
+from collections.abc import Sequence
+from importlib.metadata import PackageNotFoundError, version
+from typing import Callable
+
+import torch
+import torch.nn as nn
+from einops import rearrange
+from flash_attn.flash_attn_interface import flash_attn_varlen_func
+from transformers import PreTrainedModel
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
+from transformers.utils import ModelOutput
+
+from .config import (
+    CrossAttentionConfig,
+    DecoderHATModelConfig,
+    EncoderHATModelConfig,
+    HATArchitectureConfig,
+    TransformerHATModelConfig,
+)
+from .splitter import HATSplitter
+from .norm import RMSNorm
+from .transformer_backbone import (
+    LlamaDecoderLayer,
+    LlamaRotaryEmbedding,
+)
+
+try:
+    transformers_version = version("transformers")
+    if transformers_version != "4.46.3":
+        print(f"Warning: Expecected transformers version 4.46.3, but found {transformers_version}. Outputs might be different.")
+except PackageNotFoundError:
+    print("transformers is not installed")
+
+
+def sample_argmax(logits: torch.Tensor) -> torch.Tensor:
+    return torch.argmax(logits, dim=-1)[:, -1]
+
+
+LLAMA_TEMPLATE = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>
+You are a helpful assistant. You give engaging, well-structured answers to user inquiries.<|eot_id|><|start_header_id|>user<|end_header_id|>
+{input}<|eot_id|><|start_header_id|>assistant<|end_header_id|>"""
+
+
+class HATCache(Cache):
+    encoder_cache: DynamicCache
+    backbone_cache: DynamicCache
+    decoder_cache: DynamicCache
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.encoder_cache = DynamicCache()
+        self.backbone_cache = DynamicCache()
+        self.decoder_cache = DynamicCache()
+
+    def get_backbone_cache(self) -> DynamicCache:
+        return self.backbone_cache
+
+    def get_decoder_cache(self) -> DynamicCache:
+        return self.decoder_cache
+
+    def get_encoder_cache(self) -> DynamicCache:
+        return self.encoder_cache
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(q, k, q_cos=None, q_sin=None, k_cos=None, k_sin=None, unsqueeze_dim=1):
+    """Applies Rotary Position Embedding to the query and key tensors.
+    and allows for different sequence lengths.
+    Args:
+        q (`torch.Tensor`): The query tensor.
+        k (`torch.Tensor`): The key tensor.
+        q_cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        q_sin (`torch.Tensor`): The sine part of the rotary embedding.
+        k_cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        k_sin (`torch.Tensor`): The sine part of the rotary embedding.
+        unsqueeze_dim (`int`, *optional*, defaults to 1):
+            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze
+            cos[position_ids] and sin[position_ids] so that they can be properly
+            broadcasted to the dimensions of q and k. For example, note
+            that cos[position_ids] and sin[position_ids] have the shape
+            [batch_size, seq_len, head_dim]. Then, if q and
+            k have the shape [batch_size, heads, seq_len, head_dim], then setting
+            unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids]
+            broadcastable to the shapes of q and k. Similarly, if q and k have
+            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
+    Returns:
+        `tuple(torch.Tensor)` comprising of the query and key
+        tensors rotated using the Rotary Position Embedding.
+    """
+
+    q_cos = q_cos.unsqueeze(unsqueeze_dim)
+    q_sin = q_sin.unsqueeze(unsqueeze_dim)
+    k_cos = k_cos.unsqueeze(unsqueeze_dim)
+    k_sin = k_sin.unsqueeze(unsqueeze_dim)
+    q_embed = (q * q_cos) + (rotate_half(q) * q_sin)
+    k_embed = (k * k_cos) + (rotate_half(k) * k_sin)
+
+    return q_embed, k_embed
+
+
+class HATBackbone(nn.Module):
+    def __init__(self, config: TransformerHATModelConfig, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+        self.layers = nn.ModuleList([LlamaDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)])
+        self.rotary_emb = LlamaRotaryEmbedding(config=config)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_ids: torch.Tensor | None = None,
+        past_key_values: DynamicCache | None = None,
+        use_cache: bool | None = False,
+    ) -> BaseModelOutputWithPast:
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache()
+
+        if position_ids is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            position_ids = torch.arange(
+                past_seen_tokens,
+                past_seen_tokens + hidden_states.shape[1],
+                device=hidden_states.device,
+            ).unsqueeze(0)
+
+        # create position embeddings to be shared across the decoder layers
+        position_embeddings = self.rotary_emb(hidden_states, position_ids)
+
+        for backbone_layer in self.layers:
+            layer_outputs = backbone_layer(
+                hidden_states,
+                position_ids=position_ids,
+                past_key_value=past_key_values,
+                use_cache=use_cache,
+                position_embeddings=position_embeddings,
+            )
+            hidden_states = layer_outputs[0]
+
+        return CausalLMOutputWithPast(
+            hidden_states=hidden_states,
+            past_key_values=past_key_values if use_cache else None,
+        )
+
+
+class HATDecoderConnector(nn.Module):
+    def __init__(self, backbone_hiden_dim: int, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.first_word_embedding = torch.nn.Parameter(
+            torch.empty(
+                1,
+                1,
+                backbone_hiden_dim,
+                device="cuda",
+                dtype=torch.bfloat16,
+            )
+        )
+
+    def forward(
+        self,
+        backbone_activations: torch.Tensor,
+    ):
+        activations = backbone_activations.clone()
+        activations[:, -1:, :] = self.first_word_embedding
+        activations = torch.roll(activations, shifts=1, dims=1)
+        return activations
+
+
+class HATDecoderBlock(nn.Module):
+    def __init__(
+        self,
+        add_cross_attention: bool,
+        config: DecoderHATModelConfig,
+        layer_idx: int,
+        *args,
+        **kwargs,
+    ):
+        super().__init__(*args, **kwargs)
+        self.add_cross_attention = add_cross_attention
+        self.config = config
+        self.llama_layer = LlamaDecoderLayer(config, layer_idx)
+        self.llama_layer.self_attn.sliding_window = config.sliding_window
+        if add_cross_attention:
+            self.cross_attention = HATCrossAttention(
+                hidden_size=config.cross_attention_config.hidden_size,
+                hidden_size_kv=config.cross_attention_config.hidden_size_kv,
+                hidden_size_q=config.cross_attention_config.hidden_size_q,
+                config=config,
+                cross_attention_config=config.cross_attention_config,
+            )
+
+        self.query_norm = RMSNorm(
+            config.cross_attention_config.hidden_size_q,
+            eps=config.rms_norm_eps,
+            device=torch.device("cuda"),
+            dtype=torch.bfloat16,
+            norm_in_fp32=False,
+        )
+
+        self.kv_norm = RMSNorm(
+            config.cross_attention_config.hidden_size_kv,
+            eps=config.rms_norm_eps,
+            device=torch.device("cuda"),
+            dtype=torch.bfloat16,
+            norm_in_fp32=False,
+        )
+
+    def apply_norm(self, activations):
+        return self.query_norm(activations), self.kv_norm(activations)
+
+    def forward(
+        self,
+        encoder_activations,
+        backbone_activations,
+        byte_position_ids,
+        word_position_ids,
+        cumulative_seq_lengths_per_word,
+        position_embeddings,
+        past_key_values,
+        use_cache,
+    ):
+        if self.add_cross_attention:
+            kv_activations = self.kv_norm(backbone_activations)
+            q_activations = self.query_norm(encoder_activations)
+
+            activations = self.cross_attention.forward(
+                q_activations=q_activations,
+                kv_activations=kv_activations,
+                position_ids_q=byte_position_ids,
+                position_ids_kv=word_position_ids,
+                cumulative_seq_q=cumulative_seq_lengths_per_word,
+                cumulative_seq_kv=torch.arange(0, kv_activations.size(1) + 1, device=encoder_activations.device, dtype=torch.int32),
+                causal=False,
+            )
+            encoder_activations = encoder_activations + activations
+
+        return self.llama_layer.forward(
+            hidden_states=encoder_activations,
+            position_ids=byte_position_ids,
+            position_embeddings=position_embeddings,
+            past_key_value=past_key_values,
+            use_cache=use_cache,
+        )[0]
+
+
+class HATDecoder(nn.Module):
+    def __init__(self, config: DecoderHATModelConfig, *args, **kwargs):
+        super().__init__()
+
+        self.decoder_layers = nn.Sequential()
+        for layer_idx in range(config.num_hidden_layers):
+            add_cross_attention = config.cross_attn_every_layer or layer_idx == 0
+            self.decoder_layers.add_module(
+                str(layer_idx),
+                HATDecoderBlock(
+                    add_cross_attention,
+                    config,
+                    layer_idx,
+                ),
+            )
+
+        self.rotary_emb = LlamaRotaryEmbedding(config=config)
+
+    def forward(
+        self,
+        backbone_activations: torch.Tensor,
+        activations: torch.Tensor,
+        cumulative_seq_lengths_per_word: torch.Tensor | None = None,
+        byte_position_ids: torch.Tensor | None = None,
+        word_position_ids: torch.Tensor | None = None,
+        past_key_values: DynamicCache | None = None,
+        use_cache: bool | None = False,
+    ) -> BaseModelOutputWithPast:
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache()
+
+        if byte_position_ids is None:
+            past_seen_bytes = past_key_values.get_seq_length() if past_key_values is not None else 0
+            byte_position_ids = torch.arange(
+                past_seen_bytes,
+                past_seen_bytes + activations.size(1),
+                device=activations.device,
+                dtype=torch.int32,
+            ).unsqueeze(0)
+
+        if cumulative_seq_lengths_per_word is None:
+            cumulative_seq_lengths_per_word = torch.tensor([0, byte_position_ids.size(1)], dtype=byte_position_ids.dtype, device=byte_position_ids.device)
+
+        if word_position_ids is None:
+            raise ValueError()  # TODO
+
+        position_embeddings = self.rotary_emb(activations, byte_position_ids)
+
+        for _, layer in enumerate(self.decoder_layers):
+            activations = layer(
+                encoder_activations=activations,
+                backbone_activations=backbone_activations,
+                position_embeddings=position_embeddings,
+                cumulative_seq_lengths_per_word=cumulative_seq_lengths_per_word,
+                byte_position_ids=byte_position_ids,
+                word_position_ids=word_position_ids,
+                past_key_values=past_key_values,
+                use_cache=use_cache,
+            )
+
+        return BaseModelOutputWithPast(
+            last_hidden_state=activations,
+            past_key_values=past_key_values if use_cache else None,
+        )
+
+
+class HATCrossAttention(nn.Module):
+    def __init__(
+        self,
+        hidden_size: int,
+        hidden_size_q: int,
+        hidden_size_kv: int,
+        config: EncoderHATModelConfig | DecoderHATModelConfig,
+        cross_attention_config: CrossAttentionConfig,
+        dtype: torch.dtype = torch.bfloat16,
+    ):
+        super().__init__()
+        self.hidden_size = hidden_size
+        self.hidden_size_q = hidden_size_q
+        self.hidden_size_kv = hidden_size_kv
+        self.num_heads = cross_attention_config.num_attention_heads
+        self.num_key_value_heads = cross_attention_config.attention_num_kv_heads
+        self.num_repeat_kv = cross_attention_config.num_attention_heads // cross_attention_config.attention_num_kv_heads
+        self.head_dim = hidden_size // self.num_heads
+        self.key_query_norm = cross_attention_config.key_query_norm
+        self.key_query_norm_per_head = cross_attention_config.key_query_norm_per_head
+
+        self.q_proj = nn.Linear(
+            in_features=hidden_size_q,
+            out_features=hidden_size,
+            dtype=dtype,
+            bias=False,
+        )
+
+        self.k_proj = nn.Linear(
+            in_features=hidden_size_kv,
+            out_features=hidden_size // self.num_repeat_kv,
+            dtype=dtype,
+            bias=False,
+        )
+
+        self.v_proj = nn.Linear(
+            in_features=hidden_size_kv,
+            out_features=hidden_size // self.num_repeat_kv,
+            dtype=dtype,
+            bias=False,
+        )
+
+        if self.key_query_norm:
+            if self.key_query_norm_per_head:
+                # Both query and key have head dim equal to self.hidden_size_per_attention_head
+                query_norm_dimensions = self.head_dim
+                key_norm_dimensions = self.head_dim
+            else:
+                # Query dimensions across head is equal to hidden_size but key dimensions are divided
+                # by self.num_repeat_kv
+                query_norm_dimensions = self.hidden_size
+                key_norm_dimensions = self.hidden_size // self.num_repeat_kv
+
+            self.norm_query = RMSNorm(
+                dimensions=query_norm_dimensions,
+                eps=config.rms_norm_eps,
+                device=self.q_proj.weight.device,
+                dtype=dtype,
+            )
+            self.norm_key = RMSNorm(
+                dimensions=key_norm_dimensions,
+                eps=config.rms_norm_eps,
+                device=self.q_proj.weight.device,
+                dtype=dtype,
+            )
+
+        self.o_proj = nn.Linear(in_features=hidden_size, out_features=hidden_size_q, dtype=dtype, bias=False)
+
+        rope_theta = config.rope_theta
+        rope_type = config.rope_scaling["rope_type"]
+
+        self.rotary_emb = LlamaRotaryEmbedding(dim=self.head_dim, base=rope_theta, rope_type=rope_type)
+
+    def forward(
+        self,
+        q_activations: torch.Tensor,
+        kv_activations: torch.Tensor,
+        position_ids_q: torch.Tensor,
+        position_ids_kv: torch.Tensor,
+        cumulative_seq_kv: torch.Tensor,
+        cumulative_seq_q: torch.Tensor,
+        causal: bool = True,
+        use_cache: bool = False,
+        past_key_value: DynamicCache | None = None,
+    ):
+        q_len = cumulative_seq_q[-1]
+
+        bsz, _, _ = kv_activations.size()
+        query_states = self.q_proj(q_activations)
+        key_states = self.k_proj(kv_activations)
+        value_states = self.v_proj(kv_activations)
+
+        if self.key_query_norm:
+            assert self.norm_query is not None
+            assert self.norm_key is not None
+            # query_states and key_states are bsz seq_len (h d)
+            if self.key_query_norm_per_head:
+                # for per head qk norm we need head dim to be the last dim
+                query_states = rearrange(
+                    query_states,
+                    "bsz seq_len (h d) -> bsz seq_len h d",
+                    h=self.num_heads,
+                )
+                key_states = rearrange(
+                    key_states,
+                    "bsz seq_len (h d) -> bsz seq_len h d",
+                    h=self.num_key_value_heads,
+                )
+            query_states = self.norm_query(query_states)
+            key_states = self.norm_key(key_states)
+            if self.key_query_norm_per_head:
+                query_states = rearrange(
+                    query_states,
+                    "bsz seq_len h d -> bsz seq_len (h d)",
+                )
+                key_states = rearrange(
+                    key_states,
+                    "bsz seq_len h d -> bsz seq_len (h d)",
+                )
+
+        # TODO get rid of the double rearrange, this is just for compatibility with scaling
+        query_states = rearrange(query_states, "bsz seq_len (h d) -> bsz h seq_len d", h=self.num_heads)
+        key_states = rearrange(
+            key_states,
+            "bsz seq_len (h d) -> bsz h seq_len d",
+            h=self.num_key_value_heads,
+        )
+        value_states = rearrange(
+            value_states,
+            "bsz seq_len (h d) -> bsz h seq_len d",
+            h=self.num_key_value_heads,
+        )
+
+        # WIP: Should word_positions_id respect document boundaries?
+        q_cos, q_sin = self.rotary_emb(query_states, position_ids_q)
+        k_cos, k_sin = self.rotary_emb(key_states, position_ids_kv)
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, q_cos=q_cos, q_sin=q_sin, k_cos=k_cos, k_sin=k_sin)
+
+        query_states = rearrange(query_states, "bsz h seq_len d -> (bsz seq_len) h d")
+        key_states = rearrange(key_states, "bsz h seq_len d -> (bsz seq_len) h d")
+        value_states = rearrange(value_states, "bsz h seq_len d -> (bsz seq_len) h d")
+
+        attn_output = flash_attn_varlen_func(
+            query_states,
+            key_states,
+            value_states,
+            cu_seqlens_q=cumulative_seq_q,
+            cu_seqlens_k=cumulative_seq_kv,
+            max_seqlen_q=self._get_max_seqlen(cumulative_seq_q),
+            max_seqlen_k=self._get_max_seqlen(cumulative_seq_kv),
+            causal=False,
+        )
+
+        attn_output = attn_output.reshape(bsz, q_len, -1).contiguous()
+
+        attn_output = self.o_proj(attn_output)
+        return attn_output
+
+    def _get_max_seqlen(self, cumulative_word_lengths: torch.Tensor):
+        diffs = cumulative_word_lengths[1:] - cumulative_word_lengths[:-1]
+        return int(diffs.max().item())
+
+
+class HATEncoderConnector(nn.Module):
+    def __init__(
+        self,
+        config: EncoderHATModelConfig,
+        backbone_hidden_size: int,
+        dtype: torch.dtype = torch.bfloat16,
+        *args,
+        **kwargs,
+    ):
+        super().__init__(*args, **kwargs)
+        self.latent_query = torch.nn.Parameter(
+            torch.empty(
+                1,
+                1,
+                backbone_hidden_size,
+                device="cuda",
+                dtype=dtype,
+            )
+        )
+
+        self.cross_attention_encoder_connector = HATCrossAttention(
+            hidden_size=config.cross_attention_config.hidden_size,
+            hidden_size_q=backbone_hidden_size,
+            hidden_size_kv=config.hidden_size,
+            config=config,
+            cross_attention_config=config.cross_attention_config,
+        )
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        cumulative_seq_lengths_per_word: torch.Tensor,
+        word_position_ids: torch.Tensor,
+        byte_position_ids: torch.Tensor,
+    ):
+        q_len = cumulative_seq_lengths_per_word.shape[0] - 1
+        latent_query_repeated = self.latent_query.expand(-1, q_len, -1)
+        cumulative_seq_lengths_q = torch.arange(
+            start=0,
+            end=latent_query_repeated.shape[1] + 1,
+            step=1,
+            device=self.latent_query.device,
+            dtype=torch.int32,
+        )
+        word_embeddings = self.cross_attention_encoder_connector.forward(
+            q_activations=latent_query_repeated,
+            kv_activations=hidden_states,
+            position_ids_q=word_position_ids,
+            position_ids_kv=byte_position_ids,
+            cumulative_seq_q=cumulative_seq_lengths_q,
+            cumulative_seq_kv=cumulative_seq_lengths_per_word,
+        )
+        return word_embeddings
+
+
+class HATEncoder(nn.Module):
+    def __init__(
+        self,
+        config: EncoderHATModelConfig,
+        dtype: torch.dtype = torch.bfloat16,
+        *args,
+        **kwargs,
+    ):
+        super().__init__(*args, **kwargs)
+        self.embedding_layer = nn.Embedding(config.vocab_size, config.hidden_size, dtype=dtype)
+        self.layers = nn.ModuleList([LlamaDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)])
+        for layer in self.layers:
+            layer.self_attn.sliding_window = config.sliding_window
+
+        self.rotary_emb = LlamaRotaryEmbedding(config=config)
+
+        self.word_window_size = config.cross_attention_config.word_window_size
+
+    def forward(
+        self,
+        input_ids: torch.Tensor,
+        cumulative_seq_lengths_per_word: torch.Tensor | None = None,
+        byte_position_ids: torch.Tensor | None = None,
+        word_position_ids: torch.Tensor | None = None,  # TODO: Remove
+        past_key_values: DynamicCache | None = None,
+        use_cache: bool | None = False,
+    ):
+        input_embeds = self.embedding_layer(input_ids)
+
+        if cumulative_seq_lengths_per_word is None:
+            cumulative_seq_lengths_per_word = torch.tensor([0, input_embeds.shape[1]], dtype=torch.int32, device=input_ids.device)
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache()
+
+        if byte_position_ids is None:
+            past_seen_bytes = past_key_values.get_seq_length() if past_key_values is not None else 0
+            byte_position_ids = torch.arange(
+                past_seen_bytes,
+                past_seen_bytes + input_embeds.shape[1],
+                device=input_embeds.device,
+            ).unsqueeze(0)
+
+        if word_position_ids is None:
+            raise ValueError()  # TODO
+
+        hidden_states = input_embeds
+
+        # create position embeddings to be shared across the decoder layers
+        position_embeddings = self.rotary_emb(hidden_states, byte_position_ids)
+
+        for layer in self.layers:
+            layer_outputs = layer(
+                hidden_states,
+                position_ids=byte_position_ids,
+                past_key_value=past_key_values,
+                use_cache=use_cache,
+                position_embeddings=position_embeddings,
+            )
+            hidden_states = layer_outputs[0]
+
+        return CausalLMOutputWithPast(
+            hidden_states=hidden_states,
+            past_key_values=past_key_values if use_cache else None,
+        )
+
+
+class HATForCausalLM(PreTrainedModel):
+    config_class = HATArchitectureConfig
+    _supports_flash_attn_2 = True
+    _supports_cache_class = True
+
+    def __init__(self, config: HATArchitectureConfig, *args, **kwargs):
+        super().__init__(config, *args, **kwargs)
+        self.config = config
+        self.eos_token_id = config.eos_token_id
+        self.encoder = HATEncoder(config.encoder_config)
+        self.encoder_connector = HATEncoderConnector(config.encoder_config, config.backbone_config.hidden_size)
+        self.backbone = HATBackbone(config.backbone_config)
+        self.decoder_connector = HATDecoderConnector(config.backbone_config.hidden_size)
+        self.decoder = HATDecoder(config.decoder_config)
+        self.splitter = HATSplitter(special_token_dict=config.special_token_dict, max_word_size=config.max_word_size)
+        self.layer_norm = RMSNorm(config.decoder_config.hidden_size, eps=config.decoder_config.rms_norm_eps, device=torch.device("cuda"), dtype=torch.bfloat16, norm_in_fp32=False)
+        self.lm_head = nn.Linear(
+            in_features=config.decoder_config.hidden_size,
+            out_features=config.decoder_config.vocab_size,
+            dtype=torch.bfloat16,
+            bias=False,
+        )
+
+    def forward(
+        self,
+        input_ids: torch.Tensor,
+        byte_position_ids: torch.Tensor,
+        cumulative_seq_lengths_per_word: torch.Tensor | None = None,
+        word_position_ids: torch.Tensor | None = None,
+        past_key_values: HATCache | None = None,
+        use_cache: bool = False,
+    ):
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+
+        if past_key_values is None and use_cache:
+            past_key_values = HATCache()
+
+        encoder_past_key_values = past_key_values.get_encoder_cache() if past_key_values is not None else None
+        backbone_past_key_values = past_key_values.get_backbone_cache() if past_key_values is not None else None
+        decoder_past_key_values = past_key_values.get_decoder_cache() if past_key_values is not None else None
+
+        encoder_output: BaseModelOutputWithPast = self.encoder(
+            input_ids=input_ids,
+            cumulative_seq_lengths_per_word=cumulative_seq_lengths_per_word,
+            byte_position_ids=byte_position_ids,
+            word_position_ids=word_position_ids,
+            past_key_values=encoder_past_key_values,
+            use_cache=use_cache,
+        )
+        byte_level_activations = encoder_output.hidden_states
+
+        encoder_connector_output = self.encoder_connector(
+            byte_level_activations,
+            cumulative_seq_lengths_per_word,
+            word_position_ids,
+            byte_position_ids,
+        )
+        backbone_output: CausalLMOutputWithPast = self.backbone(
+            hidden_states=encoder_connector_output,
+            position_ids=word_position_ids,
+            past_key_values=backbone_past_key_values,
+            use_cache=use_cache,
+        )
+
+        predictive_word_embeddings = self.decoder_connector.forward(backbone_activations=backbone_output.hidden_states)
+
+        decoder_output = self.decoder.forward(
+            activations=byte_level_activations,
+            backbone_activations=predictive_word_embeddings,
+            cumulative_seq_lengths_per_word=cumulative_seq_lengths_per_word,
+            byte_position_ids=byte_position_ids,
+            word_position_ids=word_position_ids,
+            past_key_values=decoder_past_key_values,
+            use_cache=use_cache,
+        )
+
+        decoder_output = self.layer_norm(decoder_output.last_hidden_state)
+        logits = self.lm_head(decoder_output)
+
+        loss = None
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=past_key_values if use_cache else None,
+            hidden_states=backbone_output.hidden_states,
+            attentions=None,
+        )
+
+    def _append_byte(self, words: list[list[int]], token: int) -> list[list[int]]:
+        extended_last_word = words.pop() + [token]
+        try:
+            text = self.splitter.decode(extended_last_word, errors='strict', skip_special_tokens=False)
+            list_of_bytes = self.splitter.encode(text)
+            words.extend([list(word_in_bytes) for word_in_bytes in list_of_bytes])
+        except UnicodeDecodeError:
+            # if decoding fails, the token cannot be part of a new word since it is not a valid
+            # utf-8 end byte and we append it to the current word
+            words.append(extended_last_word)
+        return words
+
+    def _complete_word(
+        self,
+        input_ids: torch.Tensor,
+        byte_position_ids: torch.Tensor,
+        backbone_word_prediction: torch.Tensor,
+        word_position_id: torch.Tensor,
+        encoder_cache: DynamicCache,
+        decoder_cache: DynamicCache,
+        sample_fn: Callable[[torch.Tensor], torch.Tensor] = sample_argmax,
+    ):
+        """Generate byte tokens until we hit the first byte of a new word."""
+        words = [input_ids.squeeze(0).tolist()]
+        byte_encoder_activations = []
+        completion_logits = []
+
+        while True:
+            encoder_output = self.encoder.forward(
+                input_ids,
+                byte_position_ids=None,
+                word_position_ids=word_position_id,
+                past_key_values=encoder_cache,
+                use_cache=True,
+            )
+            byte_encoder_activations.append(encoder_output.hidden_states)
+            decoder_output = self.decoder.forward(
+                backbone_word_prediction,
+                encoder_output.hidden_states,
+                byte_position_ids=None,
+                word_position_ids=word_position_id,
+                past_key_values=decoder_cache,
+                use_cache=True,
+            )
+            decoder_output = self.layer_norm(decoder_output.last_hidden_state)
+            logits = self.lm_head(decoder_output)
+            completion_logits.append(logits[0, -1:, :])
+            next_byte = int(sample_fn(logits).item())
+            words = self._append_byte(words, next_byte)
+            if len(words) > 1 or next_byte == self.eos_token_id:
+                break
+            input_ids = torch.tensor([[next_byte]], dtype=input_ids.dtype, device=input_ids.device)
+
+        byte_encoder_activations = torch.cat(byte_encoder_activations, dim=1)
+        num_kv = encoder_cache.get_seq_length()
+        byte_position_ids = torch.arange(num_kv + 1 - byte_encoder_activations.shape[1], num_kv + 1, device=input_ids.device, dtype=torch.long).unsqueeze(0)
+        completed_word_embedding = self.encoder_connector.forward(
+            byte_encoder_activations,
+            cumulative_seq_lengths_per_word=torch.tensor([0, byte_encoder_activations.size(1)], dtype=torch.int32, device=input_ids.device),
+            word_position_ids=word_position_id,
+            byte_position_ids=byte_position_ids,
+        )
+
+        completion = sum(words, [])[-len(completion_logits) :]
+        first_byte_of_next_word = words[1]
+        return completion, completed_word_embedding, first_byte_of_next_word, byte_position_ids[:, -1].item() + 1, completion_logits
+
+    def generate(
+        self,
+        input_ids: torch.Tensor,
+        max_new_tokens: int,
+        cumulative_seq_lengths_per_word: torch.Tensor,
+        byte_position_ids: torch.Tensor | None = None,
+        word_position_ids: torch.Tensor | None = None,
+        sample_fn: Callable[[torch.Tensor], torch.Tensor] = sample_argmax,
+        use_cache: bool = True,
+        stop_sequences: Sequence[str] | None = None,
+    ):
+        if use_cache:
+            completion_text, completion_logits = self._generate_cached(input_ids, max_new_tokens, cumulative_seq_lengths_per_word, byte_position_ids, word_position_ids, sample_fn, stop_sequences=stop_sequences)
+        else:
+            completion_text, completion_logits = self._generate_uncached(input_ids, max_new_tokens, cumulative_seq_lengths_per_word, byte_position_ids, word_position_ids, sample_fn, stop_sequences=stop_sequences)
+
+        # remove stop sequence if exists
+        if stop_sequences is not None:
+            stop_sequences = sorted(stop_sequences, key=lambda i: len(i), reverse=True)
+            for stop_sequence in stop_sequences:
+                if stop_sequence in completion_text:
+                    completion_text_left = completion_text.split(stop_sequence)[0]
+                    completion_text_removed = completion_text[len(completion_text_left) :]
+
+                    completion_logits = completion_logits[: -len(list(bytes(completion_text_removed.encode("UTF-8"))))]
+                    completion_text = completion_text_left
+                    break
+
+        return ModelOutput(
+            completion_text=completion_text,
+            input_ids=input_ids,
+            completion_logits=completion_logits,
+        )
+
+    @torch.no_grad()
+    def _generate_cached(
+        self,
+        input_ids: torch.Tensor,
+        max_new_tokens: int,
+        cumulative_seq_lengths_per_word: torch.Tensor,
+        byte_position_ids: torch.Tensor | None = None,
+        word_position_ids: torch.Tensor | None = None,
+        sample_fn: Callable[[torch.Tensor], torch.Tensor] = sample_argmax,
+        stop_sequences: Sequence[str] | None = None,
+    ):
+        max_total_bytes = max_new_tokens + input_ids.shape[1]
+        if byte_position_ids is None:
+            byte_position_ids = torch.arange(0, cumulative_seq_lengths_per_word[-1].item(), device=input_ids.device, dtype=torch.int32).unsqueeze(0)
+
+        if word_position_ids is None:
+            word_position_ids = torch.arange(0, cumulative_seq_lengths_per_word.shape[0] - 1, device=input_ids.device, dtype=torch.int32).unsqueeze(0)
+
+        last_word_start, last_word_end = (
+            cumulative_seq_lengths_per_word[-2],
+            cumulative_seq_lengths_per_word[-1],
+        )
+        # Populate cache with everything except last word
+        initial_forward_output = self.forward(
+            input_ids=input_ids[:, :last_word_start],
+            cumulative_seq_lengths_per_word=cumulative_seq_lengths_per_word[:-1],
+            byte_position_ids=byte_position_ids[:, :last_word_start],
+            word_position_ids=word_position_ids[:, :-1],
+            past_key_values=None,
+            use_cache=True,
+        )
+
+        completion_bytes = []
+        completion_logits = []
+        input_ids = input_ids[:, last_word_start:last_word_end]
+        next_byte_id = last_word_end
+        byte_position_ids = byte_position_ids[:, last_word_start:last_word_end]
+        word_position_id = word_position_ids[:, -1].unsqueeze(-1)
+        backbone_last_hidden_state = initial_forward_output.hidden_states[:, -1:, :]
+        while next_byte_id < max_total_bytes:
+            completion, completed_word_embedding, first_byte_of_next_word, next_byte_id, next_completion_logits = self._complete_word(
+                input_ids=input_ids,
+                byte_position_ids=byte_position_ids,
+                backbone_word_prediction=backbone_last_hidden_state,
+                word_position_id=word_position_id,
+                encoder_cache=initial_forward_output.past_key_values.get_encoder_cache(),
+                decoder_cache=initial_forward_output.past_key_values.get_decoder_cache(),
+                sample_fn=sample_fn,
+            )
+            completion_logits.extend(next_completion_logits)
+            completion_bytes.extend(completion)
+
+            if self.eos_token_id in completion_bytes:
+                completion_bytes = completion_bytes[: completion_bytes.index(self.eos_token_id)]
+                break
+
+            if stop_sequences is not None:
+                try:
+                    completion_text_tmp = self.splitter.decode(completion_bytes)
+                    if any(stop_sequence in completion_text_tmp for stop_sequence in stop_sequences):
+                        break
+                except Exception as e:
+                    print("Cannot compare stop sequence", e)
+
+            backbone_output = self.backbone.forward(
+                hidden_states=completed_word_embedding,
+                position_ids=None,
+                past_key_values=initial_forward_output.past_key_values.get_backbone_cache(),
+                use_cache=True,
+            )
+            backbone_last_hidden_state = backbone_output.hidden_states[:, -1, :].unsqueeze(1)
+
+            input_ids = torch.tensor([first_byte_of_next_word], dtype=input_ids.dtype, device=input_ids.device)
+            byte_position_ids = torch.tensor([[next_byte_id]], dtype=input_ids.dtype, device=input_ids.device)
+            word_position_id = word_position_id + 1
+
+        completion_bytes.extend(first_byte_of_next_word)
+        completion_bytes = completion_bytes[:max_new_tokens]
+        completion_logits = torch.cat(completion_logits[:max_new_tokens], dim=0)
+        completion_text = self.splitter.decode(completion_bytes)
+
+        return completion_text, completion_logits
+
+    @torch.no_grad()
+    def _generate_uncached(
+        self,
+        input_ids: torch.Tensor,
+        max_new_tokens: int,
+        cumulative_seq_lengths_per_word: torch.Tensor,
+        byte_position_ids: torch.Tensor | None = None,
+        word_position_ids: torch.Tensor | None = None,
+        sample_fn=sample_argmax,
+        stop_sequences: Sequence[str] | None = None,
+    ):
+        if byte_position_ids is None:
+            byte_position_ids = torch.arange(0, cumulative_seq_lengths_per_word[-1].item(), device=input_ids.device, dtype=torch.int32).unsqueeze(0)
+
+        if word_position_ids is None:
+            word_position_ids = torch.arange(0, cumulative_seq_lengths_per_word.shape[0] - 1, device=input_ids.device, dtype=torch.int32).unsqueeze(0)
+
+        word_list = []
+        for i in range(1, cumulative_seq_lengths_per_word.shape[0]):
+            start_idx = cumulative_seq_lengths_per_word[i - 1]
+            end_idx = cumulative_seq_lengths_per_word[i]
+            word_list.append(input_ids[:, start_idx:end_idx].squeeze(0).tolist())
+
+        completion_bytes = []
+        for _ in range(max_new_tokens):
+            output = self.forward(
+                input_ids=input_ids,
+                cumulative_seq_lengths_per_word=cumulative_seq_lengths_per_word,
+                byte_position_ids=byte_position_ids,
+                word_position_ids=word_position_ids,
+                past_key_values=None,
+            )
+
+            next_byte = int(sample_fn(output.logits).item())
+            completion_bytes.append(next_byte)
+            if next_byte == self.eos_token_id:
+                break
+            word_list = self._append_byte(word_list, next_byte)
+
+            input_ids = torch.tensor(sum(word_list, []), dtype=torch.long, device=input_ids.device).unsqueeze(0)
+            cumulative_seq_lengths_per_word = torch.tensor([0] + list(itertools.accumulate(len(word) for word in word_list if len(word) > 0)), dtype=torch.int32, device=input_ids.device)
+            byte_position_ids = torch.arange(0, input_ids.shape[1], device=input_ids.device, dtype=torch.int32).unsqueeze(0)
+            word_position_ids = torch.arange(0, cumulative_seq_lengths_per_word.shape[0] - 1, device=input_ids.device, dtype=torch.int32).unsqueeze(0)
+
+            if stop_sequences is not None:
+                try:
+                    completion_text_tmp = self.splitter.decode(completion_bytes)
+                    if any(completion_text_tmp.endswith(stop_sequence) for stop_sequence in stop_sequences):
+                        break
+                except Exception as e:
+                    print("Cannot compare stop sequence", e)
+
+        completion_text = self.splitter.decode(completion_bytes)
+        completion_logits = output.logits[0, -len(completion_bytes) :, :]
+
+        return completion_text, completion_logits
+
+    def _prepare_input(self, input_str: str, add_llama_template: bool = True, device: torch.device | None = None) -> tuple[torch.Tensor, torch.Tensor]:
+        if add_llama_template:
+            input_str = LLAMA_TEMPLATE.format(input=input_str)
+
+        if device is None:
+            assert torch.cuda.is_available(), "CUDA is not available"
+            device = torch.device("cuda")
+        input_ids_list = []
+        cumulative_per_word_lengths_list = [0]
+
+        words = self.splitter.encode(input_str)
+        for word in words:
+            input_ids_list.extend(word)
+            word_length = len(word)
+            cumulative_per_word_lengths_list.append(cumulative_per_word_lengths_list[-1] + word_length)
+        input_ids = torch.tensor(input_ids_list, device=device, dtype=torch.int32).unsqueeze(0)
+        cumulative_per_word_lengths = torch.tensor(cumulative_per_word_lengths_list, device=device, dtype=torch.int32)
+        return input_ids, cumulative_per_word_lengths

model.safetensors.index.json

@@ -0,0 +1,512 @@
+{
+  "metadata": {
+    "total_size": 28601145856
+  },
+  "weight_map": {
+    "backbone.layers.0.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.0.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.0.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.0.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.0.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.0.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.0.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.0.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.0.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.0.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.0.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.1.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.10.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.10.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.10.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.10.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.10.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.10.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.10.self_attn.norm_key.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.10.self_attn.norm_query.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.10.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.10.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.10.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.11.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.11.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.11.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.11.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.11.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.11.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.11.self_attn.norm_key.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.11.self_attn.norm_query.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.11.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.11.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.11.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.self_attn.norm_key.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.self_attn.norm_query.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.12.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.self_attn.norm_key.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.self_attn.norm_query.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.13.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.self_attn.norm_key.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.self_attn.norm_query.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.14.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.self_attn.norm_key.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.self_attn.norm_query.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.15.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.16.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.16.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.16.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.16.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.16.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.16.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.16.self_attn.norm_key.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.16.self_attn.norm_query.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.16.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.16.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.16.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "backbone.layers.17.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.17.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.17.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.17.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.17.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.17.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.17.self_attn.norm_key.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.17.self_attn.norm_query.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.17.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.17.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.17.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.self_attn.norm_key.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.self_attn.norm_query.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.18.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.self_attn.norm_key.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.self_attn.norm_query.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.19.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.2.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.2.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.2.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.2.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.2.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.2.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.2.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.2.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.2.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.2.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.2.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.20.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.20.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.20.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.20.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.20.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.20.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.20.self_attn.norm_key.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.20.self_attn.norm_query.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.20.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.20.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.20.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.self_attn.norm_key.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.self_attn.norm_query.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.21.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.22.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.22.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.22.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.22.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.22.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.22.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.22.self_attn.norm_key.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.22.self_attn.norm_query.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.22.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.22.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.22.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "backbone.layers.23.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.23.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.23.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.23.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.23.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.23.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.23.self_attn.norm_key.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.23.self_attn.norm_query.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.23.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.23.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.23.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.self_attn.norm_key.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.self_attn.norm_query.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.24.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.self_attn.norm_key.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.self_attn.norm_query.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.25.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.self_attn.norm_key.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.self_attn.norm_query.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.26.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.27.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.27.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.27.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.27.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.27.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.27.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.27.self_attn.norm_key.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.27.self_attn.norm_query.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.27.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.27.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.27.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "backbone.layers.28.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.28.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.28.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.28.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.28.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.28.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.28.self_attn.norm_key.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.28.self_attn.norm_query.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.28.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.28.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.28.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.self_attn.norm_key.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.self_attn.norm_query.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.29.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.3.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.3.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.3.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.3.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.3.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.3.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.3.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.3.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.3.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.3.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.3.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.30.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.30.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.30.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.30.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.30.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.30.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.30.self_attn.norm_key.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.30.self_attn.norm_query.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.30.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.30.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.30.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.self_attn.norm_key.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.self_attn.norm_query.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.31.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "backbone.layers.4.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.4.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.4.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.4.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.4.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.4.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.4.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.4.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.4.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.4.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.4.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.5.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.5.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.5.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.5.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.5.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.5.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.5.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.5.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.5.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.5.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.5.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "backbone.layers.6.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.6.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.6.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.6.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.6.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.6.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.6.self_attn.norm_key.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.6.self_attn.norm_query.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.6.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.6.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.6.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.self_attn.norm_key.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.self_attn.norm_query.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.7.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.self_attn.norm_key.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.self_attn.norm_query.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.8.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.self_attn.norm_key.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.self_attn.norm_query.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "backbone.layers.9.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "decoder.decoder_layers.0.cross_attention.k_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.cross_attention.norm_key.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.cross_attention.norm_query.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.cross_attention.o_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.cross_attention.q_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.cross_attention.v_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.kv_norm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.self_attn.norm_key.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.self_attn.norm_query.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.llama_layer.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.0.query_norm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.cross_attention.k_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.cross_attention.norm_key.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.cross_attention.norm_query.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.cross_attention.o_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.cross_attention.q_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.cross_attention.v_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.kv_norm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.self_attn.norm_key.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.self_attn.norm_query.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.llama_layer.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.1.query_norm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.cross_attention.k_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.cross_attention.norm_key.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.cross_attention.norm_query.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.cross_attention.o_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.cross_attention.q_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.cross_attention.v_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.kv_norm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.self_attn.norm_key.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.self_attn.norm_query.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.llama_layer.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.2.query_norm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.cross_attention.k_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.cross_attention.norm_key.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.cross_attention.norm_query.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.cross_attention.o_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.cross_attention.q_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.cross_attention.v_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.kv_norm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.self_attn.norm_key.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.self_attn.norm_query.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.llama_layer.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "decoder.decoder_layers.3.query_norm.weight": "model-00006-of-00006.safetensors",
+    "decoder_connector.first_word_embedding": "model-00006-of-00006.safetensors",
+    "encoder.embedding_layer.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.0.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.1.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.2.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.3.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.4.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.self_attn.norm_key.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.self_attn.norm_query.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder.layers.5.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder_connector.cross_attention_encoder_connector.k_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder_connector.cross_attention_encoder_connector.norm_key.weight": "model-00001-of-00006.safetensors",
+    "encoder_connector.cross_attention_encoder_connector.norm_query.weight": "model-00001-of-00006.safetensors",
+    "encoder_connector.cross_attention_encoder_connector.o_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder_connector.cross_attention_encoder_connector.q_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder_connector.cross_attention_encoder_connector.v_proj.weight": "model-00001-of-00006.safetensors",
+    "encoder_connector.latent_query": "model-00001-of-00006.safetensors",
+    "layer_norm.weight": "model-00006-of-00006.safetensors",
+    "lm_head.weight": "model-00006-of-00006.safetensors"
+  }
+}

norm.py

@@ -0,0 +1,21 @@
+import torch
+import torch.nn as nn
+
+class RMSNorm(nn.Module):
+    def __init__(self, dimensions: int, eps: float, device: torch.device, dtype: torch.dtype = torch.bfloat16, norm_in_fp32: bool = False):
+        super().__init__()
+        self.eps = eps
+        self.weight = torch.nn.Parameter(torch.ones(dimensions, dtype=dtype).to(device))
+        self.norm_in_fp32 = norm_in_fp32
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        original_dtype = x.dtype
+        if self.norm_in_fp32:
+            x = x.float()
+
+        out = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
+
+        if out.dtype != original_dtype:
+            out = out.to(original_dtype)
+
+        return out * self.weight

splitter.py

@@ -0,0 +1,39 @@
+import re
+
+from hat_splitter import HATSplitter as RustHATSplitter
+
+
+class HATSplitter:
+    def __init__(self, special_token_dict: dict | None = None, max_word_size: int = 128):
+        self.hat_splitter = RustHATSplitter()
+        self.max_word_size = max_word_size
+        self.special_token_dict = special_token_dict
+        self.special_token_replace: dict[int, list[int]] = {token: list(text.encode("utf-8")) for text, token in self.special_token_dict.items()}
+        self.special_token_pattern = re.compile(rf"({'|'.join(map(re.escape, special_token_dict.keys()))})") if special_token_dict else re.compile(r"(?!)")
+
+    def encode(self, text: str) -> list[list[int]]:
+        chunks = []
+        for str_chunk in self.special_token_pattern.split(text):
+            if str_chunk:
+                if str_chunk in self.special_token_dict:
+                    chunks.append([self.special_token_dict[str_chunk]])
+                else:
+                    chunks.extend(list(chunk) for chunk in self.hat_splitter.split_with_limit(str_chunk, self.max_word_size))
+        return chunks
+
+    def decode(self, token_ids: list[int], errors: str = "replace", skip_special_tokens: bool = False) -> str:
+        assert isinstance(token_ids, list), "token_ids must be a list"
+        assert all(isinstance(token_id, int) for token_id in token_ids), "token_ids must be a list of integers"
+
+        new_token_ids: list[int]
+        if skip_special_tokens:
+            new_token_ids = [token_id for token_id in token_ids if token_id not in self.special_token_replace]
+        else:
+            new_token_ids = []
+            for token in token_ids:
+                if token in self.special_token_replace:
+                    new_token_ids.extend(self.special_token_replace[token])
+                else:
+                    new_token_ids.append(token)
+
+        return bytes(new_token_ids).decode("utf-8", errors=errors)

transformer_backbone.py

@@ -0,0 +1,1553 @@
+# Hacked in QK norm in LlamaDecoderLayer in transformers; keeping the version from 
+# transformers==4.46.3 since this keeps rotary within the decoder layer
+# Source: https://github.com/huggingface/transformers/blob/v4.46.3/src/transformers/models/llama/modeling_llama.py#L400
+
+# coding=utf-8
+# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
+#
+# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
+# and OPT implementations in this library. It has been modified from its
+# original forms to accommodate minor architectural differences compared
+# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import math
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torch.nn.functional as F
+import torch.utils.checkpoint
+from torch import nn
+
+from transformers.activations import ACT2FN
+from transformers.cache_utils import Cache, DynamicCache, StaticCache
+from transformers.generation import GenerationMixin
+from transformers.modeling_attn_mask_utils import AttentionMaskConverter
+from transformers.modeling_flash_attention_utils import _flash_attention_forward
+from transformers.modeling_outputs import (
+    BaseModelOutputWithPast,
+    CausalLMOutputWithPast,
+    QuestionAnsweringModelOutput,
+    SequenceClassifierOutputWithPast,
+    TokenClassifierOutput,
+)
+from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS
+from transformers.modeling_utils import PreTrainedModel
+from transformers.pytorch_utils import ALL_LAYERNORM_LAYERS
+from transformers.utils import (
+    add_code_sample_docstrings,
+    add_start_docstrings,
+    add_start_docstrings_to_model_forward,
+    is_flash_attn_greater_or_equal_2_10,
+    logging,
+    replace_return_docstrings,
+)
+from transformers.models.llama.configuration_llama import LlamaConfig
+
+from .norm import RMSNorm
+
+logger = logging.get_logger(__name__)
+
+_CHECKPOINT_FOR_DOC = "meta-llama/Llama-2-7b-hf"
+_CONFIG_FOR_DOC = "LlamaConfig"
+
+
+class LlamaRMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        """
+        LlamaRMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+    def extra_repr(self):
+        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
+
+
+ALL_LAYERNORM_LAYERS.append(LlamaRMSNorm)
+
+
+class LlamaRotaryEmbedding(nn.Module):
+    def __init__(
+        self,
+        dim=None,
+        max_position_embeddings=2048,
+        base=10000,
+        device=None,
+        scaling_factor=1.0,
+        rope_type="default",
+        config: Optional[LlamaConfig] = None,
+    ):
+        super().__init__()
+        # TODO (joao): remove the `if` below, only used for BC
+        self.rope_kwargs = {}
+        if config is None:
+            logger.warning_once(
+                "`LlamaRotaryEmbedding` can now be fully parameterized by passing the model config through the "
+                "`config` argument. All other arguments will be removed in v4.46"
+            )
+            self.rope_kwargs = {
+                "rope_type": rope_type,
+                "factor": scaling_factor,
+                "dim": dim,
+                "base": base,
+                "max_position_embeddings": max_position_embeddings,
+            }
+            self.rope_type = rope_type
+            self.max_seq_len_cached = max_position_embeddings
+            self.original_max_seq_len = max_position_embeddings
+        else:
+            # BC: "rope_type" was originally "type"
+            if config.rope_scaling is not None:
+                self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
+            else:
+                self.rope_type = "default"
+            self.max_seq_len_cached = config.max_position_embeddings
+            self.original_max_seq_len = config.max_position_embeddings
+
+        self.config = config
+        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
+
+        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device, **self.rope_kwargs)
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self.original_inv_freq = self.inv_freq
+
+    def _dynamic_frequency_update(self, position_ids, device):
+        """
+        dynamic RoPE layers should recompute `inv_freq` in the following situations:
+        1 - growing beyond the cached sequence length (allow scaling)
+        2 - the current sequence length is in the original scale (avoid losing precision with small sequences)
+        """
+        seq_len = torch.max(position_ids) + 1
+        if seq_len > self.max_seq_len_cached:  # growth
+            inv_freq, self.attention_scaling = self.rope_init_fn(
+                self.config, device, seq_len=seq_len, **self.rope_kwargs
+            )
+            self.register_buffer("inv_freq", inv_freq, persistent=False)  # TODO joao: may break with compilation
+            self.max_seq_len_cached = seq_len
+
+        if seq_len < self.original_max_seq_len and self.max_seq_len_cached > self.original_max_seq_len:  # reset
+            self.register_buffer("inv_freq", self.original_inv_freq, persistent=False)
+            self.max_seq_len_cached = self.original_max_seq_len
+
+    @torch.no_grad()
+    def forward(self, x, position_ids):
+        if "dynamic" in self.rope_type:
+            self._dynamic_frequency_update(position_ids, device=x.device)
+
+        # Core RoPE block
+        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)
+        position_ids_expanded = position_ids[:, None, :].float()
+        # Force float32 (see https://github.com/huggingface/transformers/pull/29285)
+        device_type = x.device.type
+        device_type = device_type if isinstance(device_type, str) and device_type != "mps" else "cpu"
+        with torch.autocast(device_type=device_type, enabled=False):
+            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
+            emb = torch.cat((freqs, freqs), dim=-1)
+            cos = emb.cos()
+            sin = emb.sin()
+
+        # Advanced RoPE types (e.g. yarn) apply a post-processing scaling factor, equivalent to scaling attention
+        cos = cos * self.attention_scaling
+        sin = sin * self.attention_scaling
+
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+
+class LlamaLinearScalingRotaryEmbedding(LlamaRotaryEmbedding):
+    """LlamaRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev"""
+
+    def __init__(self, *args, **kwargs):
+        logger.warning_once(
+            "`LlamaLinearScalingRotaryEmbedding` is deprecated an will be removed in v4.46. Please use "
+            "`LlamaRotaryEmbedding`, which now also does linear scaling (simply pass the model config to __init__)."
+        )
+        kwargs["rope_type"] = "linear"
+        super().__init__(*args, **kwargs)
+
+
+class LlamaDynamicNTKScalingRotaryEmbedding(LlamaRotaryEmbedding):
+    """LlamaRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla"""
+
+    def __init__(self, *args, **kwargs):
+        logger.warning_once(
+            "`LlamaDynamicNTKScalingRotaryEmbedding` is deprecated an will be removed in v4.46. Please use "
+            "`LlamaRotaryEmbedding`, which now also does dynamic ntk scaling (simply pass the model config to "
+            "__init__)."
+        )
+        kwargs["rope_type"] = "dynamic"
+        super().__init__(*args, **kwargs)
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
+    """Applies Rotary Position Embedding to the query and key tensors.
+
+    Args:
+        q (`torch.Tensor`): The query tensor.
+        k (`torch.Tensor`): The key tensor.
+        cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        sin (`torch.Tensor`): The sine part of the rotary embedding.
+        position_ids (`torch.Tensor`, *optional*):
+            Deprecated and unused.
+        unsqueeze_dim (`int`, *optional*, defaults to 1):
+            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
+            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
+            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
+            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
+            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
+            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
+    Returns:
+        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
+    """
+    cos = cos.unsqueeze(unsqueeze_dim)
+    sin = sin.unsqueeze(unsqueeze_dim)
+    q_embed = (q * cos) + (rotate_half(q) * sin)
+    k_embed = (k * cos) + (rotate_half(k) * sin)
+    return q_embed, k_embed
+
+
+class LlamaMLP(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size
+        self.intermediate_size = config.intermediate_size
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)
+        self.act_fn = ACT2FN[config.hidden_act]
+
+    def forward(self, x):
+        if self.config.pretraining_tp > 1:
+            slice = self.intermediate_size // self.config.pretraining_tp
+            gate_proj_slices = self.gate_proj.weight.split(slice, dim=0)
+            up_proj_slices = self.up_proj.weight.split(slice, dim=0)
+            down_proj_slices = self.down_proj.weight.split(slice, dim=1)
+
+            gate_proj = torch.cat(
+                [F.linear(x, gate_proj_slices[i]) for i in range(self.config.pretraining_tp)], dim=-1
+            )
+            up_proj = torch.cat([F.linear(x, up_proj_slices[i]) for i in range(self.config.pretraining_tp)], dim=-1)
+
+            intermediate_states = (self.act_fn(gate_proj) * up_proj).split(slice, dim=2)
+            down_proj = [
+                F.linear(intermediate_states[i], down_proj_slices[i]) for i in range(self.config.pretraining_tp)
+            ]
+            down_proj = sum(down_proj)
+        else:
+            down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+
+        return down_proj
+
+
+def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
+    """
+    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
+    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
+    """
+    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
+    if n_rep == 1:
+        return hidden_states
+    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
+    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
+
+
+class LlamaAttention(nn.Module):
+    """Multi-headed attention from 'Attention Is All You Need' paper"""
+
+    def __init__(self, config: LlamaConfig, layer_idx: Optional[int] = None):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        if layer_idx is None:
+            logger.warning_once(
+                f"Instantiating {self.__class__.__name__} without passing a `layer_idx` is not recommended and will "
+                "lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` "
+                "when creating this class."
+            )
+
+        self.attention_dropout = config.attention_dropout
+        self.hidden_size = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = getattr(config, "head_dim", self.hidden_size // self.num_heads)
+        self.num_key_value_heads = config.num_key_value_heads
+        self.num_key_value_groups = self.num_heads // self.num_key_value_heads
+        self.max_position_embeddings = config.max_position_embeddings
+        self.rope_theta = config.rope_theta
+        self.is_causal = True
+        self.key_query_norm = config.key_query_norm
+        self.key_query_norm_per_head = config.key_query_norm_per_head
+
+        self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=config.attention_bias)
+        self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=config.attention_bias)
+        self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=config.attention_bias)
+        self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=config.attention_bias)
+
+        if self.key_query_norm:
+            if self.key_query_norm_per_head:
+                # Both query and key have head dim equal to self.hidden_size_per_attention_head
+                query_norm_dimensions = self.head_dim
+                key_norm_dimensions = self.head_dim
+            else:
+                # Query dimensions across head is equal to hidden_size but key dimensions are divided
+                # by self.num_repeat_kv
+                query_norm_dimensions = self.hidden_size
+                key_norm_dimensions = self.hidden_size // self.num_repeat_kv
+
+            # For numerical compatibility we use RMSNorm as it was used during training
+            self.norm_query = RMSNorm(
+                dimensions=query_norm_dimensions,
+                eps=config.rms_norm_eps,
+                device=self.q_proj.weight.device,
+                dtype=self.q_proj.weight.dtype,
+            )
+            self.norm_key = RMSNorm(
+                dimensions=key_norm_dimensions,
+                eps=config.rms_norm_eps,
+                device=self.q_proj.weight.device,
+                dtype=self.q_proj.weight.dtype,
+            )
+
+        # TODO (joao): remove in v4.46 (RoPE is computed in the model, not in the decoder layers)
+        self.rotary_emb = LlamaRotaryEmbedding(config=self.config)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Cache] = None,
+        output_attentions: bool = False,
+        use_cache: bool = False,
+        cache_position: Optional[torch.LongTensor] = None,
+        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,  # will become mandatory in v4.46
+        **kwargs,
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
+        bsz, q_len, _ = hidden_states.size()
+
+        if self.key_query_norm:
+            raise ValueError("QK norm not supported for eager attention, use flash_attention_2!")
+
+        if self.config.pretraining_tp > 1:
+            key_value_slicing = (self.num_key_value_heads * self.head_dim) // self.config.pretraining_tp
+            query_slices = self.q_proj.weight.split(
+                (self.num_heads * self.head_dim) // self.config.pretraining_tp, dim=0
+            )
+            key_slices = self.k_proj.weight.split(key_value_slicing, dim=0)
+            value_slices = self.v_proj.weight.split(key_value_slicing, dim=0)
+
+            query_states = [F.linear(hidden_states, query_slices[i]) for i in range(self.config.pretraining_tp)]
+            query_states = torch.cat(query_states, dim=-1)
+
+            key_states = [F.linear(hidden_states, key_slices[i]) for i in range(self.config.pretraining_tp)]
+            key_states = torch.cat(key_states, dim=-1)
+
+            value_states = [F.linear(hidden_states, value_slices[i]) for i in range(self.config.pretraining_tp)]
+            value_states = torch.cat(value_states, dim=-1)
+
+        else:
+            query_states = self.q_proj(hidden_states)
+            key_states = self.k_proj(hidden_states)
+            value_states = self.v_proj(hidden_states)
+
+        query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
+        key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+        value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+
+        if position_embeddings is None:
+            logger.warning_once(
+                "The attention layers in this model are transitioning from computing the RoPE embeddings internally "
+                "through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed "
+                "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be "
+                "removed and `position_embeddings` will be mandatory."
+            )
+            cos, sin = self.rotary_emb(value_states, position_ids)
+        else:
+            cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_value is not None:
+            # sin and cos are specific to RoPE models; cache_position needed for the static cache
+            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
+            key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
+
+        key_states = repeat_kv(key_states, self.num_key_value_groups)
+        value_states = repeat_kv(value_states, self.num_key_value_groups)
+        attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
+
+        if attention_mask is not None:  # no matter the length, we just slice it
+            causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
+            attn_weights = attn_weights + causal_mask
+
+        # upcast attention to fp32
+        attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
+        attn_weights = nn.functional.dropout(attn_weights, p=self.attention_dropout, training=self.training)
+        attn_output = torch.matmul(attn_weights, value_states)
+
+        if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):
+            raise ValueError(
+                f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is"
+                f" {attn_output.size()}"
+            )
+
+        attn_output = attn_output.transpose(1, 2).contiguous()
+
+        attn_output = attn_output.reshape(bsz, q_len, -1)
+
+        if self.config.pretraining_tp > 1:
+            attn_output = attn_output.split(self.hidden_size // self.config.pretraining_tp, dim=2)
+            o_proj_slices = self.o_proj.weight.split(self.hidden_size // self.config.pretraining_tp, dim=1)
+            attn_output = sum([F.linear(attn_output[i], o_proj_slices[i]) for i in range(self.config.pretraining_tp)])
+        else:
+            attn_output = self.o_proj(attn_output)
+
+        if not output_attentions:
+            attn_weights = None
+
+        return attn_output, attn_weights, past_key_value
+
+
+class LlamaFlashAttention2(LlamaAttention):
+    """
+    Llama flash attention module. This module inherits from `LlamaAttention` as the weights of the module stays
+    untouched. The only required change would be on the forward pass where it needs to correctly call the public API of
+    flash attention and deal with padding tokens in case the input contains any of them.
+    """
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+        # TODO: Should be removed once Flash Attention for RoCm is bumped to 2.1.
+        # flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. Reference: https://github.com/Dao-AILab/flash-attention/releases/tag/v2.1.0.
+        # Beware that with flash_attn<2.1, using q_seqlen != k_seqlen (except for the case q_seqlen == 1) produces a wrong mask (top-left).
+        self._flash_attn_uses_top_left_mask = not is_flash_attn_greater_or_equal_2_10()
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.LongTensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Cache] = None,
+        output_attentions: bool = False,
+        use_cache: bool = False,
+        cache_position: Optional[torch.LongTensor] = None,
+        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,  # will become mandatory in v4.46
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
+        if isinstance(past_key_value, StaticCache):
+            raise ValueError(
+                "`static` cache implementation is not compatible with `attn_implementation==flash_attention_2` "
+                "make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformers"
+            )
+
+        output_attentions = False
+
+        bsz, q_len, _ = hidden_states.size()
+
+        query_states = self.q_proj(hidden_states)
+        key_states = self.k_proj(hidden_states)
+        value_states = self.v_proj(hidden_states)
+
+        if self.key_query_norm:
+            if not self.key_query_norm_per_head:
+                # norm the full hidden fdim
+                query_states = self.norm_query(query_states)
+                key_states = self.norm_key(key_states)
+                
+
+        # Flash attention requires the input to have the shape
+        # batch_size x seq_length x head_dim x hidden_dim
+        # therefore we just need to keep the original shape
+        query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
+        key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+        value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+
+        if self.key_query_norm:
+            if self.key_query_norm_per_head:
+                # norm each head (with shared weights)
+                query_states = self.norm_query(query_states)
+                key_states = self.norm_key(key_states)
+
+        if position_embeddings is None:
+            logger.warning_once(
+                "The attention layers in this model are transitioning from computing the RoPE embeddings internally "
+                "through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed "
+                "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be "
+                "removed and `position_embeddings` will be mandatory."
+            )
+            cos, sin = self.rotary_emb(value_states, position_ids)
+        else:
+            cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_value is not None:
+            # sin and cos are specific to RoPE models; cache_position needed for the static cache
+            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
+            key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
+
+        # TODO: These transpose are quite inefficient but Flash Attention requires the layout [batch_size, sequence_length, num_heads, head_dim]. We would need to refactor the KV cache
+        # to be able to avoid many of these transpose/reshape/view.
+        query_states = query_states.transpose(1, 2)
+        key_states = key_states.transpose(1, 2)
+        value_states = value_states.transpose(1, 2)
+
+        dropout_rate = self.attention_dropout if self.training else 0.0
+
+        # In PEFT, usually we cast the layer norms in float32 for training stability reasons
+        # therefore the input hidden states gets silently casted in float32. Hence, we need
+        # cast them back in the correct dtype just to be sure everything works as expected.
+        # This might slowdown training & inference so it is recommended to not cast the LayerNorms
+        # in fp32. (LlamaRMSNorm handles it correctly)
+
+        input_dtype = query_states.dtype
+        if input_dtype == torch.float32:
+            if torch.is_autocast_enabled():
+                target_dtype = torch.get_autocast_gpu_dtype()
+            # Handle the case where the model is quantized
+            elif hasattr(self.config, "_pre_quantization_dtype"):
+                target_dtype = self.config._pre_quantization_dtype
+            else:
+                target_dtype = self.q_proj.weight.dtype
+
+            logger.warning_once(
+                f"The input hidden states seems to be silently casted in float32, this might be related to"
+                f" the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in"
+                f" {target_dtype}."
+            )
+
+            query_states = query_states.to(target_dtype)
+            key_states = key_states.to(target_dtype)
+            value_states = value_states.to(target_dtype)
+
+        attn_output = _flash_attention_forward(
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            q_len,
+            position_ids=position_ids,
+            dropout=dropout_rate,
+            sliding_window=getattr(self, "sliding_window", None),
+            use_top_left_mask=self._flash_attn_uses_top_left_mask,
+            is_causal=self.is_causal,
+        )
+
+        attn_output = attn_output.reshape(bsz, q_len, -1).contiguous()
+        attn_output = self.o_proj(attn_output)
+
+        if not output_attentions:
+            attn_weights = None
+
+        return attn_output, attn_weights, past_key_value
+
+
+class LlamaSdpaAttention(LlamaAttention):
+    """
+    Llama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from
+    `LlamaAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to
+    SDPA API.
+    """
+
+    # Adapted from LlamaAttention.forward
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Cache] = None,
+        output_attentions: bool = False,
+        use_cache: bool = False,
+        cache_position: Optional[torch.LongTensor] = None,
+        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,  # will become mandatory in v4.46
+        **kwargs,
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
+        if output_attentions:
+            # TODO: Improve this warning with e.g. `model.config.attn_implementation = "manual"` once this is implemented.
+            logger.warning_once(
+                "LlamaModel is using LlamaSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, "
+                'but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.'
+            )
+            return super().forward(
+                hidden_states=hidden_states,
+                attention_mask=attention_mask,
+                position_ids=position_ids,
+                past_key_value=past_key_value,
+                output_attentions=output_attentions,
+                use_cache=use_cache,
+                cache_position=cache_position,
+                position_embeddings=position_embeddings,
+            )
+
+        bsz, q_len, _ = hidden_states.size()
+
+        query_states = self.q_proj(hidden_states)
+        key_states = self.k_proj(hidden_states)
+        value_states = self.v_proj(hidden_states)
+
+        query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
+        key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+        value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+
+        if position_embeddings is None:
+            logger.warning_once(
+                "The attention layers in this model are transitioning from computing the RoPE embeddings internally "
+                "through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed "
+                "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be "
+                "removed and `position_embeddings` will be mandatory."
+            )
+            cos, sin = self.rotary_emb(value_states, position_ids)
+        else:
+            cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_value is not None:
+            # sin and cos are specific to RoPE models; cache_position needed for the static cache
+            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
+            key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
+
+        key_states = repeat_kv(key_states, self.num_key_value_groups)
+        value_states = repeat_kv(value_states, self.num_key_value_groups)
+
+        causal_mask = attention_mask
+        if attention_mask is not None:
+            causal_mask = causal_mask[:, :, :, : key_states.shape[-2]]
+
+        # SDPA with memory-efficient backend is currently (torch==2.1.2) bugged with non-contiguous inputs with custom attn_mask,
+        # Reference: https://github.com/pytorch/pytorch/issues/112577.
+        if query_states.device.type == "cuda" and causal_mask is not None:
+            query_states = query_states.contiguous()
+            key_states = key_states.contiguous()
+            value_states = value_states.contiguous()
+
+        # We dispatch to SDPA's Flash Attention or Efficient kernels via this `is_causal` if statement instead of an inline conditional assignment
+        # in SDPA to support both torch.compile's dynamic shapes and full graph options. An inline conditional prevents dynamic shapes from compiling.
+        is_causal = True if causal_mask is None and q_len > 1 else False
+
+        attn_output = torch.nn.functional.scaled_dot_product_attention(
+            query_states,
+            key_states,
+            value_states,
+            attn_mask=causal_mask,
+            dropout_p=self.attention_dropout if self.training else 0.0,
+            is_causal=is_causal,
+        )
+
+        attn_output = attn_output.transpose(1, 2).contiguous()
+        attn_output = attn_output.view(bsz, q_len, -1)
+
+        attn_output = self.o_proj(attn_output)
+
+        return attn_output, None, past_key_value
+
+
+LLAMA_ATTENTION_CLASSES = {
+    "eager": LlamaAttention,
+    "flash_attention_2": LlamaFlashAttention2,
+    "sdpa": LlamaSdpaAttention,
+}
+
+
+class LlamaDecoderLayer(nn.Module):
+    def __init__(self, config: LlamaConfig, layer_idx: int):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+
+        self.self_attn = LLAMA_ATTENTION_CLASSES[config._attn_implementation](config=config, layer_idx=layer_idx)
+
+        self.mlp = LlamaMLP(config)
+        self.input_layernorm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.post_attention_layernorm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Cache] = None,
+        output_attentions: Optional[bool] = False,
+        use_cache: Optional[bool] = False,
+        cache_position: Optional[torch.LongTensor] = None,
+        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,  # will become mandatory in v4.46
+        **kwargs,
+    ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
+        """
+        Args:
+            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
+            attention_mask (`torch.FloatTensor`, *optional*):
+                attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1,
+                query_sequence_length, key_sequence_length)` if default attention is used.
+            output_attentions (`bool`, *optional*):
+                Whether or not to return the attentions tensors of all attention layers. See `attentions` under
+                returned tensors for more detail.
+            use_cache (`bool`, *optional*):
+                If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
+                (see `past_key_values`).
+            past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states
+            cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*):
+                Indices depicting the position of the input sequence tokens in the sequence
+            position_embeddings (`Tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*):
+                Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`,
+                with `head_dim` being the embedding dimension of each attention head.
+            kwargs (`dict`, *optional*):
+                Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code
+                into the model
+        """
+        residual = hidden_states
+
+        hidden_states = self.input_layernorm(hidden_states)
+
+        # Self Attention
+        hidden_states, self_attn_weights, present_key_value = self.self_attn(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_value=past_key_value,
+            output_attentions=output_attentions,
+            use_cache=use_cache,
+            cache_position=cache_position,
+            position_embeddings=position_embeddings,
+            **kwargs,
+        )
+        hidden_states = residual + hidden_states
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.post_attention_layernorm(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+
+        outputs = (hidden_states,)
+
+        if output_attentions:
+            outputs += (self_attn_weights,)
+
+        if use_cache:
+            outputs += (present_key_value,)
+
+        return outputs
+
+
+LLAMA_START_DOCSTRING = r"""
+    This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
+    library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
+    etc.)
+
+    This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
+    Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
+    and behavior.
+
+    Parameters:
+        config ([`LlamaConfig`]):
+            Model configuration class with all the parameters of the model. Initializing with a config file does not
+            load the weights associated with the model, only the configuration. Check out the
+            [`~PreTrainedModel.from_pretrained`] method to load the model weights.
+"""
+
+
+@add_start_docstrings(
+    "The bare LLaMA Model outputting raw hidden-states without any specific head on top.",
+    LLAMA_START_DOCSTRING,
+)
+class LlamaPreTrainedModel(PreTrainedModel):
+    config_class = LlamaConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["LlamaDecoderLayer"]
+    _skip_keys_device_placement = ["past_key_values"]
+    _supports_flash_attn_2 = True
+    _supports_sdpa = True
+    _supports_cache_class = True
+    _supports_quantized_cache = True
+    _supports_static_cache = True
+
+    def _init_weights(self, module):
+        std = self.config.initializer_range
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+
+
+LLAMA_INPUTS_DOCSTRING = r"""
+    Args:
+        input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
+            Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
+            it.
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            [What are input IDs?](../glossary#input-ids)
+        attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
+
+            - 1 for tokens that are **not masked**,
+            - 0 for tokens that are **masked**.
+
+            [What are attention masks?](../glossary#attention-mask)
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            If `past_key_values` is used, optionally only the last `input_ids` have to be input (see
+            `past_key_values`).
+
+            If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`]
+            and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more
+            information on the default strategy.
+
+            - 1 indicates the head is **not masked**,
+            - 0 indicates the head is **masked**.
+        position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
+            config.n_positions - 1]`.
+
+            [What are position IDs?](../glossary#position-ids)
+        past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*):
+            Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
+            blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values`
+            returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`.
+
+            Two formats are allowed:
+            - a [`~cache_utils.Cache`] instance, see our
+            [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache);
+            - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of
+            shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy
+            cache format.
+
+            The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the
+            legacy cache format will be returned.
+
+            If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't
+            have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids`
+            of shape `(batch_size, sequence_length)`.
+        inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
+            Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
+            is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
+            model's internal embedding lookup matrix.
+        use_cache (`bool`, *optional*):
+            If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
+            `past_key_values`).
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+        cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*):
+            Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`,
+            this tensor is not affected by padding. It is used to update the cache in the correct position and to infer
+            the complete sequence length.
+"""
+
+
+@add_start_docstrings(
+    "The bare LLaMA Model outputting raw hidden-states without any specific head on top.",
+    LLAMA_START_DOCSTRING,
+)
+class LlamaModel(LlamaPreTrainedModel):
+    """
+    Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`]
+
+    Args:
+        config: LlamaConfig
+    """
+
+    def __init__(self, config: LlamaConfig):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        self.layers = nn.ModuleList(
+            [LlamaDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+        self.norm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.rotary_emb = LlamaRotaryEmbedding(config=config)
+        self.gradient_checkpointing = False
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.embed_tokens = value
+
+    @add_start_docstrings_to_model_forward(LLAMA_INPUTS_DOCSTRING)
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Union[Cache, List[torch.FloatTensor]]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+    ) -> Union[Tuple, BaseModelOutputWithPast]:
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if self.gradient_checkpointing and self.training and use_cache:
+            logger.warning_once(
+                "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`."
+            )
+            use_cache = False
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embed_tokens(input_ids)
+
+        # kept for BC (non `Cache` `past_key_values` inputs)
+        return_legacy_cache = False
+        if use_cache and not isinstance(past_key_values, Cache):
+            return_legacy_cache = True
+            if past_key_values is None:
+                past_key_values = DynamicCache()
+            else:
+                past_key_values = DynamicCache.from_legacy_cache(past_key_values)
+                logger.warning_once(
+                    "We detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and "
+                    "will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class "
+                    "(https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)"
+                )
+
+        if cache_position is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            cache_position = torch.arange(
+                past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
+            )
+        if position_ids is None:
+            position_ids = cache_position.unsqueeze(0)
+
+        causal_mask = self._update_causal_mask(
+            attention_mask, inputs_embeds, cache_position, past_key_values, output_attentions
+        )
+        hidden_states = inputs_embeds
+
+        # create position embeddings to be shared across the decoder layers
+        position_embeddings = self.rotary_emb(hidden_states, position_ids)
+
+        # decoder layers
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attns = () if output_attentions else None
+        next_decoder_cache = None
+
+        for decoder_layer in self.layers:
+            if output_hidden_states:
+                all_hidden_states += (hidden_states,)
+
+            if self.gradient_checkpointing and self.training:
+                layer_outputs = self._gradient_checkpointing_func(
+                    decoder_layer.__call__,
+                    hidden_states,
+                    causal_mask,
+                    position_ids,
+                    past_key_values,
+                    output_attentions,
+                    use_cache,
+                    cache_position,
+                    position_embeddings,
+                )
+            else:
+                layer_outputs = decoder_layer(
+                    hidden_states,
+                    attention_mask=causal_mask,
+                    position_ids=position_ids,
+                    past_key_value=past_key_values,
+                    output_attentions=output_attentions,
+                    use_cache=use_cache,
+                    cache_position=cache_position,
+                    position_embeddings=position_embeddings,
+                )
+
+            hidden_states = layer_outputs[0]
+
+            if use_cache:
+                next_decoder_cache = layer_outputs[2 if output_attentions else 1]
+
+            if output_attentions:
+                all_self_attns += (layer_outputs[1],)
+
+        hidden_states = self.norm(hidden_states)
+
+        # add hidden states from the last decoder layer
+        if output_hidden_states:
+            all_hidden_states += (hidden_states,)
+
+        next_cache = next_decoder_cache if use_cache else None
+        if return_legacy_cache:
+            next_cache = next_cache.to_legacy_cache()
+
+        if not return_dict:
+            return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=next_cache,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attns,
+        )
+
+    def _update_causal_mask(
+        self,
+        attention_mask: torch.Tensor,
+        input_tensor: torch.Tensor,
+        cache_position: torch.Tensor,
+        past_key_values: Cache,
+        output_attentions: bool,
+    ):
+        if self.config._attn_implementation == "flash_attention_2":
+            if attention_mask is not None and 0.0 in attention_mask:
+                return attention_mask
+            return None
+
+        # For SDPA, when possible, we will rely on its `is_causal` argument instead of its `attn_mask` argument, in
+        # order to dispatch on Flash Attention 2. This feature is not compatible with static cache, as SDPA will fail
+        # to infer the attention mask.
+        past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+        using_static_cache = isinstance(past_key_values, StaticCache)
+
+        # When output attentions is True, sdpa implementation's forward method calls the eager implementation's forward
+        if self.config._attn_implementation == "sdpa" and not using_static_cache and not output_attentions:
+            if AttentionMaskConverter._ignore_causal_mask_sdpa(
+                attention_mask,
+                inputs_embeds=input_tensor,
+                past_key_values_length=past_seen_tokens,
+                is_training=self.training,
+            ):
+                return None
+
+        dtype, device = input_tensor.dtype, input_tensor.device
+        sequence_length = input_tensor.shape[1]
+        if using_static_cache:
+            target_length = past_key_values.get_max_cache_shape()
+        else:
+            target_length = (
+                attention_mask.shape[-1]
+                if isinstance(attention_mask, torch.Tensor)
+                else past_seen_tokens + sequence_length + 1
+            )
+
+        # In case the provided `attention` mask is 2D, we generate a causal mask here (4D).
+        causal_mask = self._prepare_4d_causal_attention_mask_with_cache_position(
+            attention_mask,
+            sequence_length=sequence_length,
+            target_length=target_length,
+            dtype=dtype,
+            device=device,
+            cache_position=cache_position,
+            batch_size=input_tensor.shape[0],
+        )
+
+        if (
+            self.config._attn_implementation == "sdpa"
+            and attention_mask is not None
+            and attention_mask.device.type == "cuda"
+            and not output_attentions
+        ):
+            # Attend to all tokens in fully masked rows in the causal_mask, for example the relevant first rows when
+            # using left padding. This is required by F.scaled_dot_product_attention memory-efficient attention path.
+            # Details: https://github.com/pytorch/pytorch/issues/110213
+            min_dtype = torch.finfo(dtype).min
+            causal_mask = AttentionMaskConverter._unmask_unattended(causal_mask, min_dtype)
+
+        return causal_mask
+
+    @staticmethod
+    def _prepare_4d_causal_attention_mask_with_cache_position(
+        attention_mask: torch.Tensor,
+        sequence_length: int,
+        target_length: int,
+        dtype: torch.dtype,
+        device: torch.device,
+        cache_position: torch.Tensor,
+        batch_size: int,
+        **kwargs,
+    ):
+        """
+        Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape
+        `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing.
+
+        Args:
+            attention_mask (`torch.Tensor`):
+                A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape
+                `(batch_size, 1, query_length, key_value_length)`.
+            sequence_length (`int`):
+                The sequence length being processed.
+            target_length (`int`):
+                The target length: when generating with static cache, the mask should be as long as the static cache,
+                to account for the 0 padding, the part of the cache that is not filled yet.
+            dtype (`torch.dtype`):
+                The dtype to use for the 4D attention mask.
+            device (`torch.device`):
+                The device to plcae the 4D attention mask on.
+            cache_position (`torch.Tensor`):
+                Indices depicting the position of the input sequence tokens in the sequence.
+            batch_size (`torch.Tensor`):
+                Batch size.
+        """
+        if attention_mask is not None and attention_mask.dim() == 4:
+            # In this case we assume that the mask comes already in inverted form and requires no inversion or slicing.
+            causal_mask = attention_mask
+        else:
+            min_dtype = torch.finfo(dtype).min
+            causal_mask = torch.full(
+                (sequence_length, target_length), fill_value=min_dtype, dtype=dtype, device=device
+            )
+            if sequence_length != 1:
+                causal_mask = torch.triu(causal_mask, diagonal=1)
+            causal_mask *= torch.arange(target_length, device=device) > cache_position.reshape(-1, 1)
+            causal_mask = causal_mask[None, None, :, :].expand(batch_size, 1, -1, -1)
+            if attention_mask is not None:
+                causal_mask = causal_mask.clone()  # copy to contiguous memory for in-place edit
+                mask_length = attention_mask.shape[-1]
+                padding_mask = causal_mask[:, :, :, :mask_length] + attention_mask[:, None, None, :]
+                padding_mask = padding_mask == 0
+                causal_mask[:, :, :, :mask_length] = causal_mask[:, :, :, :mask_length].masked_fill(
+                    padding_mask, min_dtype
+                )
+
+        return causal_mask
+
+
+class LlamaForCausalLM(LlamaPreTrainedModel, GenerationMixin):
+    _tied_weights_keys = ["lm_head.weight"]
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = LlamaModel(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.model.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.model.embed_tokens = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def set_decoder(self, decoder):
+        self.model = decoder
+
+    def get_decoder(self):
+        return self.model
+
+    @add_start_docstrings_to_model_forward(LLAMA_INPUTS_DOCSTRING)
+    @replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Union[Cache, List[torch.FloatTensor]]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        num_logits_to_keep: int = 0,
+        **loss_kwargs,
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+        r"""
+        Args:
+            labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+                Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
+                config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
+                (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
+
+            num_logits_to_keep (`int`, *optional*):
+                Calculate logits for the last `num_logits_to_keep` tokens. If `0`, calculate logits for all
+                `input_ids` (special case). Only last token logits are needed for generation, and calculating them only for that
+                token can save memory, which becomes pretty significant for long sequences or large vocabulary size.
+
+        Returns:
+
+        Example:
+
+        ```python
+        >>> from transformers import AutoTokenizer, LlamaForCausalLM
+
+        >>> model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
+        >>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
+
+        >>> prompt = "Hey, are you conscious? Can you talk to me?"
+        >>> inputs = tokenizer(prompt, return_tensors="pt")
+
+        >>> # Generate
+        >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
+        >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
+        "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
+        ```"""
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            cache_position=cache_position,
+        )
+
+        hidden_states = outputs[0]
+        if self.config.pretraining_tp > 1:
+            lm_head_slices = self.lm_head.weight.split(self.vocab_size // self.config.pretraining_tp, dim=0)
+            logits = [F.linear(hidden_states, lm_head_slices[i]) for i in range(self.config.pretraining_tp)]
+            logits = torch.cat(logits, dim=-1)
+        else:
+            # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
+            logits = self.lm_head(hidden_states[:, -num_logits_to_keep:, :])
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **loss_kwargs)
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+@add_start_docstrings(
+    """
+    The LLaMa Model transformer with a sequence classification head on top (linear layer).
+
+    [`LlamaForSequenceClassification`] uses the last token in order to do the classification, as other causal models
+    (e.g. GPT-2) do.
+
+    Since it does classification on the last token, it requires to know the position of the last token. If a
+    `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If
+    no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the
+    padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in
+    each row of the batch).
+    """,
+    LLAMA_START_DOCSTRING,
+)
+class LlamaForSequenceClassification(LlamaPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+        self.model = LlamaModel(config)
+        self.score = nn.Linear(config.hidden_size, self.num_labels, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.model.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.model.embed_tokens = value
+
+    @add_start_docstrings_to_model_forward(LLAMA_INPUTS_DOCSTRING)
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Union[Cache, List[torch.FloatTensor]]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, SequenceClassifierOutputWithPast]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
+            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
+            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        transformer_outputs = self.model(
+            input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        hidden_states = transformer_outputs[0]
+        logits = self.score(hidden_states)
+
+        if input_ids is not None:
+            batch_size = input_ids.shape[0]
+        else:
+            batch_size = inputs_embeds.shape[0]
+
+        if self.config.pad_token_id is None and batch_size != 1:
+            raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.")
+        if self.config.pad_token_id is None:
+            sequence_lengths = -1
+        else:
+            if input_ids is not None:
+                # if no pad token found, use modulo instead of reverse indexing for ONNX compatibility
+                sequence_lengths = torch.eq(input_ids, self.config.pad_token_id).int().argmax(-1) - 1
+                sequence_lengths = sequence_lengths % input_ids.shape[-1]
+                sequence_lengths = sequence_lengths.to(logits.device)
+            else:
+                sequence_lengths = -1
+
+        pooled_logits = logits[torch.arange(batch_size, device=logits.device), sequence_lengths]
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits=logits, labels=labels, pooled_logits=pooled_logits, config=self.config)
+
+        if not return_dict:
+            output = (pooled_logits,) + transformer_outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return SequenceClassifierOutputWithPast(
+            loss=loss,
+            logits=pooled_logits,
+            past_key_values=transformer_outputs.past_key_values,
+            hidden_states=transformer_outputs.hidden_states,
+            attentions=transformer_outputs.attentions,
+        )
+
+
+@add_start_docstrings(
+    """
+The Llama Model transformer with a span classification head on top for extractive question-answering tasks like
+SQuAD (a linear layer on top of the hidden-states output to compute `span start logits` and `span end logits`).
+    """,
+    LLAMA_START_DOCSTRING,
+)
+class LlamaForQuestionAnswering(LlamaPreTrainedModel):
+    base_model_prefix = "transformer"
+
+    # Copied from transformers.models.bloom.modeling_bloom.BloomForQuestionAnswering.__init__ with Bloom->Llama
+    def __init__(self, config):
+        super().__init__(config)
+        self.transformer = LlamaModel(config)
+        self.qa_outputs = nn.Linear(config.hidden_size, 2)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.transformer.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.transformer.embed_tokens = value
+
+    @add_start_docstrings_to_model_forward(LLAMA_INPUTS_DOCSTRING)
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Union[Cache, List[torch.FloatTensor]]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        start_positions: Optional[torch.LongTensor] = None,
+        end_positions: Optional[torch.LongTensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        **kwargs,
+    ) -> Union[Tuple, QuestionAnsweringModelOutput]:
+        r"""
+        start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for position (index) of the start of the labelled span for computing the token classification loss.
+            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
+            are not taken into account for computing the loss.
+        end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for position (index) of the end of the labelled span for computing the token classification loss.
+            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
+            are not taken into account for computing the loss.
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.transformer(
+            input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        sequence_output = outputs[0]
+
+        logits = self.qa_outputs(sequence_output)
+        start_logits, end_logits = logits.split(1, dim=-1)
+        start_logits = start_logits.squeeze(-1).contiguous()
+        end_logits = end_logits.squeeze(-1).contiguous()
+
+        loss = None
+        if start_positions is not None and end_positions is not None:
+            loss = self.loss_function(start_logits, end_logits, start_positions, end_positions, **kwargs)
+
+        if not return_dict:
+            output = (start_logits, end_logits) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return QuestionAnsweringModelOutput(
+            loss=loss,
+            start_logits=start_logits,
+            end_logits=end_logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+@add_start_docstrings(
+    """
+    The Llama Model transformer with a token classification head on top (a linear layer on top of the hidden-states
+    output) e.g. for Named-Entity-Recognition (NER) tasks.
+    """,
+    LLAMA_START_DOCSTRING,
+)
+class LlamaForTokenClassification(LlamaPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+        self.model = LlamaModel(config)
+        if getattr(config, "classifier_dropout", None) is not None:
+            classifier_dropout = config.classifier_dropout
+        elif getattr(config, "hidden_dropout", None) is not None:
+            classifier_dropout = config.hidden_dropout
+        else:
+            classifier_dropout = 0.1
+        self.dropout = nn.Dropout(classifier_dropout)
+        self.score = nn.Linear(config.hidden_size, config.num_labels)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.model.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.model.embed_tokens = value
+
+    @add_start_docstrings_to_model_forward(LLAMA_INPUTS_DOCSTRING)
+    @add_code_sample_docstrings(
+        checkpoint=_CHECKPOINT_FOR_DOC,
+        output_type=TokenClassifierOutput,
+        config_class=_CONFIG_FOR_DOC,
+    )
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, TokenClassifierOutput]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
+            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
+            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.model(
+            input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        sequence_output = outputs[0]
+        sequence_output = self.dropout(sequence_output)
+        logits = self.score(sequence_output)
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits, labels, self.config)
+
+        if not return_dict:
+            output = (logits,) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return TokenClassifierOutput(
+            loss=loss,
+            logits=logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )