---
library_name: RAT
language:
- en
license: mit
datasets:
- HuggingFaceFW/fineweb-edu
tags:
- efficient architecture
- recurrence
- attention
- pretraining
metrics:
- perplexity
- accuracy
---

## Description
Models trained from [RAT Paper](https://arxiv.org/abs/2507.04416).

## Citation
If you find it useful, please consider citing the paper:
```
@article{wei2025rat,
  title={RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling},
  author={Wei, Xiuying and Yadav, Anunay and Pascanu, Razvan and Gulcehre, Caglar},
  journal={arXiv preprint arXiv:2507.04416},
  year={2025}
}
```