|
|
--- |
|
|
library_name: RAT |
|
|
language: |
|
|
- en |
|
|
license: mit |
|
|
datasets: |
|
|
- HuggingFaceFW/fineweb-edu |
|
|
tags: |
|
|
- efficient architecture |
|
|
- recurrence |
|
|
- attention |
|
|
- pretraining |
|
|
metrics: |
|
|
- perplexity |
|
|
- accuracy |
|
|
--- |
|
|
|
|
|
## Description |
|
|
Models trained from [RAT Paper](https://arxiv.org/abs/2507.04416). |
|
|
|
|
|
## Citation |
|
|
If you find it useful, please consider citing the paper: |
|
|
``` |
|
|
@article{wei2025rat, |
|
|
title={RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling}, |
|
|
author={Wei, Xiuying and Yadav, Anunay and Pascanu, Razvan and Gulcehre, Caglar}, |
|
|
journal={arXiv preprint arXiv:2507.04416}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
