  In February this year Google proposed a new Transformer variant called FLASH,
which has a faster speed, lower VRAM footprint and better performance. This is
achieved by designing a performant layer named GAU (Gated Attention Unit),
which combines the Attention layer and FFN. In this paper, some implementation
details are re-analyzed both theoretically and practically. We then propose a
novel GAU-based model and pre-train it on a Chinese corpus. Results of the CLUE
benchmark show that our model achieves a dev average score of 75.02, 1% higher
than RoFormerV1 and being 45% faster, which is also competitive with
RoFormerV2.
