  Transformers have demonstrated effectiveness in in-context solving
data-fitting problems from various (latent) models, as reported by Garg et al.
However, the absence of an inherent iterative structure in the transformer
architecture presents a challenge in emulating the iterative algorithms, which
are commonly employed in traditional machine learning methods. To address this,
we propose the utilization of looped transformer architecture and its
associated training methodology, with the aim of incorporating iterative
characteristics into the transformer architectures. Experimental results
suggest that the looped transformer achieves performance comparable to the
standard transformer in solving various data-fitting problems, while utilizing
less than 10% of the parameter count.
