  While polyhedral compilers have shown success in implementing advanced code
transformations, they still have challenges in selecting the most profitable
transformations that lead to the best speedups. This has motivated the use of
machine learning to build cost models to guide the search for polyhedral
optimizations. State-of-the-art polyhedral compilers have demonstrated a viable
proof-of-concept of this approach. While such a proof-of-concept has shown
promise, it still has significant limitations. State-of-the-art polyhedral
compilers that use a deep-learning cost model only support a small subset of
affine transformations, limiting their ability to apply complex code
transformations. They also only support simple programs that have a single loop
nest and a rectangular iteration domain, limiting their applicability to many
programs. These limitations significantly impact the generality of such
compilers and autoschedulers and put into question the whole approach. In this
paper, we introduce LOOPer, the first polyhedral autoscheduler that uses a
deep-learning based cost model and covers a large set of affine transformations
and programs. It supports the exploration of a large set of affine
transformations, allowing the application of complex sequences of polyhedral
transformations. It also supports the optimization of programs with multiple
loop nests and with rectangular and non-rectangular iteration domains, allowing
the optimization of an extensive set of programs. We implement and evaluate
LOOPer and show that it achieves speedups over the state-of-the-art. On the
Polybench benchmark, LOOPer achieves a geometric mean speedup of 1.59x over
Tiramisu. LOOPer also achieves competitive speedups with a geometric mean
speedup of 1.34x over Pluto, a state-of-the-art polyhedral compiler that does
not use a machine-learning based cost model.
