  The ionic bonding across the lattice and ordered microscopic structures endow
crystals with unique symmetry and determine their macroscopic properties.
Unconventional crystals, in particular, exhibit non-traditional lattice
structures or possess exotic physical properties, making them intriguing
subjects for investigation. Therefore, to accurately predict the physical and
chemical properties of crystals, it is crucial to consider long-range orders.
While GNN excels at capturing the local environment of atoms in crystals, they
often face challenges in effectively capturing longer-ranged interactions due
to their limited depth. In this paper, we propose CrysToGraph
($\textbf{Crys}$tals with $\textbf{T}$ransformers $\textbf{o}$n
$\textbf{Graph}$s), a novel transformer-based geometric graph network designed
specifically for unconventional crystalline systems, and UnconvBench, a
comprehensive benchmark to evaluate models' predictive performance on
unconventional crystal materials such as defected crystals, low-dimension
crystals and MOF. CrysToGraph effectively captures short-range interactions
with transformer-based graph convolution blocks as well as long-range
interactions with graph-wise transformer blocks. CrysToGraph proofs its
effectiveness in modelling unconventional crystal materials in multiple tasks,
and moreover, it outperforms most existing methods, achieving new
state-of-the-art results on the benchmarks of both unconventional crystals and
traditional crystals.
