  Symbolic Regression (SR) searches for mathematical expressions which best
describe numerical datasets. This allows to circumvent interpretation issues
inherent to artificial neural networks, but SR algorithms are often
computationally expensive. This work proposes a new Transformer model aiming at
Symbolic Regression particularly focused on its application for Scientific
Discovery. We propose three encoder architectures with increasing flexibility
but at the cost of column-permutation equivariance violation. Training results
indicate that the most flexible architecture is required to prevent from
overfitting. Once trained, we apply our best model to the SRSD datasets
(Symbolic Regression for Scientific Discovery datasets) which yields
state-of-the-art results using the normalized tree-based edit distance, at no
extra computational cost.
