  Symbolic equations are at the core of scientific discovery. The task of
discovering the underlying equation from a set of input-output pairs is called
symbolic regression. Traditionally, symbolic regression methods use
hand-designed strategies that do not improve with experience. In this paper, we
introduce the first symbolic regression method that leverages large scale
pre-training. We procedurally generate an unbounded set of equations, and
simultaneously pre-train a Transformer to predict the symbolic equation from a
corresponding set of input-output-pairs. At test time, we query the model on a
new set of points and use its output to guide the search for the equation. We
show empirically that this approach can re-discover a set of well-known
physical equations, and that it improves over time with more data and compute.
