  Rapid strides are currently being made in the field of artificial
intelligence using Transformer-based models like Large Language Models (LLMs).
The potential of these methods for creating a single, large, versatile model in
astronomy has not yet been explored. In this work, we propose a framework for
data-driven astronomy that uses the same core techniques and architecture as
used by LLMs. Using a variety of observations and labels of stars as an
example, we build a Transformer-based model and train it in a self-supervised
manner with cross-survey data sets to perform a variety of inference tasks. In
particular, we demonstrate that a $\textit{single}$ model can perform both
discriminative and generative tasks even if the model was not trained or
fine-tuned to do any specific task. For example, on the discriminative task of
deriving stellar parameters from Gaia XP spectra, we achieve an accuracy of 47
K in $T_\mathrm{eff}$, 0.11 dex in $\log{g}$, and 0.07 dex in $[\mathrm{M/H}]$,
outperforming an expert $\texttt{XGBoost}$ model in the same setting. But the
same model can also generate XP spectra from stellar parameters, inpaint
unobserved spectral regions, extract empirical stellar loci, and even determine
the interstellar extinction curve. Our framework demonstrates that building and
training a $\textit{single}$ foundation model without fine-tuning using data
and parameters from multiple surveys to predict unmeasured observations and
parameters is well within reach. Such "Large Astronomy Models" trained on large
quantities of observational data will play a large role in the analysis of
current and future large surveys.
