  Analyzing time series of fluxes from stars, known as stellar light curves,
can reveal valuable information about stellar properties. However, most current
methods rely on extracting summary statistics, and studies using deep learning
have been limited to supervised approaches. In this research, we investigate
the scaling law properties that emerge when learning from astronomical time
series data using self-supervised techniques. By employing the GPT-2
architecture, we show the learned representation improves as the number of
parameters increases from $10^4$ to $10^9$, with no signs of performance
plateauing. We demonstrate that a self-supervised Transformer model achieves
3-10 times the sample efficiency compared to the state-of-the-art supervised
learning model when inferring the surface gravity of stars as a downstream
task. Our research lays the groundwork for analyzing stellar light curves by
examining them through large-scale auto-regressive generative models.
