  $^{171}$Yb$^{3+}$-doped Y$_2$SiO$_5$ crystals are a promising platform for
optical quantum memories in long-distance quantum communications. The relevance
of this material lies in $^{171}$Yb long optical and spin coherence times,
along with a large hyperfine splitting, enabling long quantum storage over
large bandwidths. Mechanisms affecting the optical decoherence are however not
precisely known, especially since low-temperature measurements have so far
focused on the 2 to 4 K range. In this work, we performed two- and three-pulse
photon echoes and spectral hole burning to determine optical homogeneous
linewidths in two 171 Yb:YSO crystals doped at 2 and 10 ppm. Experiments were
performed in the 40 mK to 18 K temperature range, leading to linewidths between
320 Hz, among the narrowest reported for rare-earth ions, and several MHz. Our
results show that above 6 K the homogeneous linewidth is mainly due to an
elastic two-phonon process which results in a slow broadening with temperature,
the homogeneous linewidth reaching only 25 kHz at 10 K. At lower temperatures,
interactions with $^{89}$Yb nuclear spin-flips, paramagnetic defects or
impurities, and also Yb-Yb interactions for the higher concentrated crystal,
are likely the main limiting factor to the homogeneous linewidth. In
particular, we conclude that the direct effect of spin and optical excited
state lifetime is a minor contribution to optical decoherence in the whole
temperature range studied. Our results indicate possible paths and regimes for
further decreasing the homogeneous linewidths or maintaining narrow lines at
higher $^{171}$Yb concentration.
