  Reconstructing the intrinsic Ly$\alpha$ line flux from high-$z$ QSOs can
place constraints on the neutral hydrogen content of the intergalactic medium
during reionisation. There are now $\gtrsim10$ different Ly$\alpha$
reconstruction pipelines using different methodologies to predict the
Ly$\alpha$ line flux from correlations with the spectral information redward of
Ly$\alpha$. However, there have been few attempts to directly compare the
performance of these pipelines. Therefore, we devised a blind QSO challenge to
compare these reconstruction pipelines on a uniform set of objects. Each author
was provided de-identified, observed rest-frame QSO spectra with spectral
information only redward of 1260\AA\ rest-frame to ensure unbiased
reconstruction. We constructed two samples of 30 QSOs, from X-Shooter and SDSS
both spanning $3.5<z<4.5$. Importantly, the purpose of this comparison study
was not to champion a single, best performing reconstruction pipeline but
rather to explore the relative performance of these pipelines over a range of
QSOs with broad observational characteristics to infer general trends. In
summary, we find machine learning approaches in general provide the strongest
``best guesses" but underestimate the accompanying statistical uncertainty,
although these can be recalibrated, whilst pipelines that decompose the
spectral information, for example principal component or factor analysis
generally perform better at predicting the Ly$\alpha$ profile. Further, we
found that reconstruction pipelines trained on SDSS QSOs performed similarly on
average for both the X-Shooter and SDSS samples indicating no discernible
biases owing to differences in the observational characteristics of the
training set or QSO being reconstructed, although the recovered distributions
of reconstructions for X-Shooter were broader likely due to an increased
fraction of outliers.
