  Accurate photometric redshifts are a lynchpin for many future experiments to
pin down the cosmological model and for studies of galaxy evolution. In this
study, a novel sparse regression framework for photometric redshift estimation
is presented. Simulated and real data from SDSS DR12 were used to train and
test the proposed models. We show that approaches which include careful data
preparation and model design offer a significant improvement in comparison with
several competing machine learning algorithms. Standard implementations of most
regression algorithms have as the objective the minimization of the sum of
squared errors. For redshift inference, however, this induces a bias in the
posterior mean of the output distribution, which can be problematic. In this
paper we directly target minimizing $\Delta z = (z_\textrm{s} -
z_\textrm{p})/(1+z_\textrm{s})$ and address the bias problem via a
distribution-based weighting scheme, incorporated as part of the optimization
objective. The results are compared with other machine learning algorithms in
the field such as Artificial Neural Networks (ANN), Gaussian Processes (GPs)
and sparse GPs. The proposed framework reaches a mean absolute $\Delta z =
0.0026(1+z_\textrm{s})$, over the redshift range of $0 \le z_\textrm{s} \le 2$
on the simulated data, and $\Delta z = 0.0178(1+z_\textrm{s})$ over the entire
redshift range on the SDSS DR12 survey, outperforming the standard ANNz used in
the literature. We also investigate how the relative size of the training set
affects the photometric redshift accuracy. We find that a training set of
\textgreater 30 per cent of total sample size, provides little additional
constraint on the photometric redshifts, and note that our GP formalism
strongly outperforms ANNz in the sparse data regime for the simulated data set.
