The 2-degree Field Lensing Survey: photometric redshifts from a large new training sample to $r<19.5$
We present a new training set for estimating empirical photometric redshifts of galaxies, which was created as part of the 2dFLenS project. This training set is located in a $\sim $700 deg$^2$ area of the KiDS South field and is randomly selected and nearly complete at $r<19.5$. We investigate the photometric redshift performance obtained with $ugriz$ photometry from VST-ATLAS and W1/W2 from WISE, based on several empirical and template methods. The best redshift errors are obtained with kernel-density estimation, as are the lowest biases, which are consistent with zero within statistical noise. The 68th percentiles of the redshift scatter for magnitude-limited samples at $r<(15.5, 17.5, 19.5)$ are (0.014, 0.017, 0.028). In this magnitude range, there are no known ambiguities in the colour-redshift map, consistent with a small rate of redshift outliers. In the fainter regime, the KDE method produces $p(z)$ estimates per galaxy that represent unbiased and accurate redshift frequency expectations. The $p(z)$ sum over any subsample is consistent with the true redshift frequency plus Poisson noise. Further improvements in redshift precision at $r<20$ would mostly be expected from filter sets with narrower passbands to increase the sensitivity of colours to small changes in redshift.
phz_2dFLenS.tex
\documentclass[useAMS,usenatbib,usegraphicx]{mn2e}
\usepackage{times}
\usepackage{url}

\newcommand{\myarcsec}{\hbox{$.\!\!^{\prime\prime}$}}

% this allows A4 format in astro-ph
\topmargin -0.5in

\title[Photo-z's from 2dFLenS ]{The 2-degree Field Lensing Survey: photometric redshifts from a large new training sample to $r<19.5$  }

\author[C. Wolf et al.]{Christian Wolf$^1$, Andrew S. Johnson$^{2,3}$, Maciej Bilicki$^{4,5}$, Chris Blake$^2$, A.~Amon$^6$, \newauthor
T.~Erben$^7$, K.~Glazebrook$^2$, C.~Heymans$^6$, H.~Hildebrandt$^7$, S.~Joudaki$^{2,3}$, D.~Klaes$^7$, \newauthor
K.~Kuijken$^4$, C.~Lidman$^8$, F.~Marin$^{2,3}$, D.~Parkinson$^9$, G.~Poole$^{10}$
\smallskip \smallskip \\
$^1$Research School of Astronomy and Astrophysics, Australian National University, Canberra, ACT 2611, Australia, email: christian.wolf@anu.edu.au \\
$^2$Centre for Astrophysics and Supercomputing, Swinburne University of Technology, PO Box 218, Hawthorn, VIC 3122, Australia \\
$^3$ARC Centre of Excellence for All-sky Astrophysics (CAASTRO) \\
$^4$Leiden Observatory, Leiden University, Niels Bohrweg 2, 2333 CA Leiden, The Netherlands \\
$^5$Janusz Gil Institute of Astronomy, University of Zielona Gora, ul. Szafrana 2, 65-516, Zielona Gora, Poland \\
$^6$Scottish Universities Physics Alliance, Institute for Astronomy, University of Edinburgh, Blackford Hill, Edinburgh EH9 3HJ, UK \\
$^7$Argelander-Institut f\"ur Astronomie, Auf dem H\"ugel 71, 53121 Bonn, Germany\\
$^8$Australian Astronomical Observatory, PO Box 915, North Ryde NSW 1670, Australia\\
$^9$School of Mathematics and Physics, University of Queensland, Brisbane, QLD 4072, Australia\\
$^{10}$School of Physics, University of Melbourne, Parkville, VIC 3010, Australia \\
%email: asjohnson@swin.edu.au, cblake@swin.edu.au \\
}

\begin{document}
\date{draft \today}
\maketitle

\begin{abstract}
We present a new training set for estimating empirical photometric redshifts of galaxies, which was created as part of the 2dFLenS project. This training set is located in a $\sim $700 deg$^2$ area of the KiDS South field and is randomly selected and nearly complete at $r<19.5$. We investigate the photometric redshift performance obtained with $ugriz$ photometry from VST-ATLAS and W1/W2 from WISE, based on several empirical and template methods. The best redshift errors are obtained with kernel-density estimation, as are the lowest biases, which are consistent with zero within statistical noise. The 68th percentiles of the redshift scatter for magnitude-limited samples at $r<(15.5, 17.5, 19.5)$ are (0.014, 0.017, 0.028). In this magnitude range, there are no known ambiguities in the colour-redshift map, consistent with a small rate of redshift outliers. In the fainter regime, the KDE method produces $p(z)$ estimates per galaxy that represent unbiased and accurate redshift frequency expectations. The $p(z)$ sum over any subsample is consistent with the true redshift frequency plus Poisson noise. Further improvements in redshift precision at $r<20$ would mostly be expected from filter sets with narrower passbands to increase the sensitivity of colours to small changes in redshift.
\end{abstract}

\begin{keywords}
surveys -- galaxies: distances and redshifts -- methods: statistical %-- techniques: photometric 
\end{keywords}


\section{Introduction}

Redshift estimates for galaxies can be derived from imaging photometry and are known as photometric redshifts a.k.a. {\it photo-z's or phot-z's}. They were conceived over half a century ago to extend the reach of the largest telescopes in their attempt to constrain world models \citep{SW48,Baum57}. Today, they are particularly attractive for large-area surveys, where relatively modest observing time can deliver many more redshifts than spectroscopic campaigns. The motivation for photo-z's is still largely driven by cosmological tests \citep[e.g.][]{BB05,Mas15}, but extends beyond these to studies of galaxy evolution \citep[e.g.][]{Ben01,Wolf03a,Sco07,Spitler14} and the identification of rare objects.

Two domains of photo-z application can be differentiated: (1) Deep pencil-beam surveys, such as the original Hubble Deep Field \citep{HDF}, push the frontier of exploration into the unknown, and redshifts for distant faint objects are constrained by Bayesian exploration of the data using spectral energy distribution templates and galaxy evolution models \citep[e.g.][]{Lan96}. 

(2) At the other end of the scale, wide-area surveys grow to cover most of the sky and register huge numbers of galaxies despite relatively shallow flux limits, simply because area is easier to extend than depth; the analysis of their data is usually limited by systematics rather than number statistics. In this domain, photo-z's are ideally based on accurate empirical frequency maps of redshift occurrence, where such maps are usually derived from spectroscopic training samples such as that of the Main Galaxy sample of the Sloan Digital Sky Survey \citep[SDSS,][]{SDSS,FLS03,Csa03,Oya08a}. Photo-z catalogues of vast areas of sky have not only been constructed for the area covered by SDSS, but also for all-sky footprints, such as the 2MPZ and WISE x SuperCOSMOS catalogues by \citet{Bil14,Bil16}.

Given that photometric catalogues can easily achieve greater depth than complete spectroscopic catalogues, it is tempting to derive photo-z's as deep as $r>22$, although the available large and complete training sets such as the SDSS Main Spectroscopic Sample reach only $r\sim 17.5$. Since it is evident that fainter galaxies may reach higher redshifts, and empirical non-parametric maps cannot be meaningfully extrapolated, a deeper photo-z catalogue will only be useful when similarly deep training sets are added. This is now commonly done \citep[e.g.][]{Beck16}, although we note that most of these deep spectroscopic catalogues are highly incomplete \citep{New13}, and the objects with missing redshifts have been found to reside at very different redshifts when deeper spectra became available \citep{DeepPC}. Since the incompleteness propagates equally into the training sample as into the validation sample, it is not revealed by the purely internal performance measures of photo-z precision. When large parts of the true redshift distribution are missing from a training sample, they will be missing from the empirically trained photo-z catalogue as well as from the performance statistics. Pushing empirical photo-z's deeper in a reliable fashion requires not fancier statistical methods, but simply deeper complete random spectroscopic training samples.

In this paper, we explore photo-z's derived from a new training sample, which satisfies three important criteria for the first time: (1) going deeper than the SDSS Main Galaxy sample by two magnitudes, thus pushing to higher redshifts as well as fainter galaxies, (2) going wide enough to overcome cosmic variance by drawing the training sample from $\sim $700 deg$^2$ of sky, and (3) being very complete in representing the random galaxy population. Training sets can of course be derived from other surveys as well, however, e.g. the samples from the SDSS Stripe 82 \citep[e.g.][]{Nie09,Bundy15} and the WiggleZ survey \citep{Blake10,Dr10} use particular target selections and ignore certain types of galaxies. In contrast, redshifts from the Galaxy and Mass Assembly \citep[GAMA,][]{GAMA} survey are extremely complete and reliable to a similar depth as our new sample, and indeed manifest a complete census of the galaxy population within their target fields. The effective area of the GAMA sample used by \citet{Bil16} is, however, smaller than that of the sample presented here and may thus be more affected by cosmic variance. This is because GAMA fully samples its chosen sky area, while our training set subsamples a larger area. Thus, our new training sample should provide a useful resource for deriving galaxy photo-z's to nearly magnitude $r\approx 20$, and will be publicly available (see website at \url{http://2dflens.swin.edu.au}). It is an unbiased spectroscopic sample of general value, which can be used for photo-z training by any survey that covers the region of our sample.

The purpose of this paper is to explore how well broadband photo-z's perform at $r\la 20$ using complete random validation samples. The new training set was created with the SkyMapper Southern Survey \citep{K07} in mind, which will reach a depth similar to the SDSS imaging on 20,000 deg$^2$ area by 2019,
and release its first deep data soon \citep{W17}. The SkyMapper Southern Survey addresses a broad range of science goals: stellar science and Galactic archaeology studies benefit from the SkyMapper filter set ($uvgriz$), which allows estimating the stellar parameters $(T_{\rm eff}, \log g, M/H)$ straight from photometry. SkyMapper will be the main optical counterpart to the Evolutionary Map of the Universe \citep[EMU,][]{Norris11}, a large continuum survey of the Australian SKA Pathfinder \citep[ASKAP,][]{Jo07,Jo08} planned to run from 2017 to 2019. EMU will locate 70 million radio sources and will rely on photometric redshifts from SkyMapper, combined with VHS and WISE, for much of its work. Based on an average PSF with $2\farcs 5$ FWHM, we expect a point-source completeness limit of $>21$ mag in $g$ and $r$-band, so that SkyMapper will see counterparts to over 20 million EMU sources. SkyMapper will also be the main imaging resource to underpin the massive new spectroscopic Taipan Galaxy Survey at $i\la 18$ (see \url{http://www.taipan-survey.org}). Finally, the repeat visits of SkyMapper allow addressing variability in both the stellar and extragalactic regime. Most of the galaxies in SkyMapper will be at redshifts of $z<0.5$, and with SkyMapper being a legacy survey the photometric redshifts will be used for an unpredictable range of science applications.

In the absence of complete SkyMapper data, we based the photo-z exploration work on images of the slightly deeper VST-ATLAS survey \citep{Sh15}, where the filters are $ugriz$. Once SkyMapper data is available, we expect to see a slight improvement of redshift accuracy at the low-redshift end due to the extra violet filter of SkyMapper. As usual, we explore here not only empirical methods based on our new training set, but compare with a template method as well.

Our new training set is a complete random sample of galaxies from across a wide area of $\sim $700 deg$^2$ and obtained via spare-fibre spectroscopy within the 2-degree Field Lensing Survey \citep[2dFLenS,][]{Blake16}. The 2dFLenS survey is a large-scale galaxy redshift survey that has recently collected $\sim $70,000 redshifts within the footprint of the VST-ATLAS survey, using the AAOmega spectrograph at the Anglo-Australian Telescope. The principal science goal of 2dFLenS is to test gravitational physics through the joint observation of galaxy velocities, traced by redshift-space distortions, and weak gravitational lensing measured on data of the Kilo-Degree-Survey \citep[KiDS; see][]{deJong13,Kui15}. Its secondary purpose is to test methods of photometric-redshift calibration using both direct techniques for bright objects (this paper) and cross-correlation for a fainter sample \citep{J16}.

Our 2dF spectroscopy complements and extends existing data to increase depth and reliability of the photo-z's: the footprint of VST-ATLAS already contains several ten thousand published redshifts from the 2dF Galaxy Redshift Survey \citep[2dFGRS,][]{GRS}. While this data is complete and reliable to $r\approx 17.7$ obviating the need for bright-object spectroscopy, we limit our targets at the faint end to $r=19.5$, given the observing constraints of the 2dFLenS project and the methodological requirement to obtain a complete and reliable redshift sample. For all-sky photo-z purposes we desire a random sample that will represent the whole sky as best as possible and thus demand that its cosmic variance is as low as as possible. Hence, we construct our sample not as a complete census from a compact area, but instead by heavily subsampling the galaxy population across a wide area. The use of unallocated spare fibres in the wide-area spectroscopic survey 2dFLenS is thus a perfect solution for our needs. In the future, we will explore how to optimally include data from GAMA and other sources as well.

In the following, we present our data set in Sect.~2 and our methods in Sect.~3. In Sect.~4 we discuss the photo-z's we obtain by combining optical photometry in VST-ATLAS with mid-infrared photometry from WISE \citep{WISE} and by combining redshifts from 2dFGRS and 2dFLenS. 


\section{Data}

The main objective of the 2dFLenS project is spectroscopic follow-up of the 1500 deg$^2$ VST-KiDS Survey \citep{deJong13}, which is optimised for weak-gravitational lensing studies. The target selection in 2dFLenS comprised several components with different requirements detailed in \citet{Blake16}. When 2dFLenS first started, KiDS did not yet cover all of its area, and therefore another imaging survey, VST-ATLAS \citep{Sh15}, was used in its place for 2dFLenS target selection. VST-ATLAS is less deep, but it covered the 2dFLenS area and was completely sufficient for source selection in 2dFLenS in terms of its depth and multi-band nature. For all details of survey coverage and processing of imaging data, we refer to \citet{Blake16}.


\subsection{Object selection from VST-ATLAS imaging data}

For the purpose of this paper, we only use relatively bright objects with $r<19.5$. We note that at the depth of VST-ATLAS, objects with $r=19.5$ have very low photometric errors in all bands (see Table~\ref{tab:ATLAS_charac}); in the $r$-band the median formal flux error is less than 1\%, and true uncertainties in the photometry are related almost exclusively to the usual challenges in galaxy photometry stemming from insufficiently constrained light profiles. 

 \citet{Blake16} describe the creation of our photometric catalogue for VST-ATLAS. In brief, we determine galaxy colours from isophotal magnitudes and use identical apertures in all bands. We first apply a shapelet-based PSF gaussianization and homogenisation \citep{K08,H12} to all images, then extract total object magnitudes in the detection r-band using flexible elliptical apertures, and run Source Extractor \citep{BA96} in Dual Image mode to obtain magnitudes for the other bands in consistent apertures. As a result, our galaxy photometry probes identical physical footprints on the galaxy image outside the atmosphere, despite bandpass variations of seeing. Finally, we apply illumination and standard dust corrections.  All magnitudes are calibrated to the AB system. We have also cross-matched the resulting catalogue to WISE (see Sect.~3.1.4 in \citet{Blake16}); there we find that most objects are detected in the W1 and W2 bands, while W3 and W4 have lower signal, and are thus ignored for this purpose. 
 
However, homogeneous photometric calibration turned out to be challenging in VST-ATLAS, because of the very low overlapping area between pointings (see Sect.~3.6 in \citet{Blake16}), and we noticed after creating our sample that the resulting mean photo-z biases were a function of VST field. We also found the mean object colours to vary in line with that, and hence need to apply additional zero-point offsets to remove the field-to-field variation in calibration. We thus modified the default photometry by adjusting the magnitude zero-points per VST field, using WISE as reference and adopting a very simple approach: (1) for every VST field we selected point sources with $r=[14,18]$, which is a nearly uncontaminated star sample with well-measured photometry; (2) we determined the median colours of this sample per field and compared with the overall median of the data set; finally, we (3) adjusted all bands so that the median colours per VST field are the same. In the latter step, we kept the WISE magnitudes as unchanged reference points, and only adjusted the VST-ATLAS photometry to match.


\begin{table}
  \caption{Average properties of VST-ATLAS imaging data in 2dFLenS area. $\sigma_{\rm mag}$ is the median magnitude error at the spectroscopic limit of $r=19.5$.}
\label{tab:ATLAS_charac}      
\centering          
\begin{tabular}{ccccc}
\hline\hline       
\multicolumn{1}{c}{Filter} & \multicolumn{1}{c}{exposure time [s]} &
\multicolumn{1}{c}{$\sigma_{\rm mag}$} & \multicolumn{1}{c}{seeing [$\arcsec$]} & $m_{\rm lim, 5\sigma}$ \\ 
\hline
$u$ & $2-4\times 60$ & $0.06$ & $1.11 \pm 0.20$ & 22.0\\ 
$g$ & $2\times 50$ & $0.02$ & $1.00 \pm 0.25$ & 23.0\\ 
$r$ & $2\times 45$  & $0.01$ & $0.89 \pm 0.19$ & 22.5\\ 
$i$ & $2\times 45$  & $0.01$ & $0.86 \pm 0.23$ & 21.8\\ 
$z$ & $2\times 45$ & $0.02$ & $0.87 \pm 0.22$ & 20.7\\
\hline                  
\end{tabular}
\end{table}


\subsection{Spectroscopy}

\subsubsection{The 2dFLenS direct photo-$z$ sample}

The spectroscopy for 2dFLenS was carried out using the AAOmega spectrograph at the Anglo-Australian Telescope between September 2014 and January 2016 (during the 14B, 15A and 15B semesters). In total, 2dFLenS contains about 70,000 high-quality redshifts at $z<0.9$, while this paper uses only the \textit{direct photo-$z$ sample}. This is a complete sub-sample of $\sim $30,000 galaxies in the magnitude range $17<r< 19.5$, that was added to the target pool of 2dFLenS at lower priority and has yielded spectra in the range $z\la 0.5$. 30,931 (31,864) targets were observed for the direct photo-z sample, for which 28,269 (29,123) good redshifts were obtained, where the figures in brackets include objects selected for the direct photo-z sample which were flagged for observation in other 2dFLenS target classes.

Starting from the photometric catalogues of VST-ATLAS, we selected all objects with $17<r<19.5$ as possible targets for this program. During the first two semesters (2014B and 2015A) of the 2dFLenS observations, we considered only clearly extended objects, requiring that the {\tt FLUX\_RADIUS} parameter in Source Extractor exceeded 0.9 multiplied by the seeing in all individual exposures and the co-added image. This selection produces a pure galaxy sample with extremely small stellar contamination, and ensures that little observing time is wasted on non-galaxies; however, it implies that the redshift sample does not represent the whole galaxy population randomly, but only the large portion identified as extended objects in our imaging data. Clearly extended galaxies are likely to have a different redshift distribution than compact galaxies, so this sample was intentionally incomplete.

During the last semester (2015B) of 2dFLenS observations, we included compact sources in the target list, in order to both, complement the galaxy sample and collect statistics on other types of unresolved sources with non-stellar colours. Using optical+WISE colours, it is possible to separate compact galaxies from stars, QSOs and host-dominated AGN \citep[see e.g.][]{J11,P16}, but a detailed investigation of this is beyond the scope of this paper. The number ratio between compact galaxies and extended galaxies with $r=[17,19.5]$ in our source catalogue is $1:12$.

Defining a redshift density distribution in any feature space works best when the redshift sample is both (1) representative, and (2) not sparse. While the representation criterion suggests a random sampling of galaxies, the sparsity criterion suggests to sample preferentially the low-density parts of the feature space. Their weight in the density estimation can be adjusted using appropriate selection weights, while their boosted number avoids the problem of discretisation noise. Given the steep number counts of galaxies, a pure random sample would be dominated by the faintest objects, while we would risk discretisation noise at the bright end. Hence, we decided to apply a magnitude-dependent weighting for the clearly extended galaxy sample to boost the brighter objects and moderately flatten the magnitude distribution in our observed sample: we used a factor $f(R)\propto 2^{(17-r)}$, which reduces the number count slope by 0.3. 
 

\subsubsection{Redshifting}

Redshifts in 2dFLenS are measured and assigned quality flags using a combination of automatic software fitting and visual inspection.  First, all observed spectra are passed through the {\tt AUTOZ} code \citep{Bal14} developed for the Galaxy And Mass Assembly (GAMA) survey, which uses a cross-correlation method of redshift determination including both absorption-line and emission-line template spectra.  Each observed spectrum and {\tt AUTOZ} solution is then inspected by 2dFLenS team members using either the code {\tt RUNZ}, originally developed by Will Sutherland for the 2dFGRS, or the web-based {\tt MARZ} code \citep{MARZ}. The reviewers can then manually improve the redshift determination in cases where {\tt AUTOZ} is not successful and assign final quality flags.

Quality flags range from values of 1 to 5. Q=1 means that no features are visible, the redshift is purely determined by cross-correlation of the spectrum with templates and the correlation coefficient is very low. Only for Q=2 and better quality levels, a human assessor has actually labelled our confidence in the spectroscopic redshift as "possible" (Q=2), "probable" (Q=3), and "practically certain" (Q=4/5). We further group the sample into good-quality redshifts (Q=3/4/5) and bad-quality ones (Q=1/2).

\begin{figure*}
\centering
\includegraphics[clip,angle=270,width=0.95\hsize]{merged_train.ps}
\caption{Spectroscopic sample of empirical high-quality redshifts. {\it Left:} we combine 2dFGRS at $r<17.7$ with 2dFLenS at $r>17.7$ and give LenS galaxies a weight factor taking into account the larger effective area of GRS and our magnitude selection factor. The weighted number counts of the LenS sample connect seamlessly to GRS, so their combination forms a redshift sample with a realistic magnitude distribution. The edges of the LenS distribution are softened by a photometric recalibration after the sample was observed. {\it Centre:} Up to $r<17.7$ the redshift distributions of the shallower GRS and the deeper LenS agree, but fainter than that GRS appears to lack higher-z galaxies ({\it right}). Thus $r=17.7$ is a reliable completeness limit for GRS.
\label{mergedtrain}}
\end{figure*}


\subsubsection{Catalogue cleaning and completeness}\label{cleaning}

We construct our redshift sample for this paper from the 2dFLenS fields in the Southern and Northern area of the KiDS Survey. The Southern area spans the sky from $22^h$ to $3\fh5$ in R.A. and $-36\degr$ to $-26\degr$ in declination, and the Northern area extends from $10\fh4$ to $15\fh5$ in R.A. and $-5\degr$ to $-2\degr$ in declination. Thus, the combined sample covers a total effective area of $\sim $700 deg$^2$. We then remove (i) objects for which the photometry appears incomplete, flagged as unreliable or affected by artefacts, (ii) objects that are flagged by our image masks or have Source Extractor flags $>3$, which mostly eliminates objects with corrupted aperture data that are too close to the edges of images, and (iii) objects that are identified as Galactic stars from their spectra. We further eliminate objects for which the magnitude error reported by Source Extractor is 99~mag in any band; this indicates a faulty measurement as this value may appear for objects of any flux in the sample and does \textit{not} indicate a dropout. This cleaning process reduces our effective area without introducing a selection effect. 

Finally, the current version of the 2dFLenS redshift catalogue provides no reliable flagging of QSOs yet. For the purpose of this paper, we want to eliminate them from the sample, as they are rare and clear outliers from the main galaxy distribution in a magnitude-redshift diagram and cover their locus only sparsely. They are largely removed by applying a magnitude-dependent redshift cut to the 2dFLenS targets of $z<0.3+0.12(r-17)$. At the faint end of 2dFLenS this cut is at $z=0.6$. We take advantage of WISE photometry where available, but we do not require a WISE counterpart to use the object. In our model sample the fraction of galaxies without a WISE counterpart increases from 3\% at $r\sim 17$ to 13\% at $r>19$.


\subsubsection{Merging with 2dFGRS}\label{sample}

We combine redshifts from different samples to cover a broader range in magnitude, taking advantage of the fact that the 2dF Galaxy Redshift Survey (2dFGRS) and also the 6dF Galaxy Survey \citep[6dFGS,][]{6dF} already covered brighter magnitudes of $r\la17$ very well. Comparing the 6dFGS and 2dFGRS samples in the larger Southern field of the 2dFLenS area, we found 4,311 redshifts measured by both 2dFGRS and 6dF. Among these we find 29 disagreements ($\sim 0.7$\%), which appear to be mostly better measured by the deeper 2dFGRS. At very bright magnitudes of $14<r<15$ their completeness and quality is similar, while 6dFGS gets incomplete at $r>15.5$ and 2dFGRS extends well beyond $r=17$. We thus choose to build our master redshift sample simply from 2dFGRS and 2dFLenS, and select from the 2dFGRS just the $\sim $700 deg$^2$ region that extends from $22^h$ to $3\fh5$ in R.A. and $-36\degr$ to $-26\degr$ in declination and thus fully overlaps with the larger Southern area in 2dFLenS.

When merging two redshift samples of different quality, depth and effective area, we want to select and weight the objects such that they will appear to form a single, complete and unbiased sample taken from a consistent effective area. This is because the object mix in any training sample acts as a prior in empirical redshift estimation, and for optimum performance and lowest systematics we would like the prior to be unbiased. 

In Fig.~\ref{mergedtrain} we plot number counts of the two cleaned samples individually, which demonstrate that 2dFGRS is incomplete at $r\ga 18$, whereas 2dFLenS is by design incomplete at $r\la 17$ and $\ga 19.5$. The magnitude edges of the 2dFLenS distributions are soft, because we recalibrated the photometry again after the spectroscopic sample was selected. We also plot the redshift distribution $\log n(z)$ for both samples in narrow magnitude bins and find that 2dFGRS selectively lacks higher-redshift galaxies at $r\approx 18$. Hence, we cut the 2dFGRS sample conservatively to $r\le17.7$, where both number counts and redshift distributions compared to the deeper 2dFLenS suggest that it is entirely complete.

Since the sample definition of 2dFLenS was done prior to the final photometric calibration, we get just half a magnitude overlap where 2dFGRS and 2dFLenS are both complete. In this range, $\sim 80$\% of 2dFLenS galaxies were also observed by 2dFGRS, while 2dFLenS has observed $\sim 14$\% of the 2dFGRS targets owing to its much lower fill factor. From data of the first two 2dFLenS semesters, we found 1,274 high-quality redshifts measured in both surveys, with 14 disagreements (1.1\%) of $\Delta cz > 1000$~km/s. The vast remainder has a $\Delta cz$ RMS of 120~km/sec. Most of the 14 disagreements appear to be more reliable in 2dFGRS, and thus we decided to simply use 2dFGRS to $r\le 17.7$ and 2dFLenS at $r>17.7$.

Since the 2dFLenS sample is a spare-fibre sample with sparse coverage, its effective area is 17\% of that of 2dFGRS at $r=17.7$ (14\% over the Southern field alone, but 17\% overall once we add in the 2dFLenS targets from the Northern field). Furthermore, our selection boost for brighter, clearly extended, objects has flattened the number count slope by $-0.3$. Thus, we compensate the effective area of the 2dFLenS sample using magnitude-dependent weights of $\log w = -\log 0.17 + 0.3(r-17.7)$, where $w = 1$ for each 2dFGRS galaxy. The compact galaxies observed only in the third semester need $w = 20.41$ to represent them in the training sample with a weight corresponding to their abundance in the parent sample, from which all sample selection was done. 

The effect of these weights on the effective number counts of the 2dFLenS sample is shown in Fig.~\ref{mergedtrain} and appears to be a fair extrapolation of the 2dFGRS behaviour. The sample thus combined is a random sample of galaxies with $14\la r\la 19.5$, apart from incompleteness at the faint end due to edge-softening from recalibration and a mild decrease in the fraction of good-quality spectra that will be discussed later. It contains 50,919 good quality redshifts, of which 32,765 are from 2dFGRS and 18,154 are from 2dFLenS. We also build a sample of bad-quality redshifts (Q=1 or 2) with the same weight formula, which informs our magnitude-dependent redshift completeness: this includes a total of 2,191 bad redshifts, with 316 from 2dFGRS and 1,875 from 2dFLenS.


\section{Methods for photo-z determination}

From the earliest times of photo-z history, two different kinds of models have been used for photo-z's, those based on parametric templates and those based on empirical redshift data. Parametric models have redshift as one of their axes, while SED type, dust or star-formation histories are others. Empirical models are also known as training samples, even though not all empirical methods involve a true \textit{training} step. The empirical models simply sub-sample a discrete realisation of the true galaxy population within the survey with a size-dependent Poisson sampling noise and feature noise associated with the measurement process. Then they can be used as a model directly or feed a training step that creates an abstract model from the training sample. 

Either model defines a mapping $z(\vec{c})$ from the feature space, here photometry, to the label space, here redshift. For the empirical models, we can choose the feature space freely, while for template models they are restricted by the existing code package and its template information. Relevant criteria for choosing the feature axes are (i) maximising the redshift discrimination of features in the play-off between redshift dependence of the feature and typical noise for the feature values, as well as sometimes (ii) choosing features where the model is not too sparse and (iii) independence of features to minimise covariance. 

Features can be any observable not restricted to SEDs and can include e.g. size and shape parameters, in which case the estimates might not be called purely `photometric' redshifts. For the empirical models we chose to use a feature space spanned by the $r$-band magnitude, linking the optical and WISE using $r-W1$, and forming colour indices from neighbouring passbands otherwise; thus the full feature set is $\{r,r-W1,W1-W2,u-g,g-r,r-i,i-z\}$. The template method used here employs fluxes directly, and only the optical bands $ugriz$, because reliable templates that cover the mid-infrared wavelengths are not yet available.

All methods used in this paper are probabilistic, as opposed to function-fitting methods, so they provide redshift distributions $p(z|c)$ given a colour measurement. An important point worth clarifying in the context of photometric redshifts is the usage of the term probability. In his textbook about statistical inference \citet{MacKay} reminds us that \textit{probability} can describe two different meanings: it can describe ``frequencies of outcomes in random experiments'', and it can describe ``degrees of belief in propositions that do not involve random variables'', and further notes that ``a likelihood function is not a probability distribution''. This is especially relevant when photo-z catalogues are used in a cosmological analysis, where probability distributions are often taken to be frequency distributions. E.g., weak-lensing and clustering studies have long benefitted from considering the full probability redshift distributions of individual objects \citep[e.g][]{Edd06,Kit07,Man08,Kilb15,Asorey16}. Hence, we will also investigate towards the end of the paper to what extent the $p(z)$ distributions we obtain represent actual frequency distributions $n(z)$.

We also note a fundamental floor to the precision with which the redshift of a galaxy can be estimated from its SED, irrespective of method. The intrinsic variety in galaxy colours at a fixed redshift implies that for an observed galaxy colour there is intrinsic variety or ambiguity in redshift. While this effect is obvious when the observable is a single colour index, it still dominates the redshift errors of bright well-measured galaxies in current multi-band data sets. A few broad passbands don't break all degeneracies in the space of SEDs spanned by the entire observable galaxy population, and even with a very rich training set these intrinsic limits will not be overcome, but are set by the observed features. Indeed, the primary aim of creating the \textit{direct photo-z sample} in the 2dFLenS Survey was to create a model sample at $r\la 20$, which would allow the derivation of empirical photo-z's with a precision that approaches the theoretical limit. In contrast, e.g. \citet{PHAT} compare a large number of methods on a deep photometric dataset from the Great Observatories Origins Deep Survey \citep[GOODS,][]{Giav04} with sparser and incomplete spectroscopy, which is the extreme opposite of the data domain investigated here.

At bright magnitudes calibration uncertainties may be large relative to photometric errors, and will thus drive the error budget. For template methods this concerns the calibration of the model templates relative to the observations. The issue is equally important for empirical methods, where the sky area containing the model sample needs to be on the same calibration as the query sample. Otherwise, colour offsets between query and training data will lead to systematic redshift biases.

Our photometric data is much deeper than the spectroscopic data: errors are typically $<2$\% at the faint end of the spectroscopic sample, with the exception of the u-band, where the median error at the faint limit is $\sim $6\%. This means that calibration errors dominate the results in our study, and we will find the floor of the possible photometric redshift errors. Even deeper u-band observations are expected to improve the results only marginally.


\subsection{Empirical method KDE}

Kernel-density estimation \citep[KDE, see e.g.][]{Wang07} was one particular method, where the Bayesian and empirical approach could be unified by using the empirical sample objects as discrete instances of a model: technically KDE with an empirical sample is identical to Bayesian model fitting, if the kernel function is chosen to be the error ellipsoid of the query object and the empirical features of the model object are free of random errors, or the kernel function is the square difference of query and model errors \citep[see][]{W09}.

In practice, the KDE method runs over one query object at a time and determines the redshift probability function $p(z)$ at the location of the query object from a representation of all other model objects obtained by convolving the discrete point cloud with a kernel. As a kernel function we use a Gaussian, whose width is the squared sum of the photometric error of the query object and a minimum kernel width. Mathematically, this is identical to a template-fitting code that determines a likelihood from a $\chi^2$-fit that square-adds a photometric error and a minimum error to take calibration uncertainties into account. For each query object with the features $c_{\rm query},j$ we find a probability $p_i$ that it resembles a model object $m_i$ with the features $c_{i,j}$ and located at redshift $z_i$, which, assuming Gaussian errors, is derived by the standard equation 

\begin{equation}
	\chi^2_i = \sum_j \frac{(c_{{\rm query},j} - c_{i,j})^2}{\sigma_{{\rm query},j}^2 + \sigma_{0,j}^2}   ~,\\
	p_i \propto e^{-\chi^2_i/2}  ~.
\end{equation}

We choose the minimum kernel width to be $\sigma_{0,j}=0\fm 05$ for colour indices, and $0\fm 2$ for the $r$-band magnitude: this kernel smoothing is not meant to signify a calibration uncertainty, but to cover the sparsity of the model; and in analogy to a template-fitting method, smoothing over the $r$-band magnitude limits the resolution of a magnitude prior, while smoothing over a colour index limits the impact of the SED itself. Template-fitting methods commonly assume $\sim0\fm 05$ uncertainties on zeropoints, but they require no smoothing and thus no error floor for a magnitude prior, as their nature is not discrete but continuous.

We exclude the query object itself from the model, because otherwise it would appear as an identical pair in the above equation with $\chi^2_i=0$ and mislead the results. This form of query-object exclusion allows us to use the full empirical sample both as a query sample and as a model sample. Neural network training  (discussed in Sec. 3.2), in contrast, requires splitting the sample into separate non-overlapping training and validation samples, perhaps using a 70:30 split, in order to prevent overfitting. In the KDE method each object available in the model sample gives an independent estimate of redshift precision and the whole sample can be used as a model. However, increasing the model size from 70\% of a training sample to 100\% in a KDE model sample improves the estimation performance only slightly, but the over $3\times$ bigger validation sample reduces the noise in the validation result. The latter aspect does not make the estimation better, but increases our confidence in measuring the performance.

Naturally, this approach produces probabilities for discrete redshift values. We then resample these into redshift bins to cover the continuous redshift range, by sorting all model objects and their associated probabilities $p_i$ into bins of width $\Delta z = 0.003 \times (1+z)$. We set an upper redshift limit at $z=0.5$, since in our magnitude range, objects at higher redshift are very rare and populate the feature space only sparsely. We eliminate higher-redshift objects only from the model and thus preclude an assignment of a higher redshift estimate. However, we keep the few higher-redshift objects in the query sample, effectively forcing them to appear as outliers, since they would typically form a small but real part in any blind photometric sample subjected to photo-z estimation. 

The redshift probability distribution $p(z)$ is then normalised, where we take two possible classes of objects into account: (1) The first class is the empirical sample of high-quality redshifts as described in Sect.~\ref{sample}, for which we get a meaningful $p(z)$ distribution. (2) The second class comprises the remainder of the complete target sample, where no reliable redshift was obtained from the spectra, and these are attributed an "unrecoverable $p(z)$". Each object in the query sample is compared against both model classes, and their two Bayes factors are used to normalise the probability integrals. For each object we obtain the resulting "X\% probability" that the object is drawn from the redshift distribution $p(z)$, while with ($100-$X)\% probability it is drawn from an unknown distribution. The use of these explicit models allows the probability of an unknown redshift to be measured on a per-object basis, sensitive to the local completeness in its own region of feature space. Thus we can flag more easily which specific objects in the query sample have uncertain estimates. 

The code is currently not sufficiently documented to be published, but it has been used with pre-calculated template grids by \citet{W99} in CADIS and \citet{W04} in COMBO-17; it has also been used with empirical models for obtaining the more challenging photo-z estimates of QSOs from SDSS photometry \citep{W09}, where redshift ambiguities are more common than with galaxies at low and moderate redshift. There the PDFs were shown to be particularly successful in predicting true redshift frequencies and thus relative probabilities for alternative ambiguous solutions. It was further tested with a smaller and spectroscopically incomplete model sample at very faint magnitudes in the comparison by \citet{PHAT}, where it naturally could not play out its advantages. The code uses an implementation of the above equation in C that reads FITS tables with data and models, and uses wrapper scripts to handle the metadata for features and models.


\subsection{Empirical Method: Neural Networks}

Neural Networks (NNs) are a collection of {\it neurons} arranged into layers. In the simplest case there exists an {\it input layer}, a {\it hidden layer} and an {\it output layer}. Each neuron is connected to all the other neurons in the previous layer, and these connections are assigned a specific weight. As the NN `learns' from the training data these weights are adjusted. The output of a neuron is a scalar quantity, therefore, each neuron is a function which maps a vector to a scalar quantity. As the input vector reaches a threshold value the neuron {\it activates}, that is, the output value changes from zero to a non-zero value -- the biological parallel is the firing of a neuron in a brain. This process of activation is controlled by an activation function, in our case this is a $\tanh$ function. 

In order to obtain redshift PDFs from the network instead of single-value estimates, we design it as a probabilistic neural network \citep[PNN,][]{PNN}, and thus map the estimation problem to a classification problem, where the classes are fine redshift bins; a photo-z application of this type was published by \citet{Bon15}. We used the public code {\tt Skynet} \citep{2014MNRAS.441.1741G}, which was also tested in the Dark Energy Survey comparison by \citet{San14}. It returns a weight for each object and class and turns these weights into a probability distribution using a {\it softmax} transform. For the NN architecture we used three hidden layers with 30, 40, and 50 neurons per layer. The input is 10 variables (magnitudes and colours) and the output is the redshift PDF divided into 50 separate redshift bins. We chose 50 bins from $z=0.002$ to $z = 0.40$ and chose the truncation at $z=0.4$, because the code could not handle the limited number of higher redshifts available. Also, with a constant bin width the number count in each class would vary significantly and cause the accuracy to vary with redshift. To avoid this, we adjust the width of each bin such that the number count distribution is as uniform as possible, but we cap the width at a maximum of $\Delta z=0.04$. As a result we have $N \sim 1100$ in each bin. Note that, when weights are introduced into the training process, the bin sizes are adjusted to make the weighted number count uniform.

In order to train a network, we need to break the spectroscopic dataset into a {\it training} and {\it testing} sample. For our case, the testing sample will also act as the {\it validation} sample. The training sample is used to infer the mapping from feature space to label space, while the independent testing sample is used to evaluate the performance of the redshift estimates. However, a problem with NNs is their sensitivity to noise, i.e. statistical fluctuations in the data. This occurs when network architectures are overly complicated with too many neurons. We note this problem is avoided in ${\tt Skynet}$ because of the convergence criteria implemented. While the network is training, the algorithm computes the sum of the squared redshift errors, for both the testing and training samples. When the fit to the testing sample begins to worsen, the code stops changing the network. 


\subsection{Empirical Method: Boosted Decision Trees}


We begin with a series of objects each described by a number of variables, we then define each object as either {\it signal} or {\it noise}. The objects within a predefined redshift bin will be labeled signal, and those outside this bin will be labeled noise. 

A decision tree works by iteratively dividing these objects into separate `nodes' based on a single variable at a time, where each node corresponds to a different region in parameter space. The point of division is chosen as the value which maximizes the separation between the signal and the noise. This division continues until a stopping criteria is achieved. The final nodes are labeled leafs, where each is classified as either signal or noise, finally the sum of these nodes is labeled a {\it tree}.

Decision trees can be sensitive to unphysical characteristics of the training data given they are unstable. This feature can be mitigated by the use of {\it boosting}, which works by re-weighting the objects that were previously mis-classified and then training a new tree \citep{Hastie08}. This allows one to generate multiple trees. A boosted decision tree (BDT) is formed using a weighted vote of all these trees, where the weight is computed from the mis-classification rate of each tree.   

To generate estimates of the photo-z PDF we use the boosted decision trees algorithm implemented in {\tt ANNz2} \citep{2015arXiv150700490S}. This code makes use of the machine learning methods implemented in the TMVA package4 \citep{Hoecker07}. Note other similar prediction trees algorithms have been introduced by \citet{Gerdes10} and \citet{CK13}. The advantage of BDTs over other machine learning algorithms is their simplicity and the speed at which they can be trained. Moreover, BDTs are one of the most effective methods to estimate photometric redshifts, as has been demonstrated in other comparisons \citep[e.g.][]{San14}.

For each {\rm class} $30$ different BDT are trained and then an optimal group is selected (the BDTs differ by the type of boosting, number of trees, etc.). An optimal group is found by ranking solutions with a separation parameter, this quantifies the level of distinction between the noise and signal. A PDF is then constructed from this ensemble of solutions, where each solutions is weighted and includes a contribution from the error estimated by each BDT.

To generate our results we include the following configuration settings: (1) The ratio of background (or noise) objects to signal objects is adjusted to be between 5 and 10, as a too large ratio will introduce a bias. (2) When defining the background sample for each redshift bin objects within the redshift interval where $\delta z  = z - z_{\rm bin}\sim 0.08$ are excluded.

%\item{When training the BPTs we consider two cases, one in which the input data is uniformly weighted, and another where the data is weighted to ensure our training sample matches the full photometric sample.}


\begin{figure*}
\centering
\includegraphics[clip,angle=270,width=\hsize]{phz_query_2dfgals_2_05_zz18.ps}
\caption{Spectroscopic vs. photometric redshift in a bright (top) and faint (bottom) $r$-band magnitude bin. In this figure we draw attention to the very few outliers and redshift trends of the mean bias. {\it From left to right:} Kernel-Density Estimation (our code), Boosted Decision Trees (using ANNz2), Probabilistic Neural Net (using Skynet), Templates assuming zero-point uncertainty $0\fm05$ (using BPZ), and assuming $0\fm18$. Our template results (two columns on the right) show larger biases than the empirical methods (see discussion); they also could not use the WISE bands as these are not covered by the templates. 
\label{zz}}
\end{figure*}

\begin{figure*}
\centering
\includegraphics[clip,angle=270,width=0.73\hsize]{stats_2dfgals_2_05_biasrmsE.ps}
\includegraphics[clip,angle=270,width=0.73\hsize]{stats_2dfgals_2_05_biasrms.ps}
\caption{Redshift statistics vs. $r$-band magnitude. {\it Left:} Mean redshift bias: intrinsic distribution (by definition zero, black solid line) vs. different methods. {\it Centre:} Half-width $\sigma_{683}$ of redshift deviation $\delta z/(1+z)$ containing 68.3\% of all objects. {\it Right:} Fraction of ''outliers'' with $|\delta z/(1+z)|>0.1$. {\it Top row:} Comparing the three different empirical methods and intrinsic redshift distribution (thick solid line labelled $r$). Note that the PNN method has excluded the $z>0.4$ galaxies, which would otherwise contribute to the outlier statistics. {\it Bottom:} Comparing the BPZ template method (using only $ugriz$) with the KDE method (two similar solid lines, one using all bands, one only $ugriz$) as well as the intrinsic redshift distribution (KDE using only $r$-band, thick line). 
\label{biasrms}}
\end{figure*}


\subsection{Bayesian template method BPZ}

As an example of a template-fitting method for photometric redshift derivation, we have chosen the Bayesian Photometric Redshift code \citep[BPZ,][]{B00,Coe06}, which is also the default method adopted by the KiDS survey \citep{deJong13}, although \citet{H16} relied on a spectroscopic recalibration of the obtained $p(z)$ estimates for the cosmic shear study in KiDS. BPZ applies Bayesian inference to estimate photometric redshifts by comparing broad-band photometry of a source with a set of redshifted template spectra. While the BPZ code is publicly available \citep{B11}, we used a slightly modified version which uses the numpy python package \citep{numpy} instead of the original numeric.

We used the re-calibrated template library of \citet{Cap04}, and a set of filters appropriate for the OmegaCAM instrument. Note that, unlike in the empirical approaches, we have not used the WISE photometric information as the templates do not cover the mid-infrared wavelengths.

In general, the BPZ code requires only a few parameters to be tuned, such as a bandpass used for the determining a redshift prior from the magnitude, the form of the prior itself, and the estimated uncertainty in the calibration zero point (on a band-to-band basis). As bandpass for the prior we used the $i$-band. The functional form of the default prior in BPZ was derived from the Hubble Deep Field North \citep[HDF-N, for details see][]{B00} and the Canada-France Redshift Survey \citep[CFRS,][]{Lilly95} and is optimised for high-redshift galaxies. Most of our galaxies are, however, located at relatively low redshifts, where the default prior performed badly. We thus adopted the prior by \citet{R14}, which takes into account the galaxy distribution at $i<20$ from VVDS \citep{VVDS} and is more appropriate for our data set.

We ran BPZ with several values for the zero-point uncertainty: initially, we used a fiducial estimate of $0\fm05$ (labelled BPZ05 in the following), but then varied the value as a free parameter and evaluated the results in terms of the mean bias and scatter of the photometric redshifts relative to the spectroscopic ones. We found that for our sample an uncertainty of $0.18$ gave on average optimal results, although it leads to other issues (see the discussion below).


\section{Results}

A first qualitative impression of the results is provided by Fig.~\ref{zz}, where spectroscopic redshifts are plotted vs. photometric ones for all four methods including the two versions of the BPZ template method. The two rows of the figure show a bright and a faint magnitude bin, with fainter galaxies obviously reaching higher redshifts. The closer the objects stay near the diagonal the better. The first obvious conclusion is that empirical methods hug the diagonal more closely than the template method, which is theoretically expected in the presence of rich and complete training sets. The difference between empirical and template methods is further explored in Fig.~\ref{biasrms} and Sect.~\ref{CompEmpTemp}, but first we discuss the results from the empirical methods themselves.


\subsection{Role of different passbands}

In this section, we first look at results obtained with a single method (KDE) but different sets of passbands. We assume that different empirical methods would find similar patterns for how results depend on data. In Tab.~\ref{rmsstats} we compare the redshift accuracy $\delta z/(1+z)$ in terms of the half-width of an interval containing 68.3\% of the sample, $\sigma_{683}$. The well-known factor $1+z$ accounts for the change in bandpass resolution with increasing redshift. We split the results into the bright sample from 2dFGRS ($r<17.7$) and the faint sample from 2dFLenS ($r=[17.7,19.5]$), as well as into red and blue galaxies using a rough bimodality cut defined by $g-r=0.5+2.8 z$ (based on the minimum between two modes in an observed-frame colour histogram after removing the mean slope with redshift). The empirical methods have two obvious trends in common:

\begin{enumerate}
 \item Bright objects have more accurate redshifts compared to faint objects.
 \item Red objects have more accurate redshifts compared to blue objects.
\end{enumerate}

Trend (i) makes clear that generic statements about the redshift accuracy of some photo-z method are meaningless, unless accompanied by a specification of the magnitude and the photometric signal-to-noise. At first sight, we may be tempted to consider (i) a consequence of photometric errors increasing for fainter objects. However, our relatively deep photometric data has nearly vanishing formal flux errors (see Tab.~\ref{tab:ATLAS_charac}), and hence our analysis takes place entirely within the saturation regime of photo-z quality \citep[for more details on error regimes see][]{WMR01}. Since magnitude has little effect on the photometric error ellipsoids in this work, any magnitude dependence of redshift errors must be driven by other factors; two possibilities are:

\begin{itemize}
 \item Calibration offsets between the model and data.
 \item The width of the intrinsic redshift distribution is a function of magnitude.
\end{itemize}

Given that we homogenised photometric zeropoints across our survey area, we don't expect model-data offsets in our empirical methods. However, we observe a strong trend in the redshift distribution with magnitude, both in the mean redshift and its scatter. We created a mapping of the form $z(r)$, based solely on the $r$-band magnitude while ignoring all SED information, and find that $\sigma_{683}$ increases from 0.017 at $r\sim 15$ to 0.1 at $r\sim 19.5$. The entire curve over five magnitudes of width is well fit by the relation $\log \sigma_{683} = -4.363+0.175 r$, i.e. per magnitude the true redshift scatter increases by a factor of $\sim1.5$. 

We simply implemented this process as a single-feature photo-z using the KDE formalism. For the subsamples, we find an average error of $\sim $0.04 in the bright sample and $\sim $0.08 in the faint sample, but of course these errors are simply a propagation of the width of the redshift distribution at fixed magnitude. Adding SED information to the process will shrink these errors, but in the faint sample we are starting from a wider base and don't expect to arrive at the same precision.

Trend (ii) has been observed for many years in photo-z's derived with only broad passbands using templates, where the stronger colour of red galaxies translates into a greater change of colour with redshift and hence a stronger signal\footnote{We note that medium-band filters tend to pick up emission lines as well \citep[e.g.][]{H94,WMR01,W04,W08,I09} and give blue objects an advantage they don't have in broadband data sets.}. However, the empirical methods show an additional influence from the intrinsic redshift distribution. The single-band photo-z's show again that at fixed magnitude red galaxies have a smaller redshift scatter than blue galaxies, even though both the red and the blue query sample have their redshifts estimated from the same overall map. In the fainter 2dFLenS sample, blue galaxies have an over 50\% wider intrinsic redshift scatter than blue galaxies, which is just the flip side of blue galaxies at fixed redshift showing a wider range of magnitudes, while red galaxies show a more peaked luminosity function \citep{Wolf03a,Bal04,Bell04}.

The most significant single-band addition we can make to the $r$-band is the W1 magnitude. At faint magnitudes where the intrinsic redshift distribution is wide, it helps reduce the redshift errors, e.g. from 0.08 to 0.055 in the overall 2dFLenS sample (using the same code). At $r\sim 15$, however, the intrinsic redshift distribution is narrowly concentrated to low redshifts of 0.025 to 0.06 (68th percentile), so there the W1 information is largely redundant with the $r$-band. Adding all bands to the process finally shrinks the redshift errors almost by half compared to $r$+W1 only. We note, that using the optical $ugriz$ bands without WISE photometry gives results that are 5 to 10\% worse than the full set including W1 and W2. This difference is small compared to those observed between some of the methods (see Tab.~\ref{rmsstats} and the following section).

At this point adding further broad passbands would largely add redundant information as there are no ambiguities to break, unless they probed additional sharp features outside the optical+WISE wavelength region. We thus conclude that additional improvements in redshift precision for galaxies at $r<20$ would mostly be enabled with filter sets that include narrower passbands to increase the sensitivity of colour measurements to smaller changes in redshift \citep{H94,WMR01,Spitler14,PAU,JPAS}, which routinely achieve very low bias and a redshift scatter of $\sim $0.007 for both galaxies and QSOs, even with template methods \citep{W04,W08,I09,B09}. 

\begin{table}
\caption{Redshift scatter $\sigma_{683}$, the 68th percentile of $\delta z/(1+z)$ by method and sample. Note that the BPZ code has used only the $ugriz$ bands. The mix of red:blue is 60:40 in the bright and 40:60 in the faint sample. \label{rmsstats} }
\begin{tabular}{lcccccc}
\hline  \hline  
			&  \multicolumn{3}{|c|}{$r<17.7$}	&  \multicolumn{3}{|c|}{$r=[17.7,19.5]$}	\\ 
Method		&  GRS		& red	&  blue	&  LenS 		&  red	&  blue	\\ 
\hline 
KDE $r$ only 	&  0.0401		&  0.037	&  0.045	&  0.0820		&  0.063	&  0.096	\\
KDE $r$+W1 	&  0.0338		&  0.033	&  0.034	&  0.0559		&  0.049	&  0.060	\\
KDE $ugriz$ 	&  0.0204		&  0.018	&  0.023	&  0.0310		&  0.025	&  0.035	\\
\hline
KDE all 		&  0.0179		&  0.016	&  0.021	&  0.0296		&  0.025	&  0.034	\\
BDT all 		&  0.0191		&  0.018	&  0.021	&  0.0315		&  0.027	&  0.035	\\
PNN all		&  0.0230		&  0.023	&  0.022	&  0.0354		&  0.032	&  0.037	\\
\hline
BPZ05 $ugriz$	 &  0.0454		&  0.038	&  0.057	&  0.0566		&  0.040	&  0.069	\\
BPZ18 $ugriz$	 &  0.0321		&  0.028	&  0.041	&  0.0613		&  0.047	&  0.070	\\
\hline
\end{tabular}
\end{table}


\subsection{Comparing different empirical methods}\label{CompEmpEmp}

There are two differences between the Kernel Density Estimation (KDE) method on the one hand, and the Boosted Decision Tree (BDT) and Probabilistic Neural Network (PNN) methods on the other: (i) The latter require a training procedure, and thus a split between training and validation sample, while KDE has no training process, and (ii) the trained methods didn't work at $z>0.4$, while for KDE we included in the model sample all galaxies up to $z=0.5$, which includes an additional $\sim $3\% of galaxies at faint magnitudes. In the query samples, we kept higher-redshift objects up to $z=0.6$ for KDE, above which all objects at this magnitude are QSOs that are identifiable in a photometric classification \citep[see e.g.][]{WMR01,W04,Saglia12,Kurcz16}.

In our comparison KDE is expected to be by far the slowest of all methods in terms of computer runtime, because it does not train a mapping, but calculates it on the fly during the estimation process. However, it leads in the precision of photo-z estimation in terms of both redshift bias and scatter (see Tab.~\ref{rmsstats} and Fig.~\ref{biasrms}), with BDT coming second in every single statistic, and PNN coming last in all of them. The difference is most apparent among red galaxies, where the redshift scatter in KDE is 20\% to 30\% lower than in PNN, while for blue galaxies the gain in KDE is only up to 10\%. BDT is consistently in the middle. We note that we find no significant redshift bias at all in the KDE method. The measured deviations are confined to approx. $\pm 0.001$ with the exception of the faintest quarter magnitude bin, and are consistent with the Poisson noise expected from the finite number of objects in a bin and their redshift scatter.

The lead of KDE in terms of minimised bias is theoretically expected. One variant of KDE has been shown by \citet{W09} to produce an exact frequency correspondence between query and model data, with zero bias and the only difference between true and estimated frequency being the propagation of Poisson noise arising from finite sample sizes. This variant of KDE is an exact implementation of Bayesian statistics with an empirical model sample, where the kernel function is chosen properly to take into account the feature errors in query and model objects leading to a ``zero-neighbourhood-smoothing KDE'' method. This approach takes into account that the original distribution of the model sample in feature space has already been smoothed by their photometric errors. In order to match it with the error-smoothed distribution of the query sample, the kernel smoothing should be restricted to the square \textit{difference} between query and model errors. However, the zero-smoothing approach requires that model errors are smaller than query errors, which is not the case in our study. Hence, we were only able to use a standard KDE method, which still comes close to zero bias. In Sect.~\ref{freqnz} we explore further to what extent the use of the standard KDE method implies differences between true redshift frequencies and those predicted here.

The BDT method performs nearly as well in terms of scatter, but less well in terms of bias. It demands more richly populated bins for training, and hence did not work for redshifts above $z=0.4$; and it shows a mild redshift bias at the low redshift end, and especially at higher redshifts towards $z=0.4$. Biases are expected when redshift is estimated via a classification approach working in fine redshift bins that become less fine where the training sample gets sparse. The redshift bins at lowest and highest redshift are the widest as the sample is sparsest there, and the need for training with such classes means that the full redshift resolution of the training set is not exploited. 

The PNN method performs slightly less accurately and is more biased at low and high redshifts. This is a modest disadvantage of NNs in the presence of rich training samples, where the smoothness of the map enforced by neural nets does not allow the full feature-space resolution of the training sample to express itself in the estimation process. However, the positive aspect of the smoothness criterion is that traditional regression NNs can outperform other methods when working with a sparse training sample, but this advantage is not expected to help with PNNs, where each redshift bin needs to be well-populated.

We note here a similarity with another method, not pursued in this work: Local Linear Regression (LLR) has been used by \citet{Csa03,Csa07} and \citet{Beck16} to derive photo-z's from SDSS photometry and empirical training sets. In this method a hyperplane is fitted to the redshifts of nearest-neighbour galaxies for each individual query galaxy, which implies and exploits a smooth (locally linear) colour-redshift relation to obtain more accurate redshift estimates even for locally sparse training samples. However, the construction of redshift distributions $p(z)$ does not benefit from this technique. 

The right-hand column of Fig.~\ref{biasrms} shows the rates of redshift ''outliers'', defined here as $|\delta z/(1+z)|>0.1$. These are very few objects, generally within less than 1\% at $r<18$, and increasing mildly at fainter levels. The methods appear comparable and their differences are largely due to statistical noise. It is important to clarify that these objects are not outliers in a classical sense, where true redshift ambiguities at one location in feature space leads to confusion. Instead, these objects are simply the wings of the error distribution and their fraction beyond a fixed threshold increases just as the width of the distribution increases. Fig.~\ref{dzhisto} shows the redshift error distributions of the three methods for the two samples, and proves that their distributions are nearly perfectly Gaussian apart from noise. We note that vertical offsets in Fig.~\ref{dzhisto} stem from the fact that KDE can consider every object a query object, while BDT and PNN need separate validation samples. This result is consistent with the general observation that at $r<20$ we observe a galaxy population confined to $z<0.6$ and without real ambiguities. 

However, this simplicity of the $r<20$ universe is in marked contrast to fainter data sets: at $r>20$ and $z>1$ significant ambiguities come into view, which then makes the consideration of redshift outliers worthwhile. In that different regime, we expect to see a difference in outlier handling by the KDE and BDT method: the boosting in BDT may act to suppress the propagation of unlikely signals into the trained map, as it is designed to suppress the propagation of rare false training signals. This would suppress the visibility of ``faint'' (high number-ratio) ambiguities in BDT, whereas KDE would take the model sample at face value. Thus BDT will produce cleaner results with messy training samples, while KDE will report fainter ambiguities than BDT provided the model samples are clean and trustworthy.

In summary, it appears that we have reached the intrinsic floor of redshift errors afforded by the data used in this study. The richness of the model sample has allowed KDE to outperform the neural nets and boosted decision trees and minimise the bias.

\begin{figure}
\centering
\includegraphics[clip,angle=270,width=\hsize]{dzhisto.ps}
\caption{Histogram of redshift deviations $\delta z/(1+z)$ for empirical methods. {\it Left:} Bright 2dFGRS sample. {\it Right:} Faint 2dFLenS sample. There is no sign of unusual outliers, which is expected for rich, complete, random training samples at $r<20$ and $z<0.6$, where no ambiguities appear in the colour-redshift map, although they contribute to the marginally richer-than-Gaussian tails of the distribution. 
\label{dzhisto}}
\end{figure}


\begin{figure}
\centering
\includegraphics[clip,angle=270,width=\hsize]{z_cplt.ps}
\caption{Completeness of high-quality model as seen by the query sample of high-quality objects. {\it Left:} Per-object fraction of integrated frequency in high-Q sample (grey points) and mean trend (line). At $r>19$ there is a $\sim $15\% probability that an object's redshift is not drawn from the distribution of the high-Q sample defining the redshift map, but from the low-Q sample with unknown redshifts. {\it Right:} The high-Q fraction is lowest for faint red galaxies, whose spectra display only noisy absorption lines. 
\label{cplt}}
\end{figure}


\subsection{Spectroscopic completeness and residual estimation risk}

Spectroscopic incompleteness in an empirical model sample leads to a gap in our knowledge of the true $z(\vec{c})$ map, which implies that the estimated redshift probability distribution $p(z)$ is only part of the whole picture. While objects with unknown redshifts cannot be used to quantify redshift performance, they help at least to flag residual estimation risks due to this empirical gap. Is there a risk of getting photo-z's wrong when the spectroscopic sample is incomplete? The problem is that it is a-priori unclear, at what redshifts those objects reside, which were targeted by spectroscopy, but did not deliver a trustworthy redshift. The two possibilities here are: 
\begin{enumerate}
\item Those objects are a random subsample of objects with similar SEDs, so they reside at similar redshifts, and we have no redshifts simply because the signal in the spectrum was a little too weak
\item Those objects did not reveal their redshift precisely because they reside at redshifts different from the successful sample, which is why no strong features were seen that would have given away their redshift.
\end{enumerate}

These two alternatives have different implications for the photo-z misestimation risk: in case (1) the photo-z PDF will be correct after re-weighting, while in case (2) the shape of the photo-z PDF must be acknowledged to be wrong since the empirical method is blind to an important component of the redshift space. We don't know for sure, which case is realised here, but we have reasons to assume that at the moderate depth of $r\sim 19$ our incompleteness might be of the benign sort, as there is not much room for a significant fraction of objects to reside e.g. in the redshift desert at $z>1$. The situation is known to be different at fainter levels, where higher-z objects are selectively missing from spectroscopic samples \citep{DeepPC}.

In the KDE framework we can easily measure the probability $p_{\rm unknown}$ of any object to be attributed to the unknown redshift class. The mean $p_{\rm unknown}$ increases towards faint magnitudes in step with the spectroscopic incompleteness. The left panel in Fig.~\ref{cplt} shows that $p_{\rm unknown}$ increases significantly at $r>19$, up to $\sim $15\%. 

The right panel in Fig.~\ref{cplt} illustrates that redshift completeness is a function of galaxy colour. There appears to be almost a bimodality in the high-quality probability fraction of galaxies, such that red galaxies have systematically lower redshift completeness. This is expected since blue galaxies are star-forming and show a clear emission-line signature, which leads to high-confidence quality flags even for faint galaxies. The mean incompleteness for red galaxies with $g-r\approx 1.4$ and $r=[19,19.5]$ is $\sim $20\%, but some individual high-quality query objects fall into regions of the map, where the high-quality completeness goes to nearly zero.

\begin{table}
\caption{Outlier rates $f_{0.1}$ of KDE photo-z by spectroscopic quality flag Q. Rates are statistically consistent with the expected Q-dependent fraction of spectroscopic misidentifications.  \label{outliers} }
\begin{tabular}{lccrrrrr}
\hline  \hline  
Sample	&  mag range	& $\langle z\rangle$	&  Q4/5	&  Q3	&  Q2	&  Q1	\\ %& Hi-Q (3-5) \\ 
\hline 
2dFGRS	& $r<17.7$	& 0.111			&  0.1\%	&  1.7\%	&  13\%	&  -		\\% & 0.2\% \\
2dFLenS	& $r>17.7$	& 0.201			&  1.4\%	&  2.2\%	&  17\%	&  30\%	\\% & 2.1\% \\
 \hline
\end{tabular}
\end{table}


\subsection{Spectroscopic mistakes and outlier rates}

One of the reasons for the appearance of redshift outliers is errors in the spectroscopic identification, the rate of which will depend on the redshift quality flag assigned during the spectroscopic inspection. A quality flag of 4 or 5 indicates redshifts that seem absolutely certain to the human inspector. Accidental mistakes may still happen, but this quality class is expected to be notionally "99\% correct". With decreasing quality class we expect the fraction of incorrect redshifts to grow. $Q=1$ is assigned to all spectra where the human inspector cannot see any significant feature, and the redshift is taken from a cross-correlation fit. The bright 2dFGRS sample contains no objects with $Q=1$, but 3\% of the fainter 2dFLenS sample fall into this category.

We define outliers using $|\delta z/(1+z)|>0.1$ and measure their fraction among KDE photo-z's after splitting the sample by quality flag. Table~\ref{outliers} shows that outlier rates are lower for 2dFGRS than for 2dFLenS. This is a consequence of their intrinsic difference with mean redshifts of $\langle z_{\rm GRS} \rangle = 0.11$ vs. $\langle z_{\rm LenS} \rangle = 0.20$. Outliers as defined here, with a redshift error greater than 0.1, are almost ruled out by definition in a bright sample at consistently low redshift. 

Among Q=4 and 5 objects we find outlier rates on the order of $\sim $1\%, which is fully consistent with the notional target reliability of 99\% for this quality class. We checked the hypothesis that those outliers might be due to wrong spectroscopic redshifts rather than wrong photo-z's. Revisiting their spectra, we found that less than 20\% of these conflicting cases had uncertain or wrong redshifts assigned. We have not yet investigated in detail what explains such outliers that are not expected in the absence of ambiguities in the colour-redshift map. However, we have evidence that source blending and strong lensing play a role, where two galaxies at different redshifts appear on nearly the same line-of-sight. This affects the measured photometry of the combined object and makes the redshift itself ambiguous.

We also find that the outlier rates increase with decreasing spectroscopic quality, as expected. They reach $\sim $2\% for Q=3, $\sim $15\% for Q=2 and $\sim $30\% for Q=1. It appears credible that the increases above the rate for the highest-quality bin are mostly due to spectroscopic data quality issues. Bad seeing, high background and faint objects all lead to lower signal-to-noise, and scattered light from nearby objects confuses the situation. Spectroscopic quality decreases for fainter objects and generally for worse observing conditions. However, there do exist galaxy spectra, which are intrinsically challenging due to weak lines, e.g. when low metallicity is combined with low star-formation rate. We repeat, however, that in our magnitude range we do not expect any galaxies from the redshift desert of optical spectrographs at $z\ga 1$.

The spectroscopic redshift of objects with Q=1 is almost uniquely determined by cross-correlation of the spectrum with templates, and specifically marks cases with a very weak correlation coefficient. A $\sim $30\% outlier rate above a threshold of $|\delta z/(1+z)|>0.1$ is consistent with either the photo-z or the spec-z being drawn at random from the full sample, while the other one is assumed to be correct. So, this could mean that the Q=1 cross-correlation redshifts are random, while the photo-z's are actually correct, or the photo-z's are random while the low cross-correlation redshifts are correct, or any mix of the two. Only for Q=2 and better quality levels, a human assessor has actually labelled our confidence in the spectroscopic redshift as "possible" (Q=2), "probable" (Q=3), and "practically certain" (Q=4/5).


\begin{figure}
\centering
\includegraphics[clip,angle=270,width=\hsize]{relerrors_2_05.ps}
\caption{Validity of per-object estimated redshift probability distributions. {\it Left:} Ratio of estimated rms and true redshift rms. {\it Right:} Distribution of true redshifts within estimated redshift distribution. The kernel smoothing in standard KDE makes estimated redshift distributions wider than true redshift distributions, so that true redshifts are not randomly drawn from the estimated distribution. This effect is noticeable for the brighter objects in 2dFGRS, but negligible for the fainter 2dFLenS objects. The excess in the histogram bins at both ends of each distribution are redshift outliers. 
\label{relerrors}}
\end{figure}


\subsection{Frequentist interpretation of $p(z)$}\label{freqnz}

It is often asked whether photometric redshift probability distributions represent actual redshift frequencies for objects of the colour they were derived for. If this was the case, we could choose any subsample and add their $p(z)$ to obtain a realistic $n(z)$ estimate. \citet{W09} have shown that in the limit of zero-smoothing KDE such estimates deviate from the true $n(z)$ only by Poisson noise. However, zero-smoothing KDE can only be implemented if the model photometry is less noisy than the query photometry. In this work model and query objects have statistically the same errors, which in zero-smoothing KDE implies a kernel that is a $\delta$-function. Choosing a kernel of finite width to avoid a breakdown of the method resulting from the discrete nature of the model applies additional smoothing to the $z(\vec{c})$. This kernel smoothing introduces neighbourhood information into the model where it is not expected in the data. The effect is expected to be strongest for bright objects, while the distribution of faint objects in feature space is already smoothed by larger photometric errors so that additional smoothing has less effect. We use two diagnostics to test whether $p(z)$ statistically represents $n(z)$. 

First we determine the rms redshift error $\sigma_z$ expected from the $p(z)$ and compare it to the true redshift error $\delta z$. For an accurate $p(z)$ we expect the ratio of $\delta z/\sigma_z$ to have zero mean and rms $=1$. In the left-hand panels of Fig.~\ref{relerrors} we show the distribution for the bright sample with $r<17.7$ from 2dFGRS (top row), and that for the faint sample at $17.7<r<19.5$ from 2dFLenS (bottom row). We first note the presence of $>3\sigma$-outliers, where the true redshift is much further from the estimate than the width of the $p(z)$ distribution suggests. Outliers were discussed in the previous section. We then see that the shape of the distribution is approximately Gaussian with an rms of 0.853 for 2dFGRS and 0.975 for 2dFLenS. So, the width of the $p(z)$ distribution seems on average appropriate for the fainter 2dFLenS sample, while the kernel smoothing overestimates the expected errors for its redshift estimates of the bright 2dFGRS sample, just as expected theoretically.

Now we probe the shape of the $p(z)$ distribution: we check whether the spectroscopic redshifts $z_i$ are randomly drawn from $p_i (z)$ on a per-object basis. For this purpose, we measure the fraction of the normalised estimated probability below the true redshift:

\begin{equation}
	 {\int_0^{z_{\rm spec}} p(z) dz} \Big/ { \int_0^{z_{\rm max}} p(z) dz} 	~.
\end{equation}

If the $p(z)$ is correct, we expect the distribution of the fractional probability to be precisely flat apart from noise. The right column in Fig.~\ref{relerrors} shows that this is indeed the case for 2dFLenS (bottom right panel): the distribution is flat and the median value of the sample is $0.508$, very close to the expected value of $0.5$. If we clip the outlier bins (see below), we find mean numbers per bin of $\sim $348 and an rms of $\sim $20, very close to Poisson noise. Any smaller sub-sample selected from the whole 2dFLenS sample would be even more dominated by Poisson noise, so that possible systematics would be even harder to detect. We thus take it as established that the KDE $p(z)$ estimates in any subsample at $r\ga 18$ from our dataset are consistent with true redshift frequencies plus Poisson noise.

The brighter 2dFGRS sample shows a similar median value, but the sources are found to be concentrated more toward the centre of their estimated distribution than suggested by the distribution itself. This corroborates the previous finding and theoretical expectation that the kernel smoothing has added excess probability to the wings of the expected distribution for bright objects. 

Finally, the bins at the ends of the distributions mark objects whose redshift is not within the central 96th percentile of their estimated redshift distribution, and the excess in those bins amounts to $\sim $700 objects or a $\sim $3.5\% fraction among the total 2dFLenS sample and $\sim $500 objects or $\sim $1.5\% of the 2dFGRS sample. These objects are clear outliers from their estimated $p(z)$, and may be due to source blending.


\subsection{Comparing template methods with empirical methods}\label{CompEmpTemp}

We first note that template methods have more room for mismatches between model and data than empirical methods, which lead to redshift biases when left uncorrected. We note the following issues: 

\begin{enumerate}
 \item Templates may not reliably cover all the features probed by the observations because their physics may not be understood and calibrated well enough to make a parametric model that is bias-free. E.g, we did not use the WISE W1 and W2 bands in the template comparison as the templates are unreliable at this wavelength. We may expect a little less redshift discrimination in the BPZ results, but note that excluding WISE made less than a 10\% difference to the KDE results.
 \item Redshift priors in template methods depend on the chosen template grid and will be sub-optimal given that template grids are a simplified view of the true SED space occupied by real galaxies. We found that the default prior of BPZ was inappropriate for $r<20$ objects, as it had been tuned for fainter galaxies in deep surveys, so we chose a different prior from the literature, which may still not be a perfect match.
 %\footnote{We also tried an $r$-band prior instead of the default $i$-band prior, but this made the results worse.}.
 \item Calibration mismatches between templates and photometric data have been reported fairly often, and can be addressed with photometric zero-point adjustments in the observed frame as well as with template repair methods in the restframe \citep[e.g.][]{Csa00,Bud00,Bud01,Csa03}. 
\end{enumerate}

Fixing any of these three issues is beyond the scope of this paper, and thus our results for the template method remain probably sub-optimal. As we see in Fig.~\ref{zz} and Fig.~\ref{biasrms}, the most obvious difference between our photo-z results from template methods and empirical methods are redshift biases that are a function of redshift.

In contrast to our application of the template method, an empirical method can make use of all features observed in the data set. When it is based on a random sample drawn from a large volume in a homogeneous Universe observed with homogeneous photometric zero-points, it will also naturally take care of the calibration and the prior of the model. Everyone with a rich complete training set has the option to use an empirical method, unless extrapolation in magnitude, redshift or object type are required, which would then involve uncertain parametric assumptions. 

\begin{figure}
\centering
\includegraphics[clip,angle=270,width=\hsize]{zR.ps}
\caption{Redshifts vs. magnitude. The focus of this figure is on regions that are not populated. {\it Top left:} Spectroscopic redshift. {\it Other panels:} Photo-z by method. In BPZ18 photo-z's avoid low redshifts for fainter magnitudes as the wide minimum error and the steep redshift prior drive the estimates higher. BDT and PNN are affected by the sparsity-induced upper redshift limit of $z=0.4$. 
\label{zR}}
\end{figure}

A further source of redshift biases is the profound semantic difference between a likelihood and a probability. The estimated zero-point error plays a special role in template methods: it is usually added quadratically to an object's flux error to represent a more realistic uncertainty in an SED. This widens the Bayesian likelihood contours and protects the user against trusting optimistic error estimates. However, it is clear that the $p(z)$ functions thus obtained are likelihood estimates and not probability estimates, representing degrees of belief, and not frequencies in a random experiment. We repeat that in our case the formal flux errors are nearly vanishing and the assumed error is basically the zero-point error.

We first assumed fiducially a zero-point uncertainty of $0\fm05$, but noticed that at $r<17$ a single-band redshift estimated empirically from only the $r$-band magnitude is already more accurate than the five-band template method BPZ05 (see Fig.~\ref{biasrms}). We re-ran the code with other values and found that bias and scatter were on average reduced by increasing the zero-point uncertainty to $0\fm18$. 

However, this change also introduced specific features into the results as the inflated flux confidence intervals then corresponded to a broader range of likely redshifts. Near the edges of the redshift distribution, where the redshift priors are steep, intermediate redshifts are thus rendered more likely than the correct very low or very high redshifts. This effect is shown in Fig.~\ref{zR}, where we show the redshift estimates of the entire sample vs. $r$-band magnitude. While the estimates of $z_{\rm BPZ05}$ (with $0\fm05$ zero-point uncertainty) cover the entire space of realised redshift-magnitude realisations, those with the inflated zero-point uncertainty, $z_{\rm BPZ18}$, are confined to a much tighter redshift interval at fixed magnitude and are systematically devoid of low-redshift estimates with increasing magnitude, such that e.g. no single photo-z for $z<0.1$ occurs at $r\sim 19$. But we note, that photo-z-based science applications often require that the true redshift distribution in a photo-z bin is correct, and not the converse, which would not be helpful given that in a blind photo-z situation, there is obviously no possibility to sort objects into true redshift bins. This is indeed the case for $z_{\rm BPZ18}$ at the low-redshift end, while it is not the case for $z_{\rm BPZ05}$ at either end. 



\section{Conclusion}

In this paper, we present a new training set for photometric redshifts of galaxies, which is complete and randomly selected to $r\la19.5$, and thus two magnitudes deeper than the SDSS Main Spectroscopic Galaxy sample. It is similarly deep as the GAMA redshift sample, but randomly sub-selected from a larger survey footprint to suppress large-scale structure. One new aspect of this training sample is that we counteracted the steep number counts of galaxies by selecting brighter galaxies from a larger effective area within the same survey footprint. This reduces redundancy of information at fainter magnitudes and simultaneously reduces sparsity of representation and model discretisation noise at brighter magnitudes. Together with the 2dFGRS sample at brighter magnitudes, it includes over 50,000 redshifts selected from over 700 deg$^2$ area in the Southern sky.

Based on this training set and $ugriz$ and WISE photometry, we investigate the performance of photometric redshifts for galaxies using empirical and template approaches. All our methods are set up so that they produce redshift probability distributions $p(z)$ and not simply redshift estimates in the first instance. We investigate both the comparison of best estimates derived from the $p(z)$ as well as the precision of the $p(z)$ in actually representing expected frequencies.

We find that the template method suffers from redshift biases due to several issues that are not easy to fix. The empirical methods have lower biases, especially the KDE method, which by design has no measurable bias larger than Poisson noise. None of the methods have significant outliers, since there are simply no serious colour-redshift ambiguities at $r\la 20$ and $z<0.6$, as opposed to deeper and higher-redshift surveys. 

Two of our empirical methods, using boosted decision trees and neural nets, are trained methods and demand sufficiently dense training samples. Thus they tend to fail at the edges of the redshift distribution, where the number counts are steep and the sample becomes sparse. In contrast, the KDE method is not a trained method, hence our spectroscopic sample acts directly as a model set instead of a training set, as it is simply a discrete random representation of the galaxy redshift density field in observed feature space. 

The KDE method also achieves the best photometric redshift scatter, with values for the 68th percentile of $\sigma_{683}=$(0.014, 0.017, 0.028) for samples selected to have $r<(15.5, 17.5, 19.5)$, respectively. We note that additional improvements in redshift precision for galaxies at $r<20$ would mostly be enabled with filter sets that include narrower passbands to increase the sensitivity of colour measurements to smaller changes in redshift, but are not expected from adding further, largely redundant, broad-band information.

We reiterate that photometric redshift precision is necessarily a strong function of magnitude, since intrinsic redshift variance itself is a strong function of magnitude, increasing from 0.017 at $r\sim 15$ to 0.1 at $r\sim 19.5$. Often also the signal-to-noise of the photometric feature measurement is a strong function of magnitude, although this is not the case in our data set, where photometry is uniformly good across the entire range considered here. As a result, many statements in the literature about photometric redshift scatter contain no information unless accompanied by clear magnitude and signal-to-noise references. 

We note, however, that the realisation of the KDE method presented here is still not a textbook realisation of Bayesian statistics, as such an implementation requires photometric errors to be smaller on the model sample than on the query sample. Only then can we implement KDE in such a way that $p(z)$ represents the expected redshift frequency precisely. However, at $r>18$ we don't see a measurable deviation from the expected frequencies plus Poisson noise for any subsample of objects.

In summary it appears that photometric redshifts for galaxies to $r\la 20$ are a solved problem, and the next frontier is improving model samples at $r>20$ by making them simultaneously more complete, random and less affected by large-scale structure. A helpful project to this end, scheduled for observation in the 2020s, is the Wide Area VISTA Extragalactic Survey \citep[WAVES,][]{WAVES} planned at the forthcoming 4MOST instrument at the ESO-VISTA telescope, which may end up creating a large and complete random redshift sample to $r=22$.
 
The original motivation for obtaining this training set was to use it for deriving KDE-based photometric redshifts in the SkyMapper Southern Survey, which will see a first release of its Main Survey data in 2017. At $z<0.2$, we expect the redshift scatter to be slightly better for the SkyMapper filter set than for $ugriz$, because of the narrower $uv$ (ultraviolet and violet) filter pair in SkyMapper. For SkyMapper, we also plan a KDE classification into object types, which we will explore with further training sets that we have obtained in the realm of stars and quasars.


\section*{acknowledgements}
CW was supported by Australian Research Council Laureate Grant FL0992131.
Parts of this research were conducted by the Australian Research Council Centre of Excellence for All-sky Astrophysics (CAASTRO), through project number CE110001020.
MB is supported by the Netherlands Organization for Scientific Research, NWO, through grant number 614.001.451, through FP7 grant number 279396 from the European Research Council, and by the Polish National Science Center under contract \#UMO-2012/07/D/ST9/02785.
CB acknowledges the support of the Australian Research Council through the award of a Future Fellowship. 
CH acknowledges support from the European Research Council under grant number 647112. HH was supported by the Deutsche Forschungsgemeinschaft under Emmy Noether grant Hi 1495/2-1. TE and DK are supported by the Deutsche Forschungsgemeinschaft in the framework of the TR33 `The Dark Universe'. 
Based in part on data acquired at the Australian Astronomical Observatory, through program A/2014B/08.
KK acknowledges support by the Alexander von Humboldt Foundation.
DP is supported by the Australian Research Council Future Fellowship Grant FT130101086.
This publication used data obtained by the ESO VST-ATLAS survey under program ID 177.A-3011. It also makes use of data products from the Wide-field Infrared Survey Explorer, which is a joint project of the University of California, Los Angeles, and the Jet Propulsion Laboratory/California Institute of Technology, funded by the National Aeronautics and Space Administration. 

%\pagebreak


\iffalse


%It is further helpful to be aware of an actual methodological distinction, that between probabilistic methods and function-fitting methods. Either class of method can be used in conjunction with either class of model. When authors discuss a comparison between e.g. an empirical neural-net based approach with a Bayesian fitting of a parametric model, they may change at once two different parts in the procedure, the inference method (probabilistic vs. function-fitting) and the assumed model (parametric vs. empirical). Such a comparison makes it hard to attribute differences in performance clearly to one aspect. Did the change of model cause the difference or the change of inference method? 

%The empirical approach has evolved from the earliest attempts \citep{SW48} to discover and capture the variety of galaxy SEDs from complete galaxy samples with SED and redshift information. Algorithms have included fitting of colour-redshift curves $z(\vec{c})$ \citep{Koo85,Con95}, a method that also works to some extent for QSOs \citep{R01}, artificial neural nets \citep[e.g.][]{FLS03}, nearest-neighbour and tree methods \citep{Csa03,Oya08a} as well as support-vector machines \citep{Wa05} and boosted decision trees \citep{Gerdes10}. What these methods have in common is that they aim to fit a smooth colour-redshift relation to the empirical data without assuming additional underlying facts. Such relations can include features other than colour, e.g. size and morphology, in search of additional discriminant information for redshift. As opposed to Bayesian models, the early empirical methods produced a unique mapping from observed features to redshift, such that any object had only a single scalar value attributed, i.e. the best redshift estimate. The degree of uncertainty or ambiguity had to be discovered in a validation sample from the sample difference between true and estimated redshift and was thus not provided on a per-object basis.

%A further evolution of the empirical methods allowed these to produce redshift probability distributions as well. Neural nets were upgraded from one output value to output vectors recording a probability distribution over various alternative, allowed values, and became known as probabilistic neural nets \citep[PNN,][]{PNN}; a photo-z application of this type was published by \citet{Bon15}. Nearest-neighbour methods could produce a probability distribution after resampling an objects's photometry over its error ellipsoid \citep{Ball08,Oya08b}, and examples of further methods are mentioned below. Kernel-density estimation \citep[KDE, see e.g.][]{Wang07} was one particular method, where the Bayesian and empirical approach could be unified by using the empirical sample objects as discrete instances of a model: technically KDE with an empirical sample is identical to Bayesian model fitting, if the kernel function is chosen to be the error ellipsoid of the query object and the empirical features of the model object are free of random errors, or the kernel function is the square difference of query and model errors \citep[see][]{W09}. KDE has been successfully used to estimate redshifts for QSOs as well, where colour-redshift ambiguities are common and realistic frequency distributions are desirable \citep{R09,W09}. We refer to \citet{Bud09} for a mathematical unification of various approaches. In all cases studied below, we obtain a redshift probability distribution $p(z)$, and look at Maximum-Likelihood estimates of the $p(z)$ when measuring redshift accuracy. 

%Users of photometric redshift estimates are typically looking for one of the following three performance criteria, depending on the science application: (1) a low redshift uncertainty per-object, which is required when the observed photo-z sample is to be binned in narrow redshift bins to study galaxy evolution \citep[e.g.][]{Bell04,Spitler14} or when the redshift resolution drives the contrast for the detection of overdensities \citep[e.g.][]{Tay04,GY05,HIROCS,Muzz13,Rett14, Durret15,Rozo16,Ryk16}; (2) a low outlier rate of catastrophically wrong estimates, which is relevant when the precise mean redshift of a photo-z sample is required \citep[e.g.][]{Kit08}; or (3) a small difference between co-added redshift probability functions of small sub-samples and their true redshift frequency distribution, which is important e.g. for cosmological parameter estimation from gravitational lensing or growth of structure in photo-z samples \citep[e.g.][]{Brown03,Bacon05,Leaut12}.

%Like many other authors, we use both a method based on SED templates as well as several methods based on an empirical training set. However, we expect to find that with a complete random training set of $\sim 50,000$ spectra empirical methods are superior. Indeed, the primary aim of creating the \textit{direct photo-z sample} in the 2dFLenS Survey was to create a model sample at $r\la 20$, which would allow the derivation of empirical photo-z's with a precision that approaches the theoretical limit. In contrast, \citet{PHAT} compare a large number of methods on a deep photometric dataset from the Great Observatories Origins Deep Survey \citep[GOODS,][]{Giav04} with sparser and incomplete spectroscopy, which is the extreme opposite of the data domain investigated here.



\section{Methods for photo-z'eeing}

\subsection{The evolution of two photo-z approaches}

In the mid-1900s hardly any redshifts were known for galaxies at $z>0.1$. But reaching higher redshifts was crucial for measuring the expansion history of the universe, deciding which cosmological model explains our world and suggesting what fate might await our Universe. Higher redshift, however, implied fainter objects with insufficient flux for the spectrographs of the time. \citet{Baum57} mentions that there were four magnitudes between the spectrographic and the photographic limits of the 200-inch Hale telescope at Mt Palomar, and asserted that "observations within that territory will be vital to any definitive cosmological result". He also pointed out that "the high precision of spectrographic redshifts is not needed; we can well afford to trade a factor 10 in precision for the opportunity of reaching galaxies at larger distances" (both statements copied verbatim). Today, the emphasis is not purely on reaching depths in magnitude and redshift that would not be accessible to any spectrograph, but on getting redshifts for large numbers of galaxies that are impractical with contemporary spectrographs.

\citet{Baum57} discusses fitting the template of what we call nowadays a "luminous red galaxy" (LRG) to the optical multi-band SED of luminous cluster galaxies. These methods have evolved to include a wide range of possible galaxy template SEDs \citep[e.g.][]{CWW80,BC93,Kin96,FRV97} and priors for their likelihood of occurring at different magnitudes and redshifts \citep{HyperZ,B00}. They have evolved to separate rare QSOs from galaxies and estimate redshifts for them \citep{W99,Bud01}, and to include Bayes factors for making SED-based class decisions between stars, galaxies and QSOs \citep{WMR01}, so that star-galaxy separation does not need to rely on morphology. Such  model template methods produce Bayesian likelihood distributions over redshift for every object, so they can differentiate cases where the model allows little distinction from cases where the model suggests a well-constrained redshift.

Along a different line \citet{SW48} established an empirical relation between the photographic colours of galaxies and their redshift using the 100-inch telescope at Mt Wilson, but again this work was restricted to LRGs and calibrated only to an empirical limit at $z=0.13$. The colour-redshift relation of the cluster red sequence is still a useful technique for modern surveys such as the RCS \citep{GY05} as it requires data in only two passbands, but it needs clusters to mark red-sequence galaxies through their overdensity, since the colours of galaxies are affected by their intrinsic SED variety more than by their redshifts.

The empirical approach has thus evolved to discover and capture the variety of galaxy SEDs from complete galaxy samples with SED and redshift information. Algorithms have included fitting of colour-redshift curves $z(\vec{c})$ \citep{Koo85,Con95}, a method that also works to some extend for QSOs \citep{R01}, artificial neural nets \citep[e.g.][]{FLS03} nearest-neighbour and tree methods \citep{Csa03,Oya08a}, support-vector machines \citep{Wa05} and other flavours of what we nowadays call "machine learning methods". What these methods have in common is that they aim to fit a smooth colour-redshift relation to the empirical data without assuming additional underlying facts. Such relations can include features other than colour, e.g. size and morphology, in search of additional discriminant information for redshift. As opposed to Bayesian models, the early empirical methods produced a unique mapping from observed features to redshift, such that any object had only a single scalar value attributed, i.e. the best redshift estimate. The degree of uncertainty or ambiguity had to be discovered in a validation sample from the sample difference between true and estimated redshift and was thus not provided on a per-object basis.

A further evolution of the empirical methods allowed these to produce redshift probability distributions as well. Neural nets were upgraded from one output value to output vectors recording a probability distribution over various alternative, allowed values, and became known as probabilistic neural nets \citep{PNN}; a photo-z application of this type was published by \citet{Bon15}. Nearest-neighbour methods could produce a probability distribution after resampling an objects's photometry over its error ellipsoid \citep{Ball08,Oya08b}. Kernel-density estimation \citep[KDE, see e.g.][]{Wang07} was one particular method, where the Bayesian and empirical approach could be unified by using the empirical sample objects as discrete instances of a model: technically KDE with an empirical sample is identical to Bayesian model fitting, if the kernel function is chosen to be the error ellipsoid of the query object and the empirical features of the model object are free of random errors \citep[see][]{W09}. We refer to \citet{Bud09} for a mathematical unification of various approaches.

Users of photometric redshift estimates are typically looking for one of the following three performance criteria, depending on the science application: (1) a low redshift uncertainty per-object, which is required when the observed photo-z sample is to be binned in narrow redshift bins to study galaxy evolution \citep[e.g. bins of $\Delta z=0.1$ in][]{B04} or when the redshift resolution drives the contrast for the detection of overdensities \citep[e.g.][]{GY05,Zat08}; (2) a low outlier rate of catastrophically wrong estimates, which is relevant when the precise mean redshift of a photo-z sample is required \citep[e.g.][]{Kitching}; or (3) a small difference between co-added redshift probability functions of small sub-samples and their true redshift frequency distribution, which is important e.g. for cosmological parameter estimation from gravitational lensing or growth of structure in photo-z samples \citep[e.g.][]{B03,XXX}.


\subsection{Methodological overview}

The above historical evolution suggests that there are two profoundly different kinds of photo-z approaches, those based on parametric templates and those based on empirical data. However, we should not call them different \textit{methods} as their primary difference is in the \textit{models}, i.e. parametric template models vs. empirical models. It is further helpful to be aware of another methodical distinction, that between probabilistic methods and function-fitting methods. Either class of method can be used in conjunction with either class of model. When authors discuss a comparison between e.g. an empirical neural-net based approach with a Bayesian fitting of a parametric model, they change at once two different parts in the procedure, the inference method (probabilistic vs. function-fitting) and the assumed model (parametric vs. empirical). Such a comparison makes it hard to attribute differences in performance clearly to one aspect. Did the change of model cause the difference or the change of inference method? Hence, while we are commonly aware of two classes of photo-z approach, there are actually four possibilities.

A second often over-looked point is the usage of the term probability. Textbooks about statistical inference remind us \citep[e.g.][]{MacKay} that \textit {probability} can describe two different meanings: it can describe \textit{frequencies of outcomes in random experiments}, and it can describe \textit{degrees of belief in propositions that do not involve random variables}. \citet{MacKay} also remind us that \textit{a likelihood function is not a probability distribution} (phrases in italics copied verbatim from \citet{MacKay}). Below we explore what that means for the photo-z problem.

Ensemble definition


\subsubsection{Uncertainty, possibility and frequency}

In a standard Bayesian analysis, the model is assumed to be formally free of uncertainties. A model that is even slightly different from the data is thus only viable if the measurements have uncertainties, which of course they do. The model parameter of interest here is redshift, and the uncertainties in the measured photometry of a query object inform Bayesian likelihood distributions in redshift. The very existence of Bayesian probability distributions is tempting some users to interpret them in a frequentist manner by believing that they actually represent frequencies of redshift realisations. This, however, is a leap of confidence relying on the assumption of a perfect model. Errors in the model obviously lead to the fact that the sum of Bayesian redshift probability distributions in a sample is not necessarily close to the actual redshift frequency distribution in the sample.

For the interpretation of any statistical analysis, it is important to understand what we mean by {\it uncertainty}. Is it a lack of precise knowledge on an individual unknown object drawn at random from a known distribution? In this case, we know which values in the distribution are more {\it frequent} than others, and Frequentist statistics is possible. Or is it lack of knowledge on the whole distribution? In that case, we only know which values we consider {\it possible} and Bayesian statistics is the method of choice. In both perspectives, the term probability is used, but in the Frequentist case probability means frequency and in the Bayesian case probability means possibility when it is based on an imperfect model.

Now we need to distinguish uncertainty in the data from uncertainty in a model, as this is relevant for deriving redshift distributions for samples. In photometric surveys, uncertainty in the data are flux errors, which usually have a Gaussian distribution. For an individual object, the Gaussian centred on the measurement represents the possible true fluxes of the object. In a sample of objects the distribution of measured flux values has already been broadened by convolving the true distribution with the Gaussian uncertainty to produce the measured, wider distribution, and an underlying frequency distribution could be estimated by deconvolution. In contrast, uncertainty in a model may follow known distributions in case of an empirical method or be vaguely and qualitatively assessed as in the case of templates and associated priors.

In a Bayesian framework, it is straightforward to take the measurement uncertainties on the data into account, and, assuming the model to be correct, a reliable estimate of redshift possibility is obtained. But generally photo-z methods estimate redshifts on a per-object basis, which means that measurement uncertainties across a sample are never accounted for with a deconvolution. Summing up the Bayesian redshift distributions over all objects then produces an aggregate $\sum p(z)$ distribution that is widened over the true $n(z)$ distribution by the measurement errors.

Also, it has often been found that template models don't seem to describe the observed data perfectly, owing to uncertainties in the measurements that led to the templates or imperfections in the model underlying the creation of the templates. This has motivated the introduction of both repair methods as discussed in Sect.~3.3.2 and of 'template errors' \citep[e.g.][]{EAZY}. The template error increases uncertainty in the model-data comparison and protects against a confident but wrong estimate from a wrongly calibrated model. It helps to decrease the rate of catastrophic redshift outliers by increasing our estimate of our estimation errors. 

While this makes sense from a Bayesian point-of-view, a Frequentist interpretation of this approach is that one mistake is countered with another: the procedure starts out with a wrong and perhaps too tightly defined model that produces a $p(z)$ distribution that is both too narrow and displaced, and the correction widens the $p(z)$ 'cosmetically' to make sure the correct solution is contained within the confidence interval and outliers are avoided. The problem is that the template error is an expression of model {\it possibility}, not an expression of model {\it frequency} and hence the resulting $p(z)$ will reflect redshift {\it possibility} but will not reflect redshift {\it frequency}.

Empirical methods need to deal with measurements errors in both the query sample and the empirical sample that is the foundation for the model. When training e.g. a neural network, the measurements errors do not need to be taken into account per-object as they are implicit in the distribution of the training sample. As long as the error distribution in the training sample and the query sample are similar, the network simply trains to give on average the best answer. In particular, it is not advisable to train the network with additional realisations of the training sample, created by adding noise according to the measurement uncertainties, since this will lead to over-smoothing and scatter redshift signals into neighbouring parts of feature space where they are not observed. 

In the case of KDE the model is not abstracted so that the Gaussian measurement noise of the model constituents is present and known on a per-object basis. The kernel function in KDE introduces a smoothing of the model, which is usually undesired, especially if model and data have similar measurement errors, because it will make the model $z(\vec{c})$ map smoother than the data $z(\vec{c})$ distribution, and hence broaden the $p(z)$ estimate for each object. 

However, the $p(z)$ results of KDE can be made to be exactly representative of the full empirical map, if the smoothing of the model $z(\vec{c})$ map is tuned to the uncertainties of the data of an individual query object {\it after} applying the kernel function. This means that if both model and data noise is Gaussian, the uncertainties of data and model should not be square-added but square-subtracted, which of course is impossible when model uncertainties are at least as large as those on data. However, if photometry for the empirical sample is available with smaller errors than in the query sample, then the kernel function can be chosen for each model-data pair of objects such that the model error convolved with the kernel function equates the data error and the resulting $p(z)$ will represent true redshift frequency within the statistical noise afforded by the empirical model \citep{W09}. 

This last approach was shown to produce perfect frequentist $p(z)$'s provided the empirical sample is complete and representative. In this paper, however, we can not take advantage of the zero-smoothing KDE, because the photometry of the empirical sample has the same error levels as the query samples.


\subsubsection{Application constraints}

Bayesian template methods have long been the leading strand in photo-z'eeing, since their application is not restricted in redshift or magnitude space. Especially, they have been the only valid method for exploring unknown territory at faint magnitudes or in the high-redshift Universe such as work in the Hubble Deep Field \citep{HDF,Lan96}. However, their application is restricted to observables that are predicted by the templates; currently, templates are well defined in the UV-optical-NIR regime, but there are uncertainties towards more extreme wavelengths, where the emission of stellar atmospheres is currently less well-known. Also, dust and AGN add free parameters and potentially dominate the emission at much shorter and much longer wavelengths. In this work, e.g., the template-code BPZ was not able to take advantage of the WISE observations since these bands were not covered by the templates coming with BPZ (see discussion below). Empirical methods, in contrast, extract the information content of observed features regardless of their nature.

Clearly, using templates to explore parts of parameter space that are empirically unconstrained is a risky extrapolation. But assumptions on templates can be informed by physics understanding, such as the age of the Universe at a given redshift or plausible limits for the dust content of dwarf galaxies. As long as the space of considered template parameters is not too restrictive, the Bayesian exploration remains valid. In the spirit of Bayesian work, the range of tested models needs to include all models that are known to be possible, although not all of them may turn out to be relevant upon further empirical exploration. This means a Bayesian probability distribution is always a {\it possibility distribution} that is hopefully at least as wide as the actual {\it frequency distribution}.

For empirical methods, in contrast, there is no valid extrapolation. When the $z(\vec{c})$ map is defined for a particular apparent magnitude range, it can not be applied to an apparent magnitude outside of this range, simply because apparent magnitude is related to distance and hence redshift. In this paper we establish a redshift sample to $r<19.5$, which enables us to make redshift estimates for galaxies at $r<19.5$. Fainter galaxies extend to higher redshifts than covered by our map, and we would have no chance to guess their redshifts from an empirical method that did not know about them. In addition, in this paper we use the $r$-band magnitude as one of the observed features, so that any fainter object would immediately appear as an obvious outlier. Dropping magnitudes from the feature set will not remove this limitation: we stress, that extrapolating an empirical photo-z method to fainter redshifts is akin to fitting a spline to a set of points and assuming its parameters apply beyond the fitting range!

Empirical methods appear more reliable for predicting actual redshift {\it frequency distributions} in magnitude ranges where complete empirical samples can be constructed. However, they do not protect against outliers due to their limited size; rare types of objects may be entirely missing from small empirical data sets due to their discrete-object nature, and whatever is missing cannot be identified in the query sample either. In the common situation where the query sample is much larger than the empirical sample, the Poisson noise on the result is often driven by the smaller empirical sample, depending on the nature of the science application. Hence, empirical samples potentially inhibit new discoveries by suggesting too much certainty. The {\it estimated} frequency distribution from empirical methods is thus usually narrower than the {\it true} frequency distribution \footnote{However, some empirical methods such as KDE inherently smooth the empirical data such that the estimated distribution is widened again.}.


\subsection{Profound information limits}

It is important to note a fundamental floor to the precision with which the redshift of a galaxy can be estimated from its SED, irrespective of method. The intrinsic variety in galaxy colours at a fixed redshift implies that for an observed galaxy colour there is intrinsic variety or ambiguity in redshift. While this effect is obvious when the observable is a single colour index, it still dominates the redshift errors of bright well-measured galaxies in presently common multi-band data sets. A few optical broad passbands don't break all possible degeneracies in the space of SEDs spanned by the entire observable galaxy population. 

A set of observed magnitudes for a galaxy is at least influenced by the following parameters: (1) luminosity, (2) redshift, (3) mean stellar age, (4) specific star-formation rate, (5) internal dust column density, and (6) inclination of the galaxy (changing the mixing of dust optical depth for different sub-populations). In reality, star formation histories are more complicated than this simplified 2D-picture and also the observed passbands are not fully independent; this affects broad-band colours enough to introduce intrinsic scatter to the $z(\vec{c})$ map. \footnote{\citet{Con95ortho} have shown that they can map a sequence of ten galaxy SED templates onto just two eigenspectra with little residual power, but today we know the galaxy population is not fully described by this 1-parameter family of ten supporting templates, see e.g. \citet{WGM05}.}

The uncertainty of a per-object redshift estimate is primarily a function of the photometric errors as soon as a redshift estimate is colour-dependent. From a linear approximation to a local gradient in the $z(\vec{c})$ map, we can derive $\sigma(z) \propto \sigma(\vec{c}) \propto N/S$ and hence $\log \sigma(z) \propto {\rm mag}$, which can be verified in simulations of photo-z surveys \citep{WMR01}. However, globally the $z(\vec{c})$ map is anything but linear and indeed bounded apart from band dropouts, so that the simple analytic relation will break down for large photometric errors. Thus, redshift estimates of marginally detected objects rely mostly on filter dropouts to derive rough estimates, whose precision depends on the spacing of the filter set. Redshift uncertainties towards the brighter end are a combination of the photometric errors and intrinsic redshift variance in the map, see e.g. \citet{W09} for equations and an experiment with QSOs from SDSS.

At bright magnitudes calibration uncertainties may be large relative to photometric errors, and will thus drive the error budget. For template methods this concerns the calibration of the model templates relative to the observations. The issue is equally important for empirical methods, where the sky area containing the training sample needs to be on the same calibration as the sky area with the query sample. Otherwise, colour offsets between query data and the $z(\vec{c})$ map will lead to systematic redshift biases.


\subsubsection{Calibration requirements}

Given that redshift depends on colour, it is obvious that calibration offsets between model and data lead to biases in the redshift estimates. Such offsets can stem from zeropoint shifts, from inconsistent ways of obtaining the data, or from model priors that don't represent the query sample.

Bayesian methods use model templates from stellar population synthesis or from real SEDs measured in physically consistent apertures. Hence, they need to ensure that photometric zeropoints for the bandpasses are correct and matched to the physical zeropoints of the template models \citep{Baum57}. It was often found with template methods, that the data seemed to be calibrated to slightly different standards than the templates, which motivated self-calibrating template-repair methods \citep[e.g.][]{Csa03,B04,I06,Coe06,Cap07}.

Also, the multi-band data of query objects needs to sample the same physical footprint in each observed band even if PSFs change with bandpass \citep{LS86,Dr15}. Usually, bandpass matching involves a convolution and/or aperture variation that homogenises the PSF across images, and for some choices of aperture profile this can be achieved without a convolution \citep{RM91,W04}. 

Empirical methods do not have such strict requirements, because mean differences between bandpasses are accounted for, whether they stem from PSF changes or zeropoint changes. Hence, empirical methods are often considered self-calibrating and robust, although self-calibration only works if the trends are the same for the empirical and the query sample.

The correct calibration of priors is implicit in empirical methods provided the empirical sample is complete, which might require the use of weights. No weighting, however, can repair the damage of object varieties missing entirely from the empirical sample. Template sets need to be complete as well; their priors are notoriously difficult to calibrate a-priori as the required information usually is to be discovered from the data after estimating redshifts to start with. Certainly, the extrapolation of priors to fainter magnitudes or higher redshifts is risky, but possible based on plausible assumptions for galaxy evolution.

Since calibration itself rests on measurements, calibration errors in zero-points are itself unavoidable. When these calibration errors are unique on small patches of sky and scatter across larger areas of a survey, they introduce small local redshift biases corresponding to the calibration pattern. They may average out across an entire survey area and may not contribute much to the global redshift uncertainty. This will affect any photo-z method that does not constrain its model by location on the sky. The best insurance against the propagation of zero-point errors into redshift errors is the use of many independent features, because each feature will tend to move the redshift differently, and the large number of features will make the combined redshift constraint stronger \footnote{An extreme case is high-resolution spectroscopy: each spectral bin counts as a feature and can tolerate high flux calibration errors without affecting the precision of the redshift measurement from the spectral signature.}.


\subsubsection{Data gaps}

Sometimes not all features are observed for all objects in a sample. While an obvious solution is to redefine a complete sample without gaps in the data, we often want to work with the data in hand. Bayesian methods can drop a constraint where it is not available. Empirical methods without network structure consider every feature independent of the others (such as KDE) and can marginalise over unmeasured features as well. But methods with a network structure require a value on every input node as otherwise the output is undefined. They could then not work with any training or query objects that have data missing, although upper limits can be included by representing them with a meaningful figure.


\subsubsection{Model resolution and smoothing}

Empirical methods define a continuous map from a sample of discrete points, so that the map can be evaluated at any location at which query points may appear. The price of continuity is smoothing: at any location the map is constructed using the empirical support from a neighbourhood. In contrast, templates can be evaluated at any point in parameter space without referring to a neighbourhood. Their resolution is only limited if filter-specific measurements are pre-calculated to a chosen grid, and those grids are only limited in size by practical considerations.

The power of empirical methods to resolve structure in the map thus depends on dense sampling. A sparse empirical sample can only ever allow coarse estimates. KDE will work particularly badly with sparse data as it does not fit an abstract model to the empirical data, while neural nets and other smooth-model fitters do. The difference is that fitted smooth models have continuously interpolated solutions in the gaps between the discrete empirical points, while KDE considers a parameter range to be split into bins defined by each single object in the empirical sample. The interpolation may not follow a true but unknown structure in the interpolation space, but it will do better than KDE, as long as it is prevented from overfitting and restricted to discovering a smooth model with as few parameters as are justified from the data. E.g. a neural net trained with two objects at $(c=0,z=0)$ and $(c=2,z=2)$ may be presented with a query object at $c=1$ and respond with an estimate of $z=1$, which would be correct if $z(c)$ happens to be linear. KDE, however, would respond with a 50:50 probability that the query object is either from the $z=0$-bin or the $z=2$-bin.

KDE only plays out its strength when the empirical sample is very large. The advantage of KDE then is that its estimates are based strictly on empirical knowledge and no attempt is made to involve smoothness or other constraints that involve unquantified deviations of the $z(\vec{c})$ model from the underlying reality. \citet{W09} have also shown that in KDE it is possible to have effectively zero neighbourhood smoothing despite a non-zero-width kernel function (see discussion below).


\subsubsection{Completeness and outliers}

Photometric redshifts have a reputation for producing undesirable outliers, i.e. catastrophically wrong redshift estimates. There are three main origins of outliers: (1) Genuine local redshift ambiguities in colour space, (2) model deficiencies such as inappropriate templates or priors, and (3) issues with the data such as object blending or incorrect photometric measurements. We also need to distinguish between the presence of outliers and their appearance, for which a further reason exists: (4) mistakes in spectroscopic redshifts used for quality validation. Any large validation sample is prone to include some redshift mistakes, usually at the rate of $\sim 1$\% for high-quality spectra, but larger for noisier data. Wrong spectroscopic redshifts will appear as a floor of outliers in any photo-z validation, no matter which photo-z method had been used.

Genuine ambiguities (1) correspond to bimodal redshift distributions, where two possible answers are separated by more than their uncertainty. They are a problem for standard neural nets, which will always train to deliver the majority redshift answer for a given observed input. The minor mode in an ambiguity will necessarily appear as outliers. KDE naturally estimates $p(z)$ distributions from the model, so its power to flag ambiguities rests on their visibility in the empirical sample. The richer an empirical sample, the more ambiguities it can reveal. But in any sample there might always remain invisible weak ambiguities with major/minor frequency ratios so extreme that discretisation noise and the number of objects in the ambiguous region of feature space leads to the minor mode not showing a single object (\citet{W09} discuss quantitatively the residual estimation risks due to undetected ambiguities). A typical situation of interest involves a query sample that is much larger than the empirical sample, and weak ambiguities in the query sample, while being present, may be undetectable given the lower-$n$ empirical sample. Bayesian template fitting can easily reveal ambiguities, although estimating the relative probability of the two modes is a challenge in the absence of an empirical model.

Model deficiencies (2) for templates mostly means lack of appropriate templates, so that the right redshift solution cannot be discovered in the Bayesian test. However, empirical models may have similar deficiencies if the samples they are based upon are incomplete. A first incompleteness issue appears when the sampling of the spectroscopic data set is non-random, but prefers certain parts of colour-magnitude space over others. This is a minor problem that can be compensated with a weighting scheme as demonstrated by \citet{Lima08}. A major problem, however, is that selective spectroscopic incompleteness may result from galaxies being intrinsically different. There is an unconstrained risk that galaxies without known redshifts have a redshift distribution that is different from those with similar colours but a successful redshift determination. Ambiguities, where one solution was much harder to identify spectroscopically than the other, would manifest themselves in just this way.

Overall, empirical model deficiencies may be considered a minor issue when the completeness of an empirical sample is above 90\%, but if we used samples that were, e.g., very deep but only 50\% complete, the results would not be very trustworthy. KDE allows to explicitly include those parts of an empirical sample that lack spectroscopic redshift labels as a separate object type. KDE can thus produce per-object an estimate that with X\% probability it is drawn from a specified $p(z)$, while with $100-$X\% probability it is at an unknown redshift.

Another model deficiency is incorrect priors, which in template models are derived from our understanding of galaxy evolution; they will continue to be an issue as long as the evolution of the galaxy population has not been mapped sufficiently well. In empirical models, wrong priors means either that the empirical sample is not a complete sample in a statistical random sense, or that the empirical sample is determined from a volume that is not representative of the query volume.

Photometric blends of two objects (3) are a problem, because one photometry is associated with two redshifts, while the photometry data would never be observed for a single object. Usually, template methods do not include blends in their models, although with the use of appropriate priors and assumptions on the spatial resolution of the photometry this would be possible. In empirical models, these will ideally show up with the same frequency as in query samples, but their redshift label will depend on their spectroscopic identification. Noisy spectra might preclude a clear identification as a blend, in which case they can be a problem for the model definition. At present we don't have enough constraints to address blends further.

Wrong spectroscopic redshifts (4) are a problem for photo-z validation, but they are less of a problem for defining a model: when training neural nets, they will be ignored as minor modes while the net trains to deliver the majority answer; KDE methods will report a small but measurable probability for the wrong redshift. However, if redshift mistakes cluster in colour space, they will have larger local impact and can even influence a network response. 


\subsubsection{Execution speed}

Photo-z analyses have usually two components, whose computational speed have different behaviour: these are (1) preparing the model, and (2) processing the query data. For template fitting, a colour library may first be constructed from a template library using the filter curves of the observing system. The processing time will depend on the resolution and dimension of the chosen grid. If no colour grid model is pre-calculated the convolution of templates with filter curves will need to be done on the fly. Many empirical methods abstract a model from an observed sample, e.g. by training a neural network. Empirical KDE has no requirements on model preparation as the model is both directly observed and does not need to be abstracted. 

When running a query sets against a model, the speed, at which a single query object is processed, depends on the method. Many machine-learning methods are information-compression methods that abstract a smooth map from a rich and detailed empirical data set. Since the abstract map is captured in a small number of parameters, they generally deliver high speed in the processing of a query sample. Their run time per query object is independent of the number of training objects, but depends on the number of model parameters such as network links. These methods are always to be preferred when query speed is essential.

In contrast, both empirical KDE and Bayesian template-fitting are slow in running over a query sample, since they evaluate every model object in every data-model comparison individually. Their runtime per query object scales with the size of the empirical sample or the number of points in the template $\times$ redshift grid.

Traditionally, these considerations have been a concern in large surveys. but on modern hardware execution speed has ceased to be a concern for photo-z calculations. Even a data set with 100 million objects will execute on single-core hardware in less than a few days with any of these methods.


\subsubsection{Random text}

 \citet{WMR01} distinguish between three domains in photo-z quality:
\begin{enumerate}
 \item \textit{Saturation regime:} for bright objects the photometric error is so small that it has little influence on the redshift error; the error is instead dominated by the intrinsic redshift distribution among objects of fixed colour and by calibration offsets between model and data.
 \item \textit{Transition regime:} for intermediate objects the photometric error gets larger with increasing magnitude and drives the redshift error with an approximately linear dependence as the colour-redshift mapping is locally approximately linear.
 \item \textit{Breakdown regime:} for faint objects the size of the photometric error ellipsoid approaches the bounded extent of the galaxy distribution in colour space, which means that all redshifts appear possible, irrespective of the degree of error.
\end{enumerate}

Since magnitude has little effect on the photometric errors ellipsoids in our analysis, any magnitude-dependence of redshift errors must be driven by other factors. Two other possible factors are:
\begin{itemize}
 \item Calibration offsets between model and data.
 \item The width of the redshift prior and the true redshift distribution is a strong function of magnitude.
\end{itemize}

%In our comparison KDE leads the precision of photo-z estimation in terms of redshift bias and scatter, although its computer runtime is probably by far the slowest. 

%This is not surprising since there is a variant of KDE, which has been shown by \citet{W09} to produce an exact frequency correspondence between query and model data, with the only difference being the propagation of Poisson noise arising from finite sample sizes. This variant of KDE is an exact implementation of Bayesian statistics with an empirical training sample, where the kernel function is chosen properly to take into account the feature errors in query and model objects leading to a "zero-neighbourhood-smoothing KDE" method. While the zero-smoothing approach requires that model errors are smaller than query errors, this is not the case in our study, and hence we use only a standard KDE method.

%The Boosted Decision Tree (BDT) method performs nearly exactly as well, with two exceptions: (i) its demands more richly populated bins, and hence did not work for redshifts above $z=0.4$; and (ii) it shows a mild redshift bias at lowest, but especially at higher redshifts towards $z=0.4$. 

%The Probabilistic Neural Net (PNN) method performs slightly less accurately and is more biased at low and high redshifts. This is a modest disadvantage of NNs in the presence of rich training samples, where the smoothness of the map enforced by neural nets does not allow the full feature-space resolution of the training sample to express itself in the estimation process. However, the positive aspect of the smoothness criterion is that traditional regression NNs can outperform other methods when presented with sparse training samples, but this advantage is not expected to help with classification NNs, i.e. PNNs, where each redshift bin needs to be well-populated.

%The biases present in the BDT and PNN method, but absent in the KDE method, are expected when redshift is estimated via a classification approach working in fine redshift bins. The redshift bins at lowest and highest redshift are the widest as the sample is sparsest there, and the need for training with such classes means that again the full redshift resolution of the training set is not exploited. 



\texttt{What else needs to be mentioned and go where?
PHAT 2011 paper? Jeff Newman's et al. white paper? Budavari+ 2000 method to discover templates fro photometry?
PNN, or multiple Random Forests, then determine scatter
Sample splits for the above. Clarify terminology of training, validation and testing sample?}

\fi


\begin{thebibliography}{}

\bibitem[\protect\citeauthoryear{Ahn et al.}{2014}]{Ahn14}
 Ahn, C. P., et al. 2014, ApJS, 211, 17 
\bibitem[\protect\citeauthoryear{Asorey et al.}{2016}]{Asorey16}
 Asorey, J., Carrasco Kind, M., Sevilla-Noarbe, I., Brunner, R. J. \& Thaler, J., 2016, MNRAS, 459, 1293 
%\bibitem[\protect\citeauthoryear{Bacon et al.}{2005}]{Bacon05} 
% Bacon, D. J., Taylor, A. N., Brown, M. L., Gray, M. E., Wolf, C., et al. 2005, MNRAS, 363, 723 
\bibitem[\protect\citeauthoryear{Baldry et al.}{2004}]{Bal04}
 Baldry, I. K., Glazebrook, K., Brinkmann, J., Ivezi\'{c}, \v{Z}., Lupton, R. H., Nichol, R. C. \& Szalay, A. S., 2004, ApJ, 600, 681
\bibitem[\protect\citeauthoryear{Baldry et al.}{2014}]{Bal14}
 Baldry, I. K., et al., 2014, MNRAS, 441, 244
%\bibitem[\protect\citeauthoryear{Ball et al.}{2008}]{Ball08}
% Ball, N. M.,, Brunner, R. J., Myers, A. D., Strand, N. E., Alberts, S. L. \& Tcheng, D., 2008, ApJ, 683, 12
\bibitem[\protect\citeauthoryear{Baum}{1957}]{Baum57}
 Baum, W. A., 1957, AJ, 62, 6
%\bibitem[\protect\citeauthoryear{Baum}{1962}]{Baum62}
% Baum, W. A., 1962, in {\it Problems of Extragalactic Research},
% ed. McVittie, G. C., Macmilliam, IAU Symposium 15, 390 
\bibitem[\protect\citeauthoryear{Beck et al.}{2016}]{Beck16} 
 Beck, R., Dobos, L., Budavari, T., Szalay, A. S. \& Csabai, I., 2016, MNRAS, 460, 1371 
\bibitem[\protect\citeauthoryear{Bell et al.}{2004}]{Bell04} 
 Bell, E. F., Wolf, C., et al. 2004, ApJ, 608, 752 
\bibitem[\protect\citeauthoryear{Bender et al.}{2001}]{Ben01}
 Bender, R., et al., 2001, p. 96 in \textit{Deep Fields}, ESO Astrophysics Symposia, ed. by Cristiani, S., Renzini, A. \& Williams, R. E., Springer
\bibitem[\protect\citeauthoryear{Benitez}{2000}]{B00}
 Benitez, N., 2000, ApJ, 536, 571
\bibitem[\protect\citeauthoryear{Benitez}{2009}]{B09}
 Benitez, N., Moles, M., et al., 2009, ApJ, 692, L5
\bibitem[\protect\citeauthoryear{Benitez}{2011}]{B11}
 Benitez, N., 2011, Astrophysics Source Code Library, record ascl:1108.011
\bibitem[\protect\citeauthoryear{Benitez et al.}{2014}]{JPAS}
 Benitez, N., Dupke, R., Moles, M., Sodre, L.,et al., 2014, J-PAS "Red Book", arXiv:1403.5237
\bibitem[\protect\citeauthoryear{Bertin \& Arnouts}{1996}]{BA96}
 Bertin, E., Arnouts, S., 1996, A\&AS, 117, 393
\bibitem[\protect\citeauthoryear{Bilicki et al.}{2014}]{Bil14}
 Bilicki, M., Jarrett, T. H., Peacock, J. A., Cluver, M. E., \& Steward, L., 2014, ApJS, 210, 9
\bibitem[\protect\citeauthoryear{Bilicki et al.}{2016}]{Bil16}
 Bilicki, M., Peacock, J. A., Jarrett, T. H., et al., 2016, ApJS, 225, 5
\bibitem[\protect\citeauthoryear{Blake \& Bridle}{2005}]{BB05}
 Blake, C. \& Bridle, S., 2005, MNRAS, 363, 1329
\bibitem[\protect\citeauthoryear{Blake et al.}{2010}]{Blake10}
 Blake, C., Brough, S., Colless, M., Couch, W., Croom, S., et al., 2010, MNRAS, 406, 803
\bibitem[\protect\citeauthoryear{Blake et al.}{2016}]{Blake16}
 Blake, C., et al., 2016, MNRAS, submitted
\bibitem[\protect\citeauthoryear{Bolzonella et al.}{2000}]{HyperZ} 
 Bolzonella, M., Miralles, J.-M. \& Pell\'{o}, R., 2000, A\&A 363, 476
\bibitem[\protect\citeauthoryear{Bonnett}{2015}]{Bon15}
 Bonnett, C., 2015, MNRAS, 449, 1043
%\bibitem[\protect\citeauthoryear{Brown et al.}{2003}]{Brown03} 
% Brown, M. L., Taylor, A. N., Bacon, D. J., Gray, M. E., Dye, S., Meisenheimer, K., Wolf, C., et al. 2003, MNRAS, 341, 100
\bibitem[\protect\citeauthoryear{Brammer et al.}{2008}]{EAZY}
 Brammer, G. B., van Dokkum, P G., Coppi, P., 2008, ApJ, 686, 1503
\bibitem[\protect\citeauthoryear{Bruzual \& Charlot}{1993}]{BC93} 
 Bruzual, A. G., \& Charlot, S. 1993, ApJ, 405, 538
\bibitem[\protect\citeauthoryear{Budavari et al.}{2000}]{Bud00}
 Budavari, T., Szalay, A. S., Connolly, A. J., Csabai, I. \& Dickinson, M., 2001, AJ, 120, 1588 
\bibitem[\protect\citeauthoryear{Budavari et al.}{2001}]{Bud01}
 Budavari, T., Csabai, I., Szalay, A. S., Connolly, A. J., et al., 2001, AJ, 122, 1163 
%\bibitem[\protect\citeauthoryear{Budavari}{2009}]{Bud09}
% Budavari, T., 2009, ApJ, 695, 747
\bibitem[\protect\citeauthoryear{Bundy et al.}{2015}]{Bundy15}
 Bundy, K., Leauthaud, A., Saito, S., Bolton, A., Lin, Y.-T., Maraston, C., Nichol, R. C., Schneider, D. P., Thomas, D. \& Wake, D. A., 2015, ApJSS, 221, 15
\bibitem[\protect\citeauthoryear{Capak}{2004}]{Cap04}
 Capak, P., PhD Thesis, 2004
\bibitem[\protect\citeauthoryear{Capak et al.}{2007}]{Cap07}
 Capak, P., Aussel, H., Ajiki, M., et al. 2007, ApJS, 172, 99
\bibitem[\protect\citeauthoryear{Carrasco Kind \& Brunner}{2013}]{CK13}
 Carrasco Kind, M. \& Brunner, R. J., 2013, MNRAS, 432, 1483
\bibitem[\protect\citeauthoryear{Coe et al.}{2006}]{Coe06}
 Coe, D., Benitez, N., Sanchez, S. F., et al. 2006, AJ, 132, 926
\bibitem[\protect\citeauthoryear{Coleman et al.}{1980}]{CWW80}
 Coleman, G. D., Wu, C.-C., \& Weedman, D. W. 1980, ApJS, 43, 393
\bibitem[\protect\citeauthoryear{Colless et al.}{2001}]{GRS}
 Colless, M., Dalton G., Maddox, S., Sutherland, W., Norberg, P. et al., 2001, MNRAS, 328, 1039
\bibitem[\protect\citeauthoryear{Collister \& Lahav}{2004}]{CL04}
 Collister, A. A. \& Lahav, O., 2004, PASP, 116, 345
%\bibitem[\protect\citeauthoryear{Connolly et al.}{1995a}]{Con95}
% Connolly, A. J., Csabai, I., Szalay, A. S., Koo, D. C., Kron, R. G. \& Munn, J. A., 1995, AJ, 110, 2655
\bibitem[\protect\citeauthoryear{Connolly et al.}{1995b}]{Con95ortho}
 Connolly, A. J., Szalay, A. S.,  Bershady, M. A., Kinney, A. L., Calzetti, D., 1995, AJ, 110, 1071
\bibitem[\protect\citeauthoryear{Csabai et al.}{2000}]{Csa00}
 Csabai, I., Connolly, A. J., Szalay, A. S. \& Budavari, T., 2000, AJ, 119, 69
\bibitem[\protect\citeauthoryear{Csabai et al.}{2003}]{Csa03}
 Csabai, I., Budavari, T., Connolly, A. J., Szalay, A. S., et al., 2003, ApJ, 125, 580
\bibitem[\protect\citeauthoryear{Csabai et al.}{2007}]{Csa07}
 Csabai, I., Dobos, L., Trencseni, M., Herczegh, G., Josza, P., Purger, N., Budavari \& T., Szalay, A. S., 2007, AN, 328, 852
%\bibitem[\protect\citeauthoryear{de Jong et al.}{2015}]{deJong15} 
% de Jong, J. T. A., Verdoes Kleijn, G. A., Boxhoorn, D. R.,  et al., 2015, arXiv:1507.00742  
\bibitem[\protect\citeauthoryear{de Jong et al.}{2013}]{deJong13} 
 de Jong, J. T. A. , Kuijken, K., Applegate, D., Begeman, K., Belikov, A., et al., 2013, ESO Messenger, 154, 44
\bibitem[\protect\citeauthoryear{Drinkwater et al.}{2010}]{Dr10} 
 Drinkwater, M. J., Jurek, R. J., Blake, C., Woods, D., et al., 2010, MNRAS, 401, 1429 
\bibitem[\protect\citeauthoryear{Driver et al.}{2011}]{GAMA} 
 Driver, S. P., Hill, D. T., Kelvin, L. S., Robotham, A. S. G., et al., 2011, MNRAS, 413, 971 
\bibitem[\protect\citeauthoryear{Driver et al.}{2015}]{Dr15} 
 Driver, S. P., Wright, A. H., Andrews, S. K., Davies, L. J., et al., 2015, MNRAS, arXiv:1508.02076
\bibitem[\protect\citeauthoryear{Driver et al.}{2016}]{WAVES} 
 Driver, S. P., Davies, L. J., Meyer, M., Power, C., Robotham, A. S. G., Baldry, I. K., Liske, J. \& Norberg, P., 2016, ASSP, 42, 205
%\bibitem[\protect\citeauthoryear{Durret et al.}{2015}]{Durret15} 
% Durret, F., Adami, C., Bertin, E., Hao, J., et al., 2015, A\&A, 578, A79
\bibitem[\protect\citeauthoryear{Edmondson et al.}{2006}]{Edd06} 
 Edmondson, E. M., Miller, L. \& Wolf, C., 2006, MNRAS, 371, 1693
\bibitem[\protect\citeauthoryear{Fioc \& Rocca-Volmerange}{1997}]{FRV97}
 Fioc, M., \& Rocca-Volmerange, B., 1997, A\&A, 326, 950
\bibitem[\protect\citeauthoryear{Firth et al.}{2003}]{FLS03}
 Firth, A. E., Lahav, O. \& Somerville, R. S., 2003, MNRAS, 339, 1195
\bibitem[\protect\citeauthoryear{Gerdes et al.}{2010}]{Gerdes10} 
 Gerdes, D. W., Sypniewski, A. J., McKay, T. A., Hao, J., Weis, M. R., Wechsler, R. H. \& Busha, M. T., 2010, ApJ, 715, 823
\bibitem[\protect\citeauthoryear{Giavalisco et al.}{2004}]{Giav04} 
 Giavalisco, M., Ferguson, H. C., Koekemoer, A. M., et al. 2004, ApJ, 600, L93
%\bibitem[\protect\citeauthoryear{Gladders \& Yee}{2005}]{GY05}
% Gladders, M. D. \& Yee, H. K. C., 2005, ApJS, 157, 1
\bibitem[\protect\citeauthoryear{Gonzalez \& DEEP3 Team}{priv. comm.}]{DeepPC}
 Gonzalez, V. \& DEEP2 Team, private communication 
\bibitem[\protect\citeauthoryear{Graff et al.}{2014}]{2014MNRAS.441.1741G}
 Graff, P., Feroz, F., Hobson, M. P. \& Lasenby, A., MNRAS, 2014, 441, 1741
\bibitem[\protect\citeauthoryear{Gray et al.}{2002}]{Gray02}
 Gray, M. E., Taylor, A. N., Meisenheimer, K., Dye, S., Wolf, C. \& Thommes, E., 2002, ApJ, 568, 141
\bibitem[\protect\citeauthoryear{Hastie et al.}{2008}]{Hastie08}
 Hastie, T., Tibshirani, R. \& Friedman, J., in \textit{The elements of statistical learning: data mining, inference and prediction}, Springer, 2008
\bibitem[\protect\citeauthoryear{Hickson et al.}{1994}]{H94} 
 Hickson, P., Gibson, B. K. \& Callaghan, K. 1994, MNRAS, 267, 911
\bibitem[\protect\citeauthoryear{Hildebrandt et al.}{2010}]{PHAT}
 Hildebrandt, H., Arnouts, S., Capak, P., Moustakas, L. A., Wolf, C., et al. 2010, A\&A, 523, A31
\bibitem[\protect\citeauthoryear{Hildebrandt et al.}{2012}]{H12}
 Hildebrandt, H., et al. 2012, MNRAS, 421, 2355
\bibitem[\protect\citeauthoryear{Hildebrandt et al.}{2016}]{H16}
 Hildebrandt, H., Viola, M., Heymans, C., Joudaki, S., Kuijken, K., et al. 2016, MNRAS, in press, arXiv:1606.05338
\bibitem[\protect\citeauthoryear{Hinton et al.}{2016}]{MARZ}
 Hinton, S. R., Davis, T. M., Lidman, C., Glazebrook, K. \& Lewis, G. F., 2016, Astronomy \& Computing, 15, 61
\bibitem[\protect\citeauthoryear{Hoecker et al.}{2007}]{Hoecker07}
 Hoecker et al. 2007								%%%%%%%%%%%%%%%%%%%%%
\bibitem[\protect\citeauthoryear{Ilbert et al.}{2006}]{I06}
 Ilbert, O., Arnouts, S., McCracken, H. J., et al. 2006, A\&A, 457, 841
\bibitem[\protect\citeauthoryear{Ilbert et al.}{2009}]{I09} 
 Ilbert, O., Capak, P., Salvato, M., et al. 2009, ApJ, 690, 1236
\bibitem[\protect\citeauthoryear{Jarrett et al.}{2011}]{J11} 
 Jarrett, T. H., Cohen, M., Masci, F., Wright, E., Stern, D. et al. 2011, ApJ, 735, 112
\bibitem[\protect\citeauthoryear{Johnson et al.}{2016}]{J16}
 Johnson, A. S., Blake, C., et al.  2016, MNRAS, in preparation
\bibitem[\protect\citeauthoryear{Johnston et al.}{2007}]{Jo07}
 Johnston, S., et al., 2007, PASA, 24, 174 
\bibitem[\protect\citeauthoryear{Johnston et al.}{2008}]{Jo08}
 Johnston, S., Taylor, R., Bailes, M, et al., 2008, Exp. Astron., 22, 151 (doi:10.1007/s10686-008-9124-7)
\bibitem[\protect\citeauthoryear{Jones et al.}{2009}]{6dF}
 Jones, D. H., Read, M. A., Saunders, W., Colless, M., et al.  2009, MNRAS, 399, 683
\bibitem[\protect\citeauthoryear{Keller et al.}{2007}]{K07}
 Keller, S. C., Schmidt, B. P.,  Bessell, M. S., Conroy, P. G., Francis, P., et al., 2007, PASA, 24, 1
 % Granlund, A., Kowald, E., Oates, A. P., Martin-Jones, T., Preston, T., Tisserand, P., Vaccarella, A. \& Waterson, M. F.,
\bibitem[\protect\citeauthoryear{Kilbinger}{2015}]{Kilb15}
 Kilbinger, M., 2015, Reports on Progress in Physics, 78, 8, article id. 086901
\bibitem[\protect\citeauthoryear{Kinney et al.}{1996}]{Kin96}
 Kinney, A. L., Calzetti, D., Bohlin, R. C., et al. 1996, ApJ, 467, 38
\bibitem[\protect\citeauthoryear{Kitching et al.}{2007}]{Kit07}
 Kitching, T. D., Heavens, A. F., Taylor, A. N., Brown, M. L., Meisenheimer, K., Wolf, C., Gray, M. E. \& Bacon, D. J., 2007, MNRAS, 376, 771
%\bibitem[\protect\citeauthoryear{Kitching et al.}{2008}]{Kit08}
% Kitching, T. D., Taylor, A. N. \& Heavens, A. F., 2008, MNRAS, 389, 173
%\bibitem[\protect\citeauthoryear{Koo}{1985}]{Koo85}
% Koo, D. C., 1985, AJ, 90, 418 
\bibitem[\protect\citeauthoryear{Kurcz et al.}{2016}]{Kurcz16}
 Kurcz, A., Bilicki, M., Solarz, A., Krupa, M., Pollo, A. \& Ma\l{}ek, K., 2016, A\&A, arXiv:1604.04229  
\bibitem[\protect\citeauthoryear{Kuijken}{2008}]{K08}
 Kuijken, K. H. 2008, A\&A, 482, 1053
\bibitem[\protect\citeauthoryear{Kuijken et al.}{2015}]{Kui15}
 Kuijken, K., Heymans., C., Hildebrandt, H., Nakajima, R., et al., 2015, MNRAS, 454, 3500
\bibitem[\protect\citeauthoryear{Lanzetta et al.}{1996}]{Lan96}
 Lanzetta, K. M., Webb, J. K. \& Fern\'{a}ndez-Soto, A., 1996, Nature, 381, 759 
%\bibitem[\protect\citeauthoryear{Leauthaud et al.}{2012}]{Leaut12}
% Leauthaud, A., Tinker, J., Bundy, K., Behroozi, P. S., Massey, R. Rhodes, J. et al., 2012, ApJ, 744, 159 
\bibitem[\protect\citeauthoryear{Le F\`{e}vre et al.}{2013}]{VVDS} 
 Le F\`{e}vre, O., Cassata, P., Cucciati, O., et al. 2013, A\&A, 559, A14
\bibitem[\protect\citeauthoryear{Lilly et al.}{1995}]{Lilly95}
 Lilly, S. J., Le Fevre, O., Crampton, D., Hammer, F. \& Tresse, L., 1995, ApJ, 455, 50
\bibitem[\protect\citeauthoryear{Lima et al.}{2008}]{Lima08}		% weights approach
 Lima, M., Cunha, C. E., Oyaizu, H., Frieman, J., Lin, H. \& Sheldon, E. S., 2008, MNRAS, 390, 118
\bibitem[\protect\citeauthoryear{Loh \& Spillar}{1986}]{LS86}
 Loh, E. D. \& Spillar, E. J. 1986, ApJ, 303, 154
\bibitem[\protect\citeauthoryear{MacKay}{2003}]{MacKay}
 MacKay, D. J. C., in \textit{Information Theory, Inference, and Learning Algorithms}, Cambridge University Press, 2003
\bibitem[\protect\citeauthoryear{Mandelbaum et al.}{2008}]{Man08}
 Mandelbaum, R., Seljak, U., Hirata, C. M., et al., 2008, MNRAS, 386, 781
\bibitem[\protect\citeauthoryear{Mart\'{i} et al.}{2014}]{PAU}
 Mart\'{i}, P., Miquel, R., Castander, F. J., Gazta\~{n}aga, E., Eriksen, M.  \& S\'{a}nchez, C., 2014, MNRAS, 442, 92 
\bibitem[\protect\citeauthoryear{Masters et al.}{2015}]{Mas15}
 Masters, D., Capak, P., Stern, D., Ilbert, O., Salvato, M., Schmidt, S., et al., 2015, ApJ, 813, 53
%\bibitem[\protect\citeauthoryear{Muzzin et al.}{2013}]{Muzz13}
% Muzzin, A., Wilson, G., Demarco, R., Lidman, C., Nantais, J., Hoekstra, H., Yee, H. K. C. \& Rettura, A., 2013, ApJ, 767, 39
%\bibitem[\protect\citeauthoryear{Nanayakkara et al.}{2016}]{ZFIRE}		
% Nanayakkara, T., Glazebrook, K., Kacprzak, G. G., Yuan, T., et al., 2016, ApJ, 828, 21
\bibitem[\protect\citeauthoryear{Newman}{2008}]{New08}		% STILL NEED TO CITE SOMEWHERE!
 Newman, J. A., 2008, ApJ, 684, 88
\bibitem[\protect\citeauthoryear{Newman et al.}{2013}]{New13}
 Newman, J. A., et al., 2013, arXiv:1309.5384
\bibitem[\protect\citeauthoryear{Niemack et al.}{2009}]{Nie09}		
 Niemack, M. D., Jimenez, R., Verde, L., Menanteau, F., Panter, B. \& Spergel, D., 2009, ApJ, 690, 89
\bibitem[\protect\citeauthoryear{Norris et al.}{2011}]{Norris11}
 Norris, R. P., Hopkins, A. M., et al., 2011, PASA, 28, 215
\bibitem[\protect\citeauthoryear{Oyaizu et al.}{2008}]{Oya08a}
 Oyaizu, H., Lima, M., Cunha, C. E., Lin, H., Frieman, J. \& Sheldon, E. S., 2008, ApJ, 674, 768
%\bibitem[\protect\citeauthoryear{Oyaizu et al.}{2008b}]{Oya08b}
% Oyaizu, H., Lima, M., Cunha, C. E., Lin, H. \& Frieman, J., 2008b, ApJ, 689, 709
\bibitem[\protect\citeauthoryear{Prakash et al.}{2016}]{P16}
 Prakash, A., et al., 2016, ApJS, 224, 34
\bibitem[\protect\citeauthoryear{Raichoor et al.}{2014}]{R14}
 Raichoor, A., Mei, S., Erben, T., Hildebrandt, H., Huertas-Company, M., Ilbert, O., Licitra, R., et al. 2014, ApJ, 797, 102
%\bibitem[\protect\citeauthoryear{Rettura et al.}{2014}]{Rett14}
% Rettura, A., Martinez-Manso, J., Stern, D., Mei, S. et al., 2014, ApJ, 797, 109 
%\bibitem[\protect\citeauthoryear{Richards et al.}{2001}]{R01}
% Richards, G. T., Weinstein, M. A., Schneider, D. P., et al. 2001, AJ, 122, 1151
\bibitem[\protect\citeauthoryear{Richards et al.}{2009}]{R09}
 Richards, G. T., Myers, A. D., Gray, A. G., Riegel, R. N., Nichol, R. C., Brunner, R. J., Szalay, A. S., Schneider, D. P. \& Anderson, S.F., 2009, AJSS, 180, 67
\bibitem[\protect\citeauthoryear{R\"oser \& Meisenheimer}{1991}]{RM91}
 R\"oser, H.-J., \& Meisenheimer, K., 1991, A\&A, 252, 458
%\bibitem[\protect\citeauthoryear{Rozo et al.}{2016}]{Rozo16}
% Rozo, E., Rykoff, E. S., et al., 2016, MNRAS, 461, 1431 
%\bibitem[\protect\citeauthoryear{Rykoff et al.}{2016}]{Ryk16}
% Rykoff, E. S., Rozo, E., Hollowood, D., et al., 2016, ApJSS, 224, 1 
\bibitem[\protect\citeauthoryear{Sadeh et al.}{2015}]{2015arXiv150700490S}
 Sadeh, I., Abdalla, F. B. \& Lahav, O., 2015, A\&A, 474, L5
\bibitem[\protect\citeauthoryear{Saglia et al.}{2012}]{Saglia12}
 Saglia, R. P., Tonry, J. L., Bender, R., et al. 2012, ApJ, 746, 128
\bibitem[\protect\citeauthoryear{S\'{a}nchez et al.}{2014}]{San14}
 S\'{a}nchez, C., Carrasco Kind, M., Lin, H., Miquel, R., et al., 2014, MNRAS, 445, 1482 
\bibitem[\protect\citeauthoryear{Schirmer}{2013}]{Schi13}
 Schirmer, M., 2013, ApJS, 209, 21
\bibitem[\protect\citeauthoryear{Scoville et al.}{2007}]{Sco07}
 Scoville, N., et al. 2007, ApJSS, 172, 150
\bibitem[\protect\citeauthoryear{Shanks et al.}{2015}]{Sh15}
 Shanks, T., Metcalfe, N., Chehade, B., Findlay, J. R., et al., 2015, MNRAS, 451, 4238
\bibitem[\protect\citeauthoryear{Skrutskie et al.}{2006}]{2MASS}
 Skrutskie, M. F., Cutri, R. M., Stiening, R., Weinberg, M. D., et al. 2006, AJ, 131, 1163 
\bibitem[\protect\citeauthoryear{Spitler et al.}{2014}]{Spitler14}
 Spitler, L., et al., 2014, ApJ, 787, 36
\bibitem[\protect\citeauthoryear{Specht}{1990}]{PNN}
 Specht, D. F., 1990, Neural Networks, 3, 109
\bibitem[\protect\citeauthoryear{Stebbins \& Whitford}{1948}]{SW48}
 Stebbins, J., Whitford, A. E., 1948, AJ, 53, 204
%\bibitem[\protect\citeauthoryear{Taylor et al.}{2004}]{Tay04}
% Taylor, A. N., Bacon, D. J., Gray, M. E., Wolf, C., et al., 2004, MNRAS, 353, 1176
%\bibitem[\protect\citeauthoryear{Wadadekar}{2005}]{Wa05}
% Wadadekar, Y., 2005, PASP, 117, 79
\bibitem[\protect\citeauthoryear{Van Der Walt et al.}{2011}]{numpy}
 Van Der Walt, S., Colbert, S. C. \& Varoquaux, G., 2011, Computing in Science \& Engineering, 13, 2
\bibitem[\protect\citeauthoryear{Wang et al.}{2007}]{Wang07}
 Wang, D., Zhang, Y. X., Liu, C. \& Zhao, Y. H., 2007, MNRAS, 382, 1601
\bibitem[\protect\citeauthoryear{Williams et al.}{1996}]{HDF}
 Williams, R. E., Blacker, B., Dickinson, M., Dixon, W. V. D., Ferguson, H. C. et al., 1996, AJ, 112, 1335
\bibitem[\protect\citeauthoryear{Wolf et al.}{1999}]{W99}
 Wolf, C., Meisenheimer, K., R\"oser, H.-J., et al., 1999, A\&A, 343, 399  
\bibitem[\protect\citeauthoryear{Wolf et al.}{2001}]{WMR01}
 Wolf, C., Meisenheimer, K., R\"oser, H.-J., 2001, A\&A, 365, 660
\bibitem[\protect\citeauthoryear{Wolf et al.}{2003a}]{Wolf03a}
 Wolf, C., Meisenheimer, K., Rix, H.-W., et al., 2003a, A\&A, 401, 73
\bibitem[\protect\citeauthoryear{Wolf et al.}{2004}]{W04}
 Wolf, C., Meisenheimer, K., Kleinheinrich, M. et al., 2004, A\&A, 421, 913
\bibitem[\protect\citeauthoryear{Wolf et al.}{2005}]{WGM05}
 Wolf, C., Gray, M., \& Meisenheimer, 2005, A\&A, 443, 435
\bibitem[\protect\citeauthoryear{Wolf et al.}{2008}]{W08}
 Wolf, C., Hildebrandt, H., Taylor, E. N. \& Meisenheimer, K., 2008, A\&A, 492, 933
\bibitem[\protect\citeauthoryear{Wolf}{2009}]{W09}
 Wolf, C., 2009, MNRAS, 397, 520
\bibitem[\protect\citeauthoryear{Wolf et al.}{2017}]{W17}
 Wolf, C., et al., 2017, MNRAS, in preparation
\bibitem[\protect\citeauthoryear{Wright et al.}{2010}]{WISE}
 Wright, E. L., Eisenhardt, P. R. M., Mainzer, A. K., Ressler, M. E. et al., 2010, AJ, 140, 1868 
\bibitem[\protect\citeauthoryear{York et al.}{2000}]{SDSS}
 York, D. G., et al., 2000, AJ, 120, 1579
%\bibitem[\protect\citeauthoryear{Zatloukal et al.}{2005}]{HIROCS}
% Zatloukal, M., R\"{o}ser, H.-J., Wolf, C., Hippelein, H. \& Falter, S., 2007, A\&A, 474, L5
\end{thebibliography}
 %
\iffalse
\bibitem[\protect\citeauthoryear{Boris et al.}{2007}]{Bo07}   % locally linear approximations 
  Boris, N. V., Sodre, L., Jr., Cypriano, E. S., Santos, W. A., Mendes de Oliveira, C. \& West, M.,
  2007, ApJ, 666, 747
\bibitem[\protect\citeauthoryear{Cunha et al.}{2008}]{Cun08}		% weights approach as in Lima+ 2008
  Cunha, C. E., Lima, M., Oyaizu, H., Frieman, J. \& Lin, H., 2008, MNRAS, submitted
\bibitem[\protect\citeauthoryear{Wolf et al.}{2008}]{W08}
  Wolf, C., Hildebrandt, H., Taylor, E. N. \& Meisenheimer, K., 2008, A\&A, 492, 933
\bibitem[\protect\citeauthoryear{Finkbeiner, Davis \& Schlegel}{2000}]{FDS00}
 Finkbeiner, D. P., Davis, M. \& Schlegel, D. J., 2000, ApJ, 544, 81
\bibitem[\protect\citeauthoryear{Schlegel, Finkbeiner \& Davis}{1998}]{SFD98}
 Schlegel, D. J., Finkbeiner, D. P. \& Davis, M. 1998, ApJ, 500, 525
\fi
\end{document}

1612.00839.extracted_bib
[\protect\citeauthoryear{Ahn et al.{2014]{Ahn14 Ahn, C. P., et al. 2014, ApJS, 211, 17
1612.00839.extracted_bib
[\protect\citeauthoryear{Asorey et al.{2016]{Asorey16 Asorey, J., Carrasco Kind, M., Sevilla-Noarbe, I., Brunner, R. J. \& Thaler, J., 2016, MNRAS, 459, 1293 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Bacon et al.{2005]{Bacon05 % Bacon, D. J., Taylor, A. N., Brown, M. L., Gray, M. E., Wolf, C., et al. 2005, MNRAS, 363, 723
1612.00839.extracted_bib
[\protect\citeauthoryear{Baldry et al.{2004]{Bal04 Baldry, I. K., Glazebrook, K., Brinkmann, J., Ivezi\'{c, \v{Z., Lupton, R. H., Nichol, R. C. \& Szalay, A. S., 2004, ApJ, 600, 681
1612.00839.extracted_bib
[\protect\citeauthoryear{Baldry et al.{2014]{Bal14 Baldry, I. K., et al., 2014, MNRAS, 441, 244 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Ball et al.{2008]{Ball08 % Ball, N. M.,, Brunner, R. J., Myers, A. D., Strand, N. E., Alberts, S. L. \& Tcheng, D., 2008, ApJ, 683, 12
1612.00839.extracted_bib
[\protect\citeauthoryear{Baum{1957]{Baum57 Baum, W. A., 1957, AJ, 62, 6 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Baum{1962]{Baum62 % Baum, W. A., 1962, in {Problems of Extragalactic Research, % ed. McVittie, G. C., Macmilliam, IAU Symposium 15, 390
1612.00839.extracted_bib
[\protect\citeauthoryear{Beck et al.{2016]{Beck16 Beck, R., Dobos, L., Budavari, T., Szalay, A. S. \& Csabai, I., 2016, MNRAS, 460, 1371
1612.00839.extracted_bib
[\protect\citeauthoryear{Bell et al.{2004]{Bell04 Bell, E. F., Wolf, C., et al. 2004, ApJ, 608, 752
1612.00839.extracted_bib
[\protect\citeauthoryear{Bender et al.{2001]{Ben01 Bender, R., et al., 2001, p. 96 in Deep Fields, ESO Astrophysics Symposia, ed. by Cristiani, S., Renzini, A. \& Williams, R. E., Springer
1612.00839.extracted_bib
[\protect\citeauthoryear{Benitez{2000]{B00 Benitez, N., 2000, ApJ, 536, 571
1612.00839.extracted_bib
[\protect\citeauthoryear{Benitez{2009]{B09 Benitez, N., Moles, M., et al., 2009, ApJ, 692, L5
1612.00839.extracted_bib
[\protect\citeauthoryear{Benitez{2011]{B11 Benitez, N., 2011, Astrophysics Source Code Library, record ascl:1108.011
1612.00839.extracted_bib
[\protect\citeauthoryear{Benitez et al.{2014]{JPAS Benitez, N., Dupke, R., Moles, M., Sodre, L.,et al., 2014, J-PAS "Red Book", arXiv:1403.5237
1612.00839.extracted_bib
[\protect\citeauthoryear{Bertin \& Arnouts{1996]{BA96 Bertin, E., Arnouts, S., 1996, A\&AS, 117, 393
1612.00839.extracted_bib
[\protect\citeauthoryear{Bilicki et al.{2014]{Bil14 Bilicki, M., Jarrett, T. H., Peacock, J. A., Cluver, M. E., \& Steward, L., 2014, ApJS, 210, 9
1612.00839.extracted_bib
[\protect\citeauthoryear{Bilicki et al.{2016]{Bil16 Bilicki, M., Peacock, J. A., Jarrett, T. H., et al., 2016, ApJS, 225, 5
1612.00839.extracted_bib
[\protect\citeauthoryear{Blake \& Bridle{2005]{BB05 Blake, C. \& Bridle, S., 2005, MNRAS, 363, 1329
1612.00839.extracted_bib
[\protect\citeauthoryear{Blake et al.{2010]{Blake10 Blake, C., Brough, S., Colless, M., Couch, W., Croom, S., et al., 2010, MNRAS, 406, 803
1612.00839.extracted_bib
[\protect\citeauthoryear{Blake et al.{2016]{Blake16 Blake, C., et al., 2016, MNRAS, submitted
1612.00839.extracted_bib
[\protect\citeauthoryear{Bolzonella et al.{2000]{HyperZ Bolzonella, M., Miralles, J.-M. \& Pell\'{o, R., 2000, A\&A 363, 476
1612.00839.extracted_bib
[\protect\citeauthoryear{Bonnett{2015]{Bon15 Bonnett, C., 2015, MNRAS, 449, 1043 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Brown et al.{2003]{Brown03 % Brown, M. L., Taylor, A. N., Bacon, D. J., Gray, M. E., Dye, S., Meisenheimer, K., Wolf, C., et al. 2003, MNRAS, 341, 100
1612.00839.extracted_bib
[\protect\citeauthoryear{Brammer et al.{2008]{EAZY Brammer, G. B., van Dokkum, P G., Coppi, P., 2008, ApJ, 686, 1503
1612.00839.extracted_bib
[\protect\citeauthoryear{Bruzual \& Charlot{1993]{BC93 Bruzual, A. G., \& Charlot, S. 1993, ApJ, 405, 538
1612.00839.extracted_bib
[\protect\citeauthoryear{Budavari et al.{2000]{Bud00 Budavari, T., Szalay, A. S., Connolly, A. J., Csabai, I. \& Dickinson, M., 2001, AJ, 120, 1588
1612.00839.extracted_bib
[\protect\citeauthoryear{Budavari et al.{2001]{Bud01 Budavari, T., Csabai, I., Szalay, A. S., Connolly, A. J., et al., 2001, AJ, 122, 1163 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Budavari{2009]{Bud09 % Budavari, T., 2009, ApJ, 695, 747
1612.00839.extracted_bib
[\protect\citeauthoryear{Bundy et al.{2015]{Bundy15 Bundy, K., Leauthaud, A., Saito, S., Bolton, A., Lin, Y.-T., Maraston, C., Nichol, R. C., Schneider, D. P., Thomas, D. \& Wake, D. A., 2015, ApJSS, 221, 15
1612.00839.extracted_bib
[\protect\citeauthoryear{Capak{2004]{Cap04 Capak, P., PhD Thesis, 2004
1612.00839.extracted_bib
[\protect\citeauthoryear{Capak et al.{2007]{Cap07 Capak, P., Aussel, H., Ajiki, M., et al. 2007, ApJS, 172, 99
1612.00839.extracted_bib
[\protect\citeauthoryear{Carrasco Kind \& Brunner{2013]{CK13 Carrasco Kind, M. \& Brunner, R. J., 2013, MNRAS, 432, 1483
1612.00839.extracted_bib
[\protect\citeauthoryear{Coe et al.{2006]{Coe06 Coe, D., Benitez, N., Sanchez, S. F., et al. 2006, AJ, 132, 926
1612.00839.extracted_bib
[\protect\citeauthoryear{Coleman et al.{1980]{CWW80 Coleman, G. D., Wu, C.-C., \& Weedman, D. W. 1980, ApJS, 43, 393
1612.00839.extracted_bib
[\protect\citeauthoryear{Colless et al.{2001]{GRS Colless, M., Dalton G., Maddox, S., Sutherland, W., Norberg, P. et al., 2001, MNRAS, 328, 1039
1612.00839.extracted_bib
[\protect\citeauthoryear{Collister \& Lahav{2004]{CL04 Collister, A. A. \& Lahav, O., 2004, PASP, 116, 345 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Connolly et al.{1995a]{Con95 % Connolly, A. J., Csabai, I., Szalay, A. S., Koo, D. C., Kron, R. G. \& Munn, J. A., 1995, AJ, 110, 2655
1612.00839.extracted_bib
[\protect\citeauthoryear{Connolly et al.{1995b]{Con95ortho Connolly, A. J., Szalay, A. S., Bershady, M. A., Kinney, A. L., Calzetti, D., 1995, AJ, 110, 1071
1612.00839.extracted_bib
[\protect\citeauthoryear{Csabai et al.{2000]{Csa00 Csabai, I., Connolly, A. J., Szalay, A. S. \& Budavari, T., 2000, AJ, 119, 69
1612.00839.extracted_bib
[\protect\citeauthoryear{Csabai et al.{2003]{Csa03 Csabai, I., Budavari, T., Connolly, A. J., Szalay, A. S., et al., 2003, ApJ, 125, 580
1612.00839.extracted_bib
[\protect\citeauthoryear{Csabai et al.{2007]{Csa07 Csabai, I., Dobos, L., Trencseni, M., Herczegh, G., Josza, P., Purger, N., Budavari \& T., Szalay, A. S., 2007, AN, 328, 852 %
1612.00839.extracted_bib
[\protect\citeauthoryear{de Jong et al.{2015]{deJong15 % de Jong, J. T. A., Verdoes Kleijn, G. A., Boxhoorn, D. R., et al., 2015, arXiv:1507.00742
1612.00839.extracted_bib
[\protect\citeauthoryear{de Jong et al.{2013]{deJong13 de Jong, J. T. A. , Kuijken, K., Applegate, D., Begeman, K., Belikov, A., et al., 2013, ESO Messenger, 154, 44
1612.00839.extracted_bib
[\protect\citeauthoryear{Drinkwater et al.{2010]{Dr10 Drinkwater, M. J., Jurek, R. J., Blake, C., Woods, D., et al., 2010, MNRAS, 401, 1429
1612.00839.extracted_bib
[\protect\citeauthoryear{Driver et al.{2011]{GAMA Driver, S. P., Hill, D. T., Kelvin, L. S., Robotham, A. S. G., et al., 2011, MNRAS, 413, 971
1612.00839.extracted_bib
[\protect\citeauthoryear{Driver et al.{2015]{Dr15 Driver, S. P., Wright, A. H., Andrews, S. K., Davies, L. J., et al., 2015, MNRAS, arXiv:1508.02076
1612.00839.extracted_bib
[\protect\citeauthoryear{Driver et al.{2016]{WAVES Driver, S. P., Davies, L. J., Meyer, M., Power, C., Robotham, A. S. G., Baldry, I. K., Liske, J. \& Norberg, P., 2016, ASSP, 42, 205 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Durret et al.{2015]{Durret15 % Durret, F., Adami, C., Bertin, E., Hao, J., et al., 2015, A\&A, 578, A79
1612.00839.extracted_bib
[\protect\citeauthoryear{Edmondson et al.{2006]{Edd06 Edmondson, E. M., Miller, L. \& Wolf, C., 2006, MNRAS, 371, 1693
1612.00839.extracted_bib
[\protect\citeauthoryear{Fioc \& Rocca-Volmerange{1997]{FRV97 Fioc, M., \& Rocca-Volmerange, B., 1997, A\&A, 326, 950
1612.00839.extracted_bib
[\protect\citeauthoryear{Firth et al.{2003]{FLS03 Firth, A. E., Lahav, O. \& Somerville, R. S., 2003, MNRAS, 339, 1195
1612.00839.extracted_bib
[\protect\citeauthoryear{Gerdes et al.{2010]{Gerdes10 Gerdes, D. W., Sypniewski, A. J., McKay, T. A., Hao, J., Weis, M. R., Wechsler, R. H. \& Busha, M. T., 2010, ApJ, 715, 823
1612.00839.extracted_bib
[\protect\citeauthoryear{Giavalisco et al.{2004]{Giav04 Giavalisco, M., Ferguson, H. C., Koekemoer, A. M., et al. 2004, ApJ, 600, L93 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Gladders \& Yee{2005]{GY05 % Gladders, M. D. \& Yee, H. K. C., 2005, ApJS, 157, 1
1612.00839.extracted_bib
[\protect\citeauthoryear{Gonzalez \& DEEP3 Team{priv. comm.]{DeepPC Gonzalez, V. \& DEEP2 Team, private communication
1612.00839.extracted_bib
[\protect\citeauthoryear{Graff et al.{2014]{2014MNRAS.441.1741G Graff, P., Feroz, F., Hobson, M. P. \& Lasenby, A., MNRAS, 2014, 441, 1741
1612.00839.extracted_bib
[\protect\citeauthoryear{Gray et al.{2002]{Gray02 Gray, M. E., Taylor, A. N., Meisenheimer, K., Dye, S., Wolf, C. \& Thommes, E., 2002, ApJ, 568, 141
1612.00839.extracted_bib
[\protect\citeauthoryear{Hastie et al.{2008]{Hastie08 Hastie, T., Tibshirani, R. \& Friedman, J., in The elements of statistical learning: data mining, inference and prediction, Springer, 2008
1612.00839.extracted_bib
[\protect\citeauthoryear{Hickson et al.{1994]{H94 Hickson, P., Gibson, B. K. \& Callaghan, K. 1994, MNRAS, 267, 911
1612.00839.extracted_bib
[\protect\citeauthoryear{Hildebrandt et al.{2010]{PHAT Hildebrandt, H., Arnouts, S., Capak, P., Moustakas, L. A., Wolf, C., et al. 2010, A\&A, 523, A31
1612.00839.extracted_bib
[\protect\citeauthoryear{Hildebrandt et al.{2012]{H12 Hildebrandt, H., et al. 2012, MNRAS, 421, 2355
1612.00839.extracted_bib
[\protect\citeauthoryear{Hildebrandt et al.{2016]{H16 Hildebrandt, H., Viola, M., Heymans, C., Joudaki, S., Kuijken, K., et al. 2016, MNRAS, in press, arXiv:1606.05338
1612.00839.extracted_bib
[\protect\citeauthoryear{Hinton et al.{2016]{MARZ Hinton, S. R., Davis, T. M., Lidman, C., Glazebrook, K. \& Lewis, G. F., 2016, Astronomy \& Computing, 15, 61
1612.00839.extracted_bib
[\protect\citeauthoryear{Hoecker et al.{2007]{Hoecker07 Hoecker et al. 2007 %%%%%%%%%%%%%%%%%%%%%
1612.00839.extracted_bib
[\protect\citeauthoryear{Ilbert et al.{2006]{I06 Ilbert, O., Arnouts, S., McCracken, H. J., et al. 2006, A\&A, 457, 841
1612.00839.extracted_bib
[\protect\citeauthoryear{Ilbert et al.{2009]{I09 Ilbert, O., Capak, P., Salvato, M., et al. 2009, ApJ, 690, 1236
1612.00839.extracted_bib
[\protect\citeauthoryear{Jarrett et al.{2011]{J11 Jarrett, T. H., Cohen, M., Masci, F., Wright, E., Stern, D. et al. 2011, ApJ, 735, 112
1612.00839.extracted_bib
[\protect\citeauthoryear{Johnson et al.{2016]{J16 Johnson, A. S., Blake, C., et al. 2016, MNRAS, in preparation
1612.00839.extracted_bib
[\protect\citeauthoryear{Johnston et al.{2007]{Jo07 Johnston, S., et al., 2007, PASA, 24, 174
1612.00839.extracted_bib
[\protect\citeauthoryear{Johnston et al.{2008]{Jo08 Johnston, S., Taylor, R., Bailes, M, et al., 2008, Exp. Astron., 22, 151 (doi:10.1007/s10686-008-9124-7)
1612.00839.extracted_bib
[\protect\citeauthoryear{Jones et al.{2009]{6dF Jones, D. H., Read, M. A., Saunders, W., Colless, M., et al. 2009, MNRAS, 399, 683
1612.00839.extracted_bib
[\protect\citeauthoryear{Keller et al.{2007]{K07 Keller, S. C., Schmidt, B. P., Bessell, M. S., Conroy, P. G., Francis, P., et al., 2007, PASA, 24, 1 % Granlund, A., Kowald, E., Oates, A. P., Martin-Jones, T., Preston, T., Tisserand, P., Vaccarella, A. \& Waterson, M. F.,
1612.00839.extracted_bib
[\protect\citeauthoryear{Kilbinger{2015]{Kilb15 Kilbinger, M., 2015, Reports on Progress in Physics, 78, 8, article id. 086901
1612.00839.extracted_bib
[\protect\citeauthoryear{Kinney et al.{1996]{Kin96 Kinney, A. L., Calzetti, D., Bohlin, R. C., et al. 1996, ApJ, 467, 38
1612.00839.extracted_bib
[\protect\citeauthoryear{Kitching et al.{2007]{Kit07 Kitching, T. D., Heavens, A. F., Taylor, A. N., Brown, M. L., Meisenheimer, K., Wolf, C., Gray, M. E. \& Bacon, D. J., 2007, MNRAS, 376, 771 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Kitching et al.{2008]{Kit08 % Kitching, T. D., Taylor, A. N. \& Heavens, A. F., 2008, MNRAS, 389, 173 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Koo{1985]{Koo85 % Koo, D. C., 1985, AJ, 90, 418
1612.00839.extracted_bib
[\protect\citeauthoryear{Kurcz et al.{2016]{Kurcz16 Kurcz, A., Bilicki, M., Solarz, A., Krupa, M., Pollo, A. \& Ma\l{ek, K., 2016, A\&A, arXiv:1604.04229
1612.00839.extracted_bib
[\protect\citeauthoryear{Kuijken{2008]{K08 Kuijken, K. H. 2008, A\&A, 482, 1053
1612.00839.extracted_bib
[\protect\citeauthoryear{Kuijken et al.{2015]{Kui15 Kuijken, K., Heymans., C., Hildebrandt, H., Nakajima, R., et al., 2015, MNRAS, 454, 3500
1612.00839.extracted_bib
[\protect\citeauthoryear{Lanzetta et al.{1996]{Lan96 Lanzetta, K. M., Webb, J. K. \& Fern\'{andez-Soto, A., 1996, Nature, 381, 759 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Leauthaud et al.{2012]{Leaut12 % Leauthaud, A., Tinker, J., Bundy, K., Behroozi, P. S., Massey, R. Rhodes, J. et al., 2012, ApJ, 744, 159
1612.00839.extracted_bib
[\protect\citeauthoryear{Le F\`{evre et al.{2013]{VVDS Le F\`{evre, O., Cassata, P., Cucciati, O., et al. 2013, A\&A, 559, A14
1612.00839.extracted_bib
[\protect\citeauthoryear{Lilly et al.{1995]{Lilly95 Lilly, S. J., Le Fevre, O., Crampton, D., Hammer, F. \& Tresse, L., 1995, ApJ, 455, 50
1612.00839.extracted_bib
[\protect\citeauthoryear{Lima et al.{2008]{Lima08 % weights approach Lima, M., Cunha, C. E., Oyaizu, H., Frieman, J., Lin, H. \& Sheldon, E. S., 2008, MNRAS, 390, 118
1612.00839.extracted_bib
[\protect\citeauthoryear{Loh \& Spillar{1986]{LS86 Loh, E. D. \& Spillar, E. J. 1986, ApJ, 303, 154
1612.00839.extracted_bib
[\protect\citeauthoryear{MacKay{2003]{MacKay MacKay, D. J. C., in Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003
1612.00839.extracted_bib
[\protect\citeauthoryear{Mandelbaum et al.{2008]{Man08 Mandelbaum, R., Seljak, U., Hirata, C. M., et al., 2008, MNRAS, 386, 781
1612.00839.extracted_bib
[\protect\citeauthoryear{Mart\'{i et al.{2014]{PAU Mart\'{i, P., Miquel, R., Castander, F. J., Gazta\~{naga, E., Eriksen, M. \& S\'{anchez, C., 2014, MNRAS, 442, 92
1612.00839.extracted_bib
[\protect\citeauthoryear{Masters et al.{2015]{Mas15 Masters, D., Capak, P., Stern, D., Ilbert, O., Salvato, M., Schmidt, S., et al., 2015, ApJ, 813, 53 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Muzzin et al.{2013]{Muzz13 % Muzzin, A., Wilson, G., Demarco, R., Lidman, C., Nantais, J., Hoekstra, H., Yee, H. K. C. \& Rettura, A., 2013, ApJ, 767, 39 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Nanayakkara et al.{2016]{ZFIRE % Nanayakkara, T., Glazebrook, K., Kacprzak, G. G., Yuan, T., et al., 2016, ApJ, 828, 21
1612.00839.extracted_bib
[\protect\citeauthoryear{Newman{2008]{New08 % STILL NEED TO CITE SOMEWHERE! Newman, J. A., 2008, ApJ, 684, 88
1612.00839.extracted_bib
[\protect\citeauthoryear{Newman et al.{2013]{New13 Newman, J. A., et al., 2013, arXiv:1309.5384
1612.00839.extracted_bib
[\protect\citeauthoryear{Niemack et al.{2009]{Nie09 Niemack, M. D., Jimenez, R., Verde, L., Menanteau, F., Panter, B. \& Spergel, D., 2009, ApJ, 690, 89
1612.00839.extracted_bib
[\protect\citeauthoryear{Norris et al.{2011]{Norris11 Norris, R. P., Hopkins, A. M., et al., 2011, PASA, 28, 215
1612.00839.extracted_bib
[\protect\citeauthoryear{Oyaizu et al.{2008]{Oya08a Oyaizu, H., Lima, M., Cunha, C. E., Lin, H., Frieman, J. \& Sheldon, E. S., 2008, ApJ, 674, 768 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Oyaizu et al.{2008b]{Oya08b % Oyaizu, H., Lima, M., Cunha, C. E., Lin, H. \& Frieman, J., 2008b, ApJ, 689, 709
1612.00839.extracted_bib
[\protect\citeauthoryear{Prakash et al.{2016]{P16 Prakash, A., et al., 2016, ApJS, 224, 34
1612.00839.extracted_bib
[\protect\citeauthoryear{Raichoor et al.{2014]{R14 Raichoor, A., Mei, S., Erben, T., Hildebrandt, H., Huertas-Company, M., Ilbert, O., Licitra, R., et al. 2014, ApJ, 797, 102 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Rettura et al.{2014]{Rett14 % Rettura, A., Martinez-Manso, J., Stern, D., Mei, S. et al., 2014, ApJ, 797, 109 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Richards et al.{2001]{R01 % Richards, G. T., Weinstein, M. A., Schneider, D. P., et al. 2001, AJ, 122, 1151
1612.00839.extracted_bib
[\protect\citeauthoryear{Richards et al.{2009]{R09 Richards, G. T., Myers, A. D., Gray, A. G., Riegel, R. N., Nichol, R. C., Brunner, R. J., Szalay, A. S., Schneider, D. P. \& Anderson, S.F., 2009, AJSS, 180, 67
1612.00839.extracted_bib
[\protect\citeauthoryear{R\"oser \& Meisenheimer{1991]{RM91 R\"oser, H.-J., \& Meisenheimer, K., 1991, A\&A, 252, 458 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Rozo et al.{2016]{Rozo16 % Rozo, E., Rykoff, E. S., et al., 2016, MNRAS, 461, 1431 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Rykoff et al.{2016]{Ryk16 % Rykoff, E. S., Rozo, E., Hollowood, D., et al., 2016, ApJSS, 224, 1
1612.00839.extracted_bib
[\protect\citeauthoryear{Sadeh et al.{2015]{2015arXiv150700490S Sadeh, I., Abdalla, F. B. \& Lahav, O., 2015, A\&A, 474, L5
1612.00839.extracted_bib
[\protect\citeauthoryear{Saglia et al.{2012]{Saglia12 Saglia, R. P., Tonry, J. L., Bender, R., et al. 2012, ApJ, 746, 128
1612.00839.extracted_bib
[\protect\citeauthoryear{S\'{anchez et al.{2014]{San14 S\'{anchez, C., Carrasco Kind, M., Lin, H., Miquel, R., et al., 2014, MNRAS, 445, 1482
1612.00839.extracted_bib
[\protect\citeauthoryear{Schirmer{2013]{Schi13 Schirmer, M., 2013, ApJS, 209, 21
1612.00839.extracted_bib
[\protect\citeauthoryear{Scoville et al.{2007]{Sco07 Scoville, N., et al. 2007, ApJSS, 172, 150
1612.00839.extracted_bib
[\protect\citeauthoryear{Shanks et al.{2015]{Sh15 Shanks, T., Metcalfe, N., Chehade, B., Findlay, J. R., et al., 2015, MNRAS, 451, 4238
1612.00839.extracted_bib
[\protect\citeauthoryear{Skrutskie et al.{2006]{2MASS Skrutskie, M. F., Cutri, R. M., Stiening, R., Weinberg, M. D., et al. 2006, AJ, 131, 1163
1612.00839.extracted_bib
[\protect\citeauthoryear{Spitler et al.{2014]{Spitler14 Spitler, L., et al., 2014, ApJ, 787, 36
1612.00839.extracted_bib
[\protect\citeauthoryear{Specht{1990]{PNN Specht, D. F., 1990, Neural Networks, 3, 109
1612.00839.extracted_bib
[\protect\citeauthoryear{Stebbins \& Whitford{1948]{SW48 Stebbins, J., Whitford, A. E., 1948, AJ, 53, 204 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Taylor et al.{2004]{Tay04 % Taylor, A. N., Bacon, D. J., Gray, M. E., Wolf, C., et al., 2004, MNRAS, 353, 1176 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Wadadekar{2005]{Wa05 % Wadadekar, Y., 2005, PASP, 117, 79
1612.00839.extracted_bib
[\protect\citeauthoryear{Van Der Walt et al.{2011]{numpy Van Der Walt, S., Colbert, S. C. \& Varoquaux, G., 2011, Computing in Science \& Engineering, 13, 2
1612.00839.extracted_bib
[\protect\citeauthoryear{Wang et al.{2007]{Wang07 Wang, D., Zhang, Y. X., Liu, C. \& Zhao, Y. H., 2007, MNRAS, 382, 1601
1612.00839.extracted_bib
[\protect\citeauthoryear{Williams et al.{1996]{HDF Williams, R. E., Blacker, B., Dickinson, M., Dixon, W. V. D., Ferguson, H. C. et al., 1996, AJ, 112, 1335
1612.00839.extracted_bib
[\protect\citeauthoryear{Wolf et al.{1999]{W99 Wolf, C., Meisenheimer, K., R\"oser, H.-J., et al., 1999, A\&A, 343, 399
1612.00839.extracted_bib
[\protect\citeauthoryear{Wolf et al.{2001]{WMR01 Wolf, C., Meisenheimer, K., R\"oser, H.-J., 2001, A\&A, 365, 660
1612.00839.extracted_bib
[\protect\citeauthoryear{Wolf et al.{2003a]{Wolf03a Wolf, C., Meisenheimer, K., Rix, H.-W., et al., 2003a, A\&A, 401, 73
1612.00839.extracted_bib
[\protect\citeauthoryear{Wolf et al.{2004]{W04 Wolf, C., Meisenheimer, K., Kleinheinrich, M. et al., 2004, A\&A, 421, 913
1612.00839.extracted_bib
[\protect\citeauthoryear{Wolf et al.{2005]{WGM05 Wolf, C., Gray, M., \& Meisenheimer, 2005, A\&A, 443, 435
1612.00839.extracted_bib
[\protect\citeauthoryear{Wolf et al.{2008]{W08 Wolf, C., Hildebrandt, H., Taylor, E. N. \& Meisenheimer, K., 2008, A\&A, 492, 933
1612.00839.extracted_bib
[\protect\citeauthoryear{Wolf{2009]{W09 Wolf, C., 2009, MNRAS, 397, 520
1612.00839.extracted_bib
[\protect\citeauthoryear{Wolf et al.{2017]{W17 Wolf, C., et al., 2017, MNRAS, in preparation
1612.00839.extracted_bib
[\protect\citeauthoryear{Wright et al.{2010]{WISE Wright, E. L., Eisenhardt, P. R. M., Mainzer, A. K., Ressler, M. E. et al., 2010, AJ, 140, 1868
1612.00839.extracted_bib
[\protect\citeauthoryear{York et al.{2000]{SDSS York, D. G., et al., 2000, AJ, 120, 1579 %
1612.00839.extracted_bib
[\protect\citeauthoryear{Zatloukal et al.{2005]{HIROCS % Zatloukal, M., R\"{oser, H.-J., Wolf, C., Hippelein, H. \& Falter, S., 2007, A\&A, 474, L5
1612.00839.extracted_bib
[\protect\citeauthoryear{Boris et al.{2007]{Bo07 % locally linear approximations Boris, N. V., Sodre, L., Jr., Cypriano, E. S., Santos, W. A., Mendes de Oliveira, C. \& West, M., 2007, ApJ, 666, 747
1612.00839.extracted_bib
[\protect\citeauthoryear{Cunha et al.{2008]{Cun08 % weights approach as in Lima+ 2008 Cunha, C. E., Lima, M., Oyaizu, H., Frieman, J. \& Lin, H., 2008, MNRAS, submitted
1612.00839.extracted_bib
[\protect\citeauthoryear{Wolf et al.{2008]{W08 Wolf, C., Hildebrandt, H., Taylor, E. N. \& Meisenheimer, K., 2008, A\&A, 492, 933
1612.00839.extracted_bib
[\protect\citeauthoryear{Finkbeiner, Davis \& Schlegel{2000]{FDS00 Finkbeiner, D. P., Davis, M. \& Schlegel, D. J., 2000, ApJ, 544, 81
1612.00839.extracted_bib
[\protect\citeauthoryear{Schlegel, Finkbeiner \& Davis{1998]{SFD98 Schlegel, D. J., Finkbeiner, D. P. \& Davis, M. 1998, ApJ, 500, 525 \fi