A Breakthrough in Astronomical Software Development
Raymond Plante
Doug Roberts
% R.\ M.\ Crutcher\altaffilmark{1}
% and the start of your text. \begin{abstract} % Leave intact % Place the text of your abstract here - NO BLANK LINES The advent of wide-area multicolour synoptic sky surveys is leading to data sets unprecedented in size, complexity and data throughput. VO technology offers a way to exploit these to the full but requires changes in design philosophy. The Palomar-QUEST survey is a major new survey being undertaken by Caltech, Yale, JPL and Indiana University to repeatedly observe $\frac{1}{3}$ of the sky ($\sim 15000$ sq. deg. between $-27^\circ \le \delta \le 27^\circ$) in seven passbands. Utilising the 48-inch Oschin Schmidt Telescope at the Palomar Observatory with the 112-CCD QUEST camera covering the full 4$^\circ$ x 4$^\circ$ field of view, it will generate $\sim 1$TB of data per month. In this paper, we review the design of QUEST as a VO resource, a federated data set and an exemplar of VO standards.
adass2003.tex
%APN3_PROCEEDINGS_FORM%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% TEMPLATE.TEX -- APN3 (2003) ASP Conference Proceedings template.
%
% Derived from ADASS VIII (98) ASP Conference Proceedings template
% Updated by N. Manset for ADASS IX (99), F. Primini for ADASS 2000,
% D.Bohlender for ADASS 2001, and H. Payne for ADASS XII and LaTeX2e.
%
% Use this template to create your proceedings paper in LaTeX format
% by following the instructions given below.  Much of the input will
% be enclosed by braces (i.e., { }).  The percent sign, "%", denotes
% the start of a comment; text after it will be ignored by LaTeX.  
% You might also notice in some of the examples below the use of "\ "
% after a period; this prevents LaTeX from interpreting the period as
% the end of a sentence and putting extra space after it.  
% 
% You should check your paper by processing it with LaTeX.  For
% details about how to run LaTeX as well as how to print out the User
% Guide, consult the README file.  You should also consult the sample
% LaTeX papers, sample1.tex and sample2.tex, for examples of including
% figures, html links, special symbols, and other advanced features.
%
% If you do not have access to the LaTeX software or a laser printer
% at your site, you can still prepare your paper following the
% instructions in the User Guide.  In such cases, the editors will
% process the file and make any necessary editorial adjustments.
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
\documentclass[11pt,twoside]{article}  % Leave intact
\usepackage{adassconf}

% If you have the old LaTeX 2.09, and not the current LaTeX2e, comment
% out the \documentclass and \usepackage lines above and uncomment
% the following:

%\documentstyle[11pt,twoside,adassconf]{article}

\begin{document}   % Leave intact

%-----------------------------------------------------------------------
%			    Paper ID Code
%-----------------------------------------------------------------------
% Enter the proper paper identification code.  The ID code for your
% paper is the session number associated with your presentation as
% published in the official conference proceedings.  You can           
% find this number locating your abstract in the printed proceedings
% that you received at the meeting or on-line at the conference web
% site; the ID code is the letter/number sequence proceeding the title 
% of your presentation. 
%
% This will not appear in your paper; however, it allows different
% papers in the proceedings to cross-reference each other.  Note that
% you should only have one \paperID, and it should not include a
% trailing period.
%
% EXAMPLE: \paperID{O4-1}
% EXAMPLE: \paperID{P7-7}
%

\paperID{O1-3}

%-----------------------------------------------------------------------
%		            Paper Title 
%-----------------------------------------------------------------------
% Enter the title of the paper.
%
% EXAMPLE: \title{A Breakthrough in Astronomical Software Development}
% 
% If your title is so long as to fill the page header when you print it,
% then please supply a short form as a \titlemark.
%
% EXAMPLE: 
%  \title{Rapid Development for Distributed Computing, with Implications
%         for the Virtual Observatory}
%  \titlemark{Rapid Development for Distributed Computing}
%

\title{Palomar-QUEST: A case study in designing sky surveys in the VO era}
\titlemark{The Palomar-QUEST survey}

%-----------------------------------------------------------------------
%		          Authors of Paper
%-----------------------------------------------------------------------
% Enter the authors followed by their affiliations.  The \author and
% \affil commands may appear multiple times as necessary (see example
% below).  List each author by giving the first name or initials first
% followed by the last name.  Authors with the same affiliations
% should grouped together. 
%
% EXAMPLE: \author{Raymond Plante, Doug Roberts, 
%                  R.\ M.\ Crutcher\altaffilmark{1}}
%          \affil{National Center for Supercomputing Applications, 
%                 University of Illinois Urbana-Champaign, Urbana, IL
%                 61801}
%          \author{Tom Troland}
%          \affil{University of Kentucky}
%
%          \altaffiltext{1}{Astronomy Department, UIUC}
%
% In this example, the first three authors, "Plante", "Roberts", and
% "Crutcher" are affiliated with "NCSA".  "Crutcher" has an alternate 
% affiliation with the "Astronomy Department".  The fourth author,
% "Troland", is affiliated with "University of Kentucky"

\author{Matthew J. Graham, Roy Williams, S. G. Djorgovski, Ashish Mahabal}
\affil{California Institute of Technology, Pasadena, CA 91125, USA}
\author{Charles Baltay, Dave Rabinowitz, Anne Bauer, Jeff Snyder, Nick Morgan, Peter Andrews}
\affil{Yale University, New Haven, CT 06511, USA}
\author{Alexander S.  Szalay}
\affil{Dept. Physics and Astronomy, The Johns Hopkins University, Baltimore, MD 21218, USA}
\author{Robert J. Brunner}
\affil{Dept. of Astronomy \& NCSA, University of Illinois, Urbana, IL 61801, USA}
\author{Jim Musser}
\affil{Indiana University, Bloomington, Indiana, USA}

%-----------------------------------------------------------------------
%			 Contact Information
%-----------------------------------------------------------------------
% This information will not appear in the paper but will be used by
% the editors in case you need to be contacted concerning your
% submission.  Enter your name as the contact along with your email
% address.
% 
% EXAMPLE:  \contact{Dennis Crabtree}
%           \email{crabtree@cfht.hawaii.edu}
%

\contact{Matthew Graham }
\email{mjg@astro.caltech.edu }

%-----------------------------------------------------------------------
%		      Author Index Specification
%-----------------------------------------------------------------------
% Specify how each author name should appear in the author index.  The 
% \paindex{ } should be used to indicate the primary author, and the
% \aindex for all other co-authors.  You MUST use the following
% syntax: 
%
% SYNTAX:  \aindex{Lastname, F. M.}
% 
% where F is the first initial and M is the second initial (if
% used).  This guarantees that authors that appear in multiple papers
% will appear only once in the author index.  
%
% EXAMPLE: \paindex{Crabtree, D.}
%          \aindex{Manset, N.}        
%          \aindex{Veillet, C.}        
%
% NOTE: this information is also used to build the author list that
% appears in the table of contents.  Authors will be listed in the order
% of the \paindex and \aindex commmands.
%

\paindex{Graham, M. J.}
\aindex{Williams, R.}     % Remove this line if there is only one author
\aindex{Djorgovski, S. G.}
\aindex{Mahabal, A.}
\aindex{Baltay, C.}
\aindex{Rabinowitz, D.}
\aindex{Bauer, A.}
\aindex{Snyder, J.}
\aindex{Morgan, N.}
\aindex{Andrews, P.}
\aindex{Szalay, A. S.}
\aindex{Brunner, R. J.}
\aindex{Musser, J.}


%-----------------------------------------------------------------------
%		      Author list for page header	
%-----------------------------------------------------------------------
% Please supply a list of author last names for the page header. in
% one of these formats:
%
% EXAMPLES:
% \authormark{Lastname}
% \authormark{Lastname1 \& Lastname2}
% \authormark{Lastname1, Lastname2, ... \& LastnameN}
% \authormark{Lastname et al.}
%
% Use the "et al." form in the case of seven or more authors, or if
% the preferred form is too long to fit in the header.

\authormark{Graham et al.}

%-----------------------------------------------------------------------
%			Subject Index keywords
%-----------------------------------------------------------------------
% Enter a comma separated list of up to 6 keywords describing your
% paper.  These will NOT be printed as part of your paper; however,
% they will be used to generate the subject index for the proceedings.
% There is no standard list; however, you can consult the indices
% for past proceedings (http://adass.org/adass/proceedings/).
%
% EXAMPLE:  \keywords{visualization, astronomy: radio, parallel
%                     computing, AIPS++, Galactic Center}
%
% In this example, the author noticed that "radio astronomy" appeared
% in the ADASS VII Index as "astronomy" being the major keyword and
% "radio" as the minor keyword.  The colon is used to introduce another
% level into the index.

\keywords{ synoptic surveys, multicolour surveys, wide area surveys, VO }

%-----------------------------------------------------------------------
%			       Abstract
%-----------------------------------------------------------------------
% Type abstract in the space below.  Consult the User Guide and Latex
% Information file for a list of supported macros (e.g. for typesetting 
% special symbols). Do not leave a blank line between \begin{abstract} 
% and the start of your text.

\begin{abstract}          % Leave intact
% Place the text of your abstract here - NO BLANK LINES
The advent of wide-area multicolour synoptic sky surveys is leading to data sets unprecedented in size, complexity and data throughput. VO technology offers a way to exploit these to the full but requires changes in design philosophy.
The Palomar-QUEST survey is a major new survey being undertaken by Caltech, Yale, JPL and Indiana University to repeatedly observe $\frac{1}{3}$ of the sky ($\sim 15000$ sq. deg. between $-27^\circ \le \delta \le 27^\circ$) in seven passbands. Utilising the 48-inch Oschin Schmidt Telescope at the Palomar Observatory with the 112-CCD QUEST camera covering the full 4$^\circ$ x 4$^\circ$ field of view, it will generate $\sim 1$TB of data per month.
In this paper, we review the design of QUEST as a VO resource, a federated data set and an exemplar of VO standards.
\end{abstract}

%-----------------------------------------------------------------------
%			      Main Body
%-----------------------------------------------------------------------
% Place the text for the main body of the paper here.  You should use
% the \section command to label the various sections; use of
% \subsection is optional.  Significant words in section titles should
% be capitalized.  Sections and subsections will be numbered
% automatically. 
%
% EXAMPLE:  \section{Introduction}
%           ...
%           \subsection{Our View of the World}
%           ...
%           \section{A New Approach}
%
% It is recommended that you look at the sample papers, sample1.tex
% and sample2.tex, for examples for formatting references, footnotes,
% figures, equations, html links, lists, and other special features.  

\section{A new era in astronomy}

The new availability of wide-field images from Schmidt telescopes in the 1940's meant that astronomers no longer had to make educated guesses about where to look to find new and interesting phenomena but were now spoilt for choice. The advent of synoptic surveys presents more extreme opportunities; as an illustration, consider the SDSS which over the course of 5 years represents a factor of a million increase in information over previous surveys; however, the \htmladdnormallinkfoot{LSST}{http://www.lssto.org} (Large Sky Synoptic Telescope, Tyson (2002)) will amass a SDSS every 3 nights. 

Although overviews of synoptic surveys are riddled with cliches concerning undiscovered countries and uncharted waters, the exploration of the temporal domain results in data sets that are not just more voluminous than before, but far richer and more complex (Paczynski 2001; Djorgovski et al. 2000). This presents challenges to all aspects of astronomy: data gathering, distribution, reduction, analysis, storage, archiving, dissemination and mining. VO technologies are being designed precisely to meet these types of challenges, but to use them requires changes in survey design philosophies. 

\section{The Palomar-QUEST survey}

The \htmladdnormallinkfoot{Palomar-QUEST survey}{http://hepwww.physics.yale.edu/quest/palomar.html; http://www.astro.caltech.edu/~aam/science/quest}  is a major new survey being undertaken by Caltech, Yale, JPL and Indiana University employing the world's largest astronomical camera and the recently refurbished  Oschin Schmidt telescope at Palomar to observe a third of the sky ($\sim 15000$ sq. deg. between $-27^\circ \le \delta \le 27^\circ$) a minimum of 8 times in 7 passbands to nominally twice the depth of SDSS. 

The QUEST camera consists of 112 CCDs arranged in four filter strips. Each CCD has 2400 $\times$ 600  13$\mu$m $\times$ 13$\mu$m pixels, giving a total of 161 $\times$ 10$^6$ pixels. At the prime focus of the Oschin Schmidt, QUEST covers a sky area of 4.6$^\circ \times$ 3.6$^\circ$ (the effective area is $\sim 10$ sq. deg) and in a night can survey $\sim 500$ sq. deg. Two filter sets are used: Johnson $UBRI$ and Gunn $riz$, with a doubling of Gunn $z$ to afford extra depth. 

The data rate is 2.45MB/s and with a monthly average of 10 nights' observing, QUEST produces $\sim 1$TB of data/month.

Some of the immediate science goals are searching for high redshift quasars, strong gravitational lensing, supernovae and GRBs, and near-Earth asteroids and trans-Neptunian objects. Obviously once there is a sufficient body of repeat observations, searching for new types of variable object and phenomena will play a dominant part; in particular, a rapid response mechanism to transients (see section 4) is planned.

\section{QUEST and VO technologies}

As this survey is one of the first of the new breed of synoptic surveys, it is being used as a testbed for the VO technologies which will enable astronomers to exploit such surveys to the full.
There are currently four areas of attention:

\subsection{Data distribution}

Different groups want to process the raw data in different ways to optimize the detection of specific types of object. Access requirements to the data are also either near real-time or delayed. Data distribution must be secure, fault tolerant (error checking, multiply redundant) and accountable (transaction logging).

\subsection{Data processing}

The nature of the data is extremely well suited to parallelization, either on a multi-processor machine or in a more general distributed computing environment, e.g. an advanced highly CPU-intensive pipeline would be a suitable Grid-level application.

\subsection{Data analysis}

The identification of variable objects poses many problems:
\begin{itemize}
\item associating different observations under different conditions (e.g. seeing) with the same identification;
\item handling objects which only appear once (e.g. supernovae)
\item handling moving objects (e.g. asteroids)
\item optimally characterizing the variability of an object (periodic/aperiodic)
\item determining the best sampling strategy to maximize the range of temporal baselines covered
\end{itemize}

Other federated data sets will be employed in the data analysis to assist identification, e.g. SDSS, DPOSS, 2MASS.

\subsection{Data dissemination}

The deployment of QUEST as a federated data set needs to support both interactive and batch mode access.  Access to data products also needs to be transparent to the access rights of different users: QUEST survey team, collaborators and the general astronomy community.

\section{Integration example: 4 minute alert of detection of transient events}

To illustrate how QUEST will make use of VO technologies in an integrated fashion, consider one of the pipeline systems under construction (see Fig. 1 for a cartoon depiction): this will produce real time (within four minutes of the data being taken) alerts of transient events (e.g. supernovae). The specific processes which need to mesh are:
\begin{itemize}
\item {\em Distribution:} Every 140s, 112 $\times$ 3.1MB raw fits files are produced at Palomar and streamed to Caltech (at 10MB/s) where the CIT Data Broker distributes the data to other sites, the raw image archive and the reduction pipeline
\item {\em Processing:} The CIT Fast Pipeline computes a real-time flat and extracts objects - each field produces $\sim$10000 objects
\item {\em Analysis:} Variable and transient objects are detected by comparing the latest observations with the fiducial sky (composed from all QUEST observations and possibly other data sets) in the master archive. They are processed to determine whether they might be asteroids and checked against lists of known variables. Source classification is attempted using other federated data archives.
\item {\em Dissemination:} The Alert Decision Engine decides whether an alert should be issued based on decision algorithms and all available data and posts results to the website.
\end{itemize}

\begin{figure}
\plotone{realtime.eps}
\caption{Cartoon of the real time variable object detection pipeline which will issue an alert within four minutes of detection.}
\end{figure} 

\section {Conclusions}
Palomar-QUEST is the prototype VO-integrated synoptic sky survey and marks the beginning of an exciting new era in astronomy: the characterization of the variable optical sky.

\acknowledgements
MJG, RW, SGD and AAM acknowledge support from the NSF NVO program.
SGD, AAM and MJG also acknowledge a partial support from the NASA AISRP program.

\begin{references}
\reference Djorgovski, S.\ G.\  et al. 2001, in  Virtual Observatories of the Future, ed.\ R.\ J.\ Brunner, S.\ G.\  Djorgovski \& A.\ S.\ Szalay (ASP Conf. Ser. 225), 52  
\reference Paczynski, B.\ 2000, \pasp, 112, 1281
\reference Tyson, J.\ A.\  2002, SPIE, 4836, 10 
\end{references}
%-----------------------------------------------------------------------
%			      References
%-----------------------------------------------------------------------
% List your references below within the reference environment
% (i.e. between the \begin{references} and \end{references} tags).
% Each new reference should begin with a \reference command which sets
% up the proper indentation.  Observe the following order when listing
% bibliographical information for each reference:  author name(s),
% publication year, journal name, volume, and page number for
% articles.  Note that many journal names are available as macros; see
% the User Guide listing "macro-ized" journals.   
%
% EXAMPLE:  \reference Hagiwara, K., \& Zeppenfeld, D.\  1986, 
%                Nucl.Phys., 274, 1
%           \reference H\'enon, M.\  1961, Ann.d'Ap., 24, 369
%           \reference King, I.\ R.\  1966, \aj, 71, 276
%           \reference King, I.\ R.\  1975, in Dynamics of Stellar 
%                Systems, ed.\ A.\ Hayli (Dordrecht: Reidel), 99
%           \reference Tody, D.\  1998, \adassvii, 146
%           \reference Zacharias, N.\ \& Zacharias, M.\ 2003,
%                \adassxii, \paperref{P7.6}
% 
% Note the following tricks used in the example above:
%
%   o  \& is used to format an ampersand symbol (&).
%   o  \'e puts an accent agu over the letter e.  See the User Guide
%      and the sample files for details on formatting special
%      characters.  
%   o  "\ " after a period prevents LaTeX from interpreting the period 
%      as an end of a sentence.
%   o  \aj is a macro that expands to "Astron. J."  See the User Guide
%      for a full list of journal macros
%   o  \adassvii is a macro that expands to the full title, editor,
%      and publishing information for the ADASS VII conference
%      proceedings.  Such macros are defined for ADASS conferences I
%      through XI.
%   o  When referencing a paper in the current volume, use the
%      \adassxii and \paperref macros.  The argument to \paperref is
%      the paper ID code for the paper you are referencing.  See the 
%      note in the "Paper ID Code" section above for details on how to 
%      determine the paper ID code for the paper you reference.  
%
%\begin{references}
%\end{references}

% Do not place any material after the references section

\end{document}  % Leave intact
