A Breakthrough in Astronomical Software Development
Raymond Plante
Doug Roberts
% R.\ M.\ Crutcher\altaffilmark{1}
% and the start of your text. \begin{abstract} % Leave intact % Place the text of your abstract here - NO BLANK LINES The Grist project is developing a grid-technology based system as a research environment for astronomy with massive and complex datasets. This knowledge extraction system will consist of a library of distributed grid services controlled by a workflow system, compliant with standards emerging from the grid computing, web services, and virtual observatory communities. This new technology is being used to find high redshift quasars, study peculiar variable objects, search for transients in real time, and fit SDSS QSO spectra to measure black hole masses. Grist services are also a component of the ``hyperatlas'' project to serve high-resolution multi-wavelength imagery over the Internet. In support of these science and outreach objectives, the Grist framework will provide the enabling fabric to tie together distributed grid services in the areas of data access, federation, mining, subsetting, source extraction, image mosaicking, statistics, and visualization.
P1-3-8.tex
%APN3_PROCEEDINGS_FORM%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% TEMPLATE.TEX -- APN3 (2003) ASP Conference Proceedings template.
%
% Derived from ADASS VIII (98) ASP Conference Proceedings template
% Updated by N. Manset for ADASS IX (99), F. Primini for ADASS 2000,
% D.Bohlender for ADASS 2001, and H. Payne for ADASS XII and LaTeX2e.
%
% Use this template to create your proceedings paper in LaTeX format
% by following the instructions given below.  Much of the input will
% be enclosed by braces (i.e., { }).  The percent sign, "%", denotes
% the start of a comment; text after it will be ignored by LaTeX.  
% You might also notice in some of the examples below the use of "\ "
% after a period; this prevents LaTeX from interpreting the period as
% the end of a sentence and putting extra space after it.  
% 
% You should check your paper by processing it with LaTeX.  For
% details about how to run LaTeX as well as how to print out the User
% Guide, consult the README file.  You should also consult the sample
% LaTeX papers, sample1.tex and sample2.tex, for examples of including
% figures, html links, special symbols, and other advanced features.
%
% If you do not have access to the LaTeX software or a laser printer
% at your site, you can still prepare your paper following the
% instructions in the User Guide.  In such cases, the editors will
% process the file and make any necessary editorial adjustments.
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
\documentclass[11pt,twoside]{article}  % Leave intact
\usepackage{adassconf}
\usepackage{graphicx}

% If you have the old LaTeX 2.09, and not the current LaTeX2e, comment
% out the \documentclass and \usepackage lines above and uncomment
% the following:

%\documentstyle[11pt,twoside,adassconf]{article}

\begin{document}   % Leave intact

%-----------------------------------------------------------------------
%			    Paper ID Code
%-----------------------------------------------------------------------
% Enter the proper paper identification code.  The ID code for your
% paper is the session number associated with your presentation as
% published in the official conference proceedings.  You can           
% find this number locating your abstract in the printed proceedings
% that you received at the meeting or on-line at the conference web
% site; the ID code is the letter/number sequence proceeding the title 
% of your presentation. 
%
% This will not appear in your paper; however, it allows different
% papers in the proceedings to cross-reference each other.  Note that
% you should only have one \paperID, and it should not include a
% trailing period.
%
% EXAMPLE: \paperID{O4-1}
% EXAMPLE: \paperID{P7-7}
%

\paperID{P1.3.8}

%-----------------------------------------------------------------------
%		            Paper Title 
%-----------------------------------------------------------------------
% Enter the title of the paper.
%
% EXAMPLE: \title{A Breakthrough in Astronomical Software Development}
% 
% If your title is so long as to fill the page header when you print it,
% then please supply a short form as a \titlemark.
%
% EXAMPLE: 
%  \title{Rapid Development for Distributed Computing, with Implications
%         for the Virtual Observatory}
%  \titlemark{Rapid Development for Distributed Computing}
%

\title{Grist: Grid-based Data Mining for Astronomy}
%\titlemark{ }

%-----------------------------------------------------------------------
%		          Authors of Paper
%-----------------------------------------------------------------------
% Enter the authors followed by their affiliations.  The \author and
% \affil commands may appear multiple times as necessary (see example
% below).  List each author by giving the first name or initials first
% followed by the last name.  Authors with the same affiliations
% should grouped together. 
%
% EXAMPLE: \author{Raymond Plante, Doug Roberts, 
%                  R.\ M.\ Crutcher\altaffilmark{1}}
%          \affil{National Center for Supercomputing Applications, 
%                 University of Illinois Urbana-Champaign, Urbana, IL
%                 61801}
%          \author{Tom Troland}
%          \affil{University of Kentucky}
%
%          \altaffiltext{1}{Astronomy Department, UIUC}
%
% In this example, the first three authors, "Plante", "Roberts", and
% "Crutcher" are affiliated with "NCSA".  "Crutcher" has an alternate 
% affiliation with the "Astronomy Department".  The fourth author,
% "Troland", is affiliated with "University of Kentucky"

\author{Joseph C. Jacob, Daniel S. Katz, Craig D. Miller, Harshpreet Walia}
\affil{Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA 91109-8099}

\author{Roy Williams, S. George Djorgovski, Matthew J. Graham, Ashish Mahabal}
\affil{California Institute of Technology, Pasadena, CA 91125}

\author{Jogesh Babu, Daniel E. Vanden Berk}
\affil{The Pennsylvania State University, University Park, PA, 16802}

\author{Robert Nichol}
\affil{ICG, University of Portsmouth, PO1 2EG, UK}

%-----------------------------------------------------------------------
%			 Contact Information
%-----------------------------------------------------------------------
% This information will not appear in the paper but will be used by
% the editors in case you need to be contacted concerning your
% submission.  Enter your name as the contact along with your email
% address.
% 
% EXAMPLE:  \contact{Dennis Crabtree}
%           \email{crabtree@cfht.hawaii.edu}
%

\contact{Joseph Jacob}
\email{Joseph.Jacob@jpl.nasa.gov}

%-----------------------------------------------------------------------
%		      Author Index Specification
%-----------------------------------------------------------------------
% Specify how each author name should appear in the author index.  The 
% \paindex{ } should be used to indicate the primary author, and the
% \aindex for all other co-authors.  You MUST use the following
% syntax: 
%
% SYNTAX:  \aindex{Lastname, F. M.}
% 
% where F is the first initial and M is the second initial (if
% used).  This guarantees that authors that appear in multiple papers
% will appear only once in the author index.  
%
% EXAMPLE: \paindex{Crabtree, D.}
%          \aindex{Manset, N.}        
%          \aindex{Veillet, C.}        
%
% NOTE: this information is also used to build the author list that
% appears in the table of contents.  Authors will be listed in the order
% of the \paindex and \aindex commmands.
%

\paindex{Jacob, J. C.}
\aindex{Williams, R.}
\aindex{Babu, J.}
\aindex{Djorgovski, S. G.}
\aindex{Graham, M. J.}
\aindex{Katz, D. S.}
\aindex{Mahabal, A.}
\aindex{Miller, C. D.}
\aindex{Nichol, R.}
\aindex{Vanden Berk, D. E.}
\aindex{Walia, H.}

%-----------------------------------------------------------------------
%		      Author list for page header	
%-----------------------------------------------------------------------
% Please supply a list of author last names for the page header. in
% one of these formats:
%
% EXAMPLES:
% \authormark{Lastname}
% \authormark{Lastname1 \& Lastname2}
% \authormark{Lastname1, Lastname2, ... \& LastnameN}
% \authormark{Lastname et al.}
%
% Use the "et al." form in the case of seven or more authors, or if
% the preferred form is too long to fit in the header.

\authormark{Jacob, Williams, et al.}

%-----------------------------------------------------------------------
%			Subject Index keywords
%-----------------------------------------------------------------------
% Enter a comma separated list of up to 6 keywords describing your
% paper.  These will NOT be printed as part of your paper; however,
% they will be used to generate the subject index for the proceedings.
% There is no standard list; however, you can consult the indices
% for past proceedings (http://adass.org/adass/proceedings/).
%
% EXAMPLE:  \keywords{visualization, astronomy: radio, parallel
%                     computing, AIPS++, Galactic Center}
%
% In this example, the author noticed that "radio astronomy" appeared
% in the ADASS VII Index as "astronomy" being the major keyword and
% "radio" as the minor keyword.  The colon is used to introduce another
% level into the index.

\keywords{Grid services, Data mining, Astronomy, Virtual observatory,
Quasar search, Transient search}

%-----------------------------------------------------------------------
%			       Abstract
%-----------------------------------------------------------------------
% Type abstract in the space below.  Consult the User Guide and Latex
% Information file for a list of supported macros (e.g. for typesetting 
% special symbols). Do not leave a blank line between \begin{abstract} 
% and the start of your text.

\begin{abstract}          % Leave intact
% Place the text of your abstract here - NO BLANK LINES
The Grist project is developing a grid-technology based system
as a research environment for astronomy with massive and
complex datasets.  This knowledge extraction system
will consist of a library of distributed grid services controlled by a
workflow system, compliant with standards emerging from the grid
computing, web services, and virtual observatory communities.  This
new technology is being used to find high redshift quasars,
study peculiar variable objects, search for transients in
real time, and fit SDSS QSO spectra to measure black hole masses.
Grist services are also a component of the
``hyperatlas'' project to serve high-resolution multi-wavelength
imagery over the Internet.  In support of these science and outreach 
objectives, 
the Grist framework will provide the enabling fabric to tie together
distributed grid services in the areas of data access, federation,
mining, subsetting, source extraction, image mosaicking,
statistics, and visualization.
\end{abstract}

%-----------------------------------------------------------------------
%			      Main Body
%-----------------------------------------------------------------------
% Place the text for the main body of the paper here.  You should use
% the \section command to label the various sections; use of
% \subsection is optional.  Significant words in section titles should
% be capitalized.  Sections and subsections will be numbered
% automatically. 
%
% EXAMPLE:  \section{Introduction}
%           ...
%           \subsection{Our View of the World}
%           ...
%           \section{A New Approach}
%
% It is recommended that you look at the sample papers, sample1.tex
% and sample2.tex, for examples for formatting references, footnotes,
% figures, equations, html links, lists, and other special features.  

\section{Overview}

The Grist\footnote{Part of this research was carried out at the Jet Propulsion
Laboratory, California Institute of Technology, and was sponsored by
the National Science Foundation through an agreement with the National
Aeronautics and Space Administration.} project 
(\htmladdURL{http://grist.caltech.edu/}) is enabling 
astronomers and the public to interact with the grid
projects that are being constructed worldwide, and bring to flower the
promise of easy, powerful, distributed computing. Our objectives are
to understand the role of service-oriented architectures in
astronomical research, to bring the astronomical community to the grid
-- particularly TeraGrid, -- and to work with the National Virtual
Observatory (NVO) to build a library of compute-based web services.

The scientific motivation for Grist derives from creation and mining
of wide-area federated images, catalogs, and spectra. An astronomical
image collection may include multiple pixel layers covering the same
region on the sky, with each layer representing a different waveband,
time, instrument, observing condition, etc. The data analysis should
combine these multiple observations into a unified
understanding of the physical processes in the Universe. The familiar
way to do this is to
cross-match source lists extracted from different images. However, there is
growing interest in another method of federating images that
reprojects each image to a common set of pixel planes, then stacks
images and detects sources therein. While this has been done for
years for small pointing fields, we are using the TeraGrid to perform
this processing over wide areas of the sky in a systematic way, using
\htmladdnormallinkfoot{Palomar-Quest}{http://www.astro.caltech.edu/pq/} 
(PQ) survey data.  We expect this ``hyperatlas'' approach 
will enable us
to identify much fainter sources than can be detected in any individual
image; to detect unusual objects such as transients; and to deeply
compare (e.g., using principal component analysis) the large surveys
such as SDSS, 2MASS, DPOSS, etc. (Williams et al. 2003).

Grist is helping to build an image-federation pipeline for the
Palomar-Quest synoptic sky survey (Djorgovski et al. 2004), 
with the objectives of mining PQ data
to find high redshift quasars, to study peculiar variable
objects, and to search for transients in real-time (Mahabal et al. 2004).
Our PQ processing pipeline will use the TeraGrid for processing and will
comply with widely-accepted data formats and 
protocols supported by the VO community.

\section{Service-Oriented Architectures for Astronomy}
\label{sec:workflow}

The Grist project is building web and grid services as well as the
enabling workflow fabric to tie together these distributed services in
the areas of data federation, mining, source extraction, image
mosaicking, coordinate transformations, data subsetting, statistics
-- histograms, kernel density estimation, and R language utilities
exposed by \htmladdnormallinkfoot{VOStatistics}{http://vostat.org/} services 
(Graham et al. 2004), -- and visualization.  Composing multiple 
services into a distributed workflow architecture, 
as illustrated in Figure~\ref{fig:workflow}, with domain experts in different
areas deploying and exposing their own services, has a number of distinct 
advantages, including:
\begin{itemize}
\item Proprietary algorithms can be made available to end users without
the need to distribute the underlying software.
\item Software updates done on the server are immediately available to all 
users.
\item A particular service can be used in different ways as a component of
multiple workflows.
\item A service may be deployed close to the data source, for efficiency.
\end{itemize}
Interactive deployment and control of these distributed services will
be provided from a workflow manager. We expect to use NVO services for
data access -- images, catalogs, and spectra -- as well as the NVO
registry for service discovery.

\begin{figure}[hbt] 
\plotone{P1-3-8_1.eps}
\caption{Grist will deploy a library of interoperable services, which
may be composed in different ways for astronomical data mining
(e.g., two distinct workflows are indicated by the solid and dashed
arrows).}
\label{fig:workflow} 
\end{figure}

\section{Graduated Security}

As described in Section~\ref{sec:workflow}, 
much of the pipeline and mining software for Grist will be built in
the form of web services. One of the reasons for building services is
to be able to use them from a thin client, such as 
a web browser. However, for such services to be able to
process private data or use high-end computing, there must be strong
authentication of the user. The VO and Grid communities are converging
around the idea of X.509 certificates as a suitable credential for
such authentication. However, most astronomers do not have such a
certificate, and we don't want to make them go through the trouble of
getting one unless it is truly necessary. Therefore, we are building
services with ``graduated security'', meaning not only that small
requests on public data are available anonymously and simply, but also
that large requests on private data can be serviced through the same
interface. However in the latter case, a certificate is
necessary. Thus the service ``proves its usefulness'' with a simple
learning curve, but requires a credential to be used at
full-strength (see illustration in Figure~\ref{fig:grad_sec}).

\begin{figure}[hbt] 
\plotone{P1-3-8_2.eps}
\caption{``Graduated security'' will shorten the hurdles that stand in the way
of scientists who would like to take advantage of the power of computational 
grids for their research.}
\label{fig:grad_sec} 
\end{figure}

\section{Palomar-Quest Data Mining}

A key science-driven workflow we are constructing is illustrated in
the schematic in Figure~\ref{fig:pq-pipeline}.  The primary objectives
are to search for high redshift quasars and optical transients in data
from the Palomar-Quest sky survey.  The pipeline begins by federating
multiwavelength datasets, and
matching objects detected with the {\em z} filter with catalogs at
other frequencies.  Cluster analysis performed on the resulting
color-color plots (e.g., {\em i-z} vs.\ {\em z-J}) yield new quasar
candidates, and outliers may indicate the presence of other objects of
interest.

Single epoch transients are indicated by objects that are detected in one 
filter but not others. An object that is detected in the
reddest filter is of special interest since it could be a
highly obscured object or a high redshift quasar.
For multi-epoch transient search,
illustrated in the lower part of Figure~\ref{fig:pq-pipeline}, we
compare new data with a database of past epochs to detect new transients
or other variable objects.

\begin{figure}[hbt] 
\plotone{P1-3-8_3.eps}
\caption{A schematic pipeline to look for quasars, transients and
other variables. Combining multi-filter information with multi-epoch
datasets through a set of well established techniques will yield a
rich set of astronomically interesting objects.}
\label{fig:pq-pipeline} 
\end{figure}

As described above, a
primary objective of the PQ survey is the fast discovery of new types of
transient sources by comparing data taken at different times. Such
transients should be immediately re-observed to get maximum scientific
impact, so we are experimenting with ``dawn processing'' on the
TeraGrid, meaning that data is streamed from the telescope to the
compute facility as it is taken (rather than days later). The pipeline
itself is being built with streaming protocols 
so that unknown transients (e.g., newly identified
variables or asteroids) can be examined within hours of
observation with a view to broadcasting an email alert to interested parties.

\section{Summary}

Grist is developing a library of interoperable grid services for
astronomical data mining on the TeraGrid, compliant with Grid and VO
data formats, standards, and protocols.  For ease of use, Grist
services are built with graduated security, requiring no more formal
authentication than is appropriate for a given level of usage.  Grist
technology is part of a Palomar-Quest data processing pipeline, under 
construction, to search for high red-shift quasars and optical transients.
More information on Grist can be found on our project web site at 
\htmladdURL{http://grist.caltech.edu/}.
%-----------------------------------------------------------------------
%			      References
%-----------------------------------------------------------------------
% List your references below within the reference environment
% (i.e. between the \begin{references} and \end{references} tags).
% Each new reference should begin with a \reference command which sets
% up the proper indentation.  Observe the following order when listing
% bibliographical information for each reference:  author name(s),
% publication year, journal name, volume, and page number for
% articles.  Note that many journal names are available as macros; see
% the User Guide listing "macro-ized" journals.   
%
% EXAMPLE:  \reference Hagiwara, K., \& Zeppenfeld, D.\  1986, 
%                Nucl.Phys., 274, 1
%           \reference H\'enon, M.\  1961, Ann.d'Ap., 24, 369
%           \reference King, I.\ R.\  1966, \aj, 71, 276
%           \reference King, I.\ R.\  1975, in Dynamics of Stellar 
%                Systems, ed.\ A.\ Hayli (Dordrecht: Reidel), 99
%           \reference Tody, D.\  1998, \adassvii, 146
%           \reference Zacharias, N.\ \& Zacharias, M.\ 2003,
%                \adassxii, \paperref{P7.6}
% 
% Note the following tricks used in the example above:
%
%   o  \& is used to format an ampersand symbol (&).
%   o  \'e puts an accent agu over the letter e.  See the User Guide
%      and the sample files for details on formatting special
%      characters.  
%   o  "\ " after a period prevents LaTeX from interpreting the period 
%      as an end of a sentence.
%   o  \aj is a macro that expands to "Astron. J."  See the User Guide
%      for a full list of journal macros
%   o  \adassvii is a macro that expands to the full title, editor,
%      and publishing information for the ADASS VII conference
%      proceedings.  Such macros are defined for ADASS conferences I
%      through XI.
%   o  When referencing a paper in the current volume, use the
%      \adassxii and \paperref macros.  The argument to \paperref is
%      the paper ID code for the paper you are referencing.  See the 
%      note in the "Paper ID Code" section above for details on how to 
%      determine the paper ID code for the paper you reference.  
%
\begin{references}

\reference Djorgovski, S.\ G., et al. 2004, \baas, 36, 805

\reference Graham, M.\ J., et al. 2004, \adassxiv, \paperref{P1.2.7}

\reference Mahabal, A., et al. 2004, \adassxiv, \paperref{P2.2.7}

\reference Williams, R.\ D., et al. 2004, \adassxii, \paperref{O4-3a}

\end{references}

% Do not place any material after the references section

\end{document}  % Leave intact
