A Case for Data Commons\protect\\ {\Large Towards Data Science as a Service
Robert L. Grossman
Allison Heath
Mark Murphy
Maria Patterson \protect
University of Chicago \protect
\protect
Walt Wells \protect
Open Commons Consortium
osdc-cise-16-v1.tex
\documentclass{article}[11pt]

\newcommand{\ph}[1]{\medbreak \noindent {\bf #1}}

\usepackage{graphicx} % Required for including images

\title{A Case for Data Commons\protect\\
{\Large Towards Data Science as a Service}}


\author{Robert L. Grossman \and Allison Heath \and Mark Murphy \and Maria Patterson
\protect\\
University of Chicago
\protect\\
\protect\\
Walt Wells
\protect\\
Open Commons Consortium
}
\date{January 1, 2016}

% \date{\today}


\begin{document}
\maketitle

\section*{Abstract}

As the amount of scientific data continues to grow at ever faster rates, the research community is increasingly in need of flexible computational infrastructure that can support the entirety of the data science lifecycle, including long-term data storage, data exploration and discovery services, and compute capabilities to support data analysis and re-analysis, as new data are added and as scientific pipelines are refined.  We describe our experience developing data commons-- interoperable infrastructure that co-locates data, storage, and compute with common analysis tools-- and present several cases studies.  Across these case studies, several common requirements emerge, including the need for persistent digital identifier and metadata services, APIs, data portability, pay for compute capabilities, and data peering agreements between data commons.  Though many challenges, including sustainability and developing appropriate standards remain, interoperable data commons bring us one step closer to effective Data Science as Service for the scientific research community.

\ph{Keywords:} data commons, software as services, science as a service,
data as a service, cloud computing


\section{Introduction}

With the amount of available scientific data far larger than the ability of the research community to analyze it, there is a critical need for new algorithms, software applications, software services, and cyber-infrastructure to support data throughout its lifecycle in data science.  In this paper, we make a case for the role of data commons in meeting this need. We describe the design and architecture of several data commons that we have developed and operated for the research community in conjunction with the Open Science Data Cloud (OSDC), a multi-petabyte science cloud that the not-for-profit Open Commons Consortium (OCC) has managed and operated since 2009 \cite{Grossman:SC12}.

One of the distinguishing characteristics of the OSDC is that it interoperates with a data commons containing over 1 PB of public research data through a service-based architecture.  This is an example of what is sometimes called ``Data as a Service,'' which play an important role in some Science as as Service frameworks.

There are at least two definitions for {\em Science as a Service}.  The first is analogous to the Software as a Service \cite{Mell:2011} model in which instead of managing data and software locally using your own storage and computing resources, one uses the storage, computing, and software services offered by a Cloud Service Provider (CSP).  With this approach, instead of setting up your own storage and computing infrastructure and installing the required software, a scientist uploads data to a CSP and uses pre-installed software for data analysis.  Note that a trained scientist is still required to run the software and analyze the data.

Science as a Service can also refer more generally to a service model that relaxes the requirement of needing a trained scientist to process and analyze the data.  With this service model, specific software and analysis tools are available for specific types of scientific data.  Data are uploaded to the Science as a Service Provider, processed using the appropriate pipelines, and then made available to the researcher for further analysis if required.   

Obviously these two definitions are closely connected in that a scientist can set up the required Science as a Service framework as in the first definition so that less trained technicians can use the service to process their research data as in the second definition.  By and large, we focus on the first definition in this paper.

We close this section by discussing Science as a Service more generally and how data commons fit in.
There are various Science as a Service frameworks, including variants of the types of clouds formalized by NIST in \cite{Mell:2011} (Infrastructure as a Service, Platform as a Service, and Software as a Service), as well as some more specialized services that are relevant for data science (data science support services and data commons).  These Science as a Service frameworks include the following:

\begin{itemize}

\item Data science infrastructure and platform services, in which virtual machines, containers, or platform environments containing commonly used applications, tools, services, and datasets are made available to researchers. The OSDC is an example of this service model \cite{Grossman:SC12}.

\item Data science Software as a Service, in which data are uploaded and processed by one or more pipelines, with the results being downloaded or stored in a cloud. There are general purpose platforms offering data science as a service, such as Agave \cite{dooley2012agave}, as well as more specialized services, such as those designed to process genomics data.  For example, there are several commercial companies that enable a researcher to upload genomic sequence data, select a bioinformatics pipeline, and download the processed data.  

\item Data science support services, including data storage services, data sharing services, data transfer services, data collaboration services, etc.  A prominent example of this model is Globus \cite{foster2011globus}.

\item Data commons, in which data, data science computing infrastructure, and data science applications and services are co-located.  We describe data commons in more detail in Section~\ref{section:data-commons} below.

\end{itemize}

The paper is organized as follows.  In Section~\ref{data-commons-section}, we introduce data commons. In Section~\ref{section:osdc}, we introduce the Open Science Data Cloud and some of the data commons associated with it.  In Section~\ref{section:design}, we describe the design and architecture of the OSDC, which includes the services required to set up, manage, and operate data commons.  Section~\ref{section:case-studies} contains two case studies of OSDC data commons, and Section~\ref{section:discussion} contains some challenges related to data commons.



\section{Data Commons}
\label{data-commons-section}

\subsection{What is a Data Commons?}
\label{section:data-commons}

By a {\em data commons} we mean cyber-infrastructure that co-locates data, storage, and computing infrastructure with commonly used tools for analyzing and sharing data to create an interoperable resource for the research community.

In the discussion below, we distinguish between several stakeholders that are involved in data commons: 1) the {\em data commons service provider (DCSP)}, the entity operating the data commons; 2) the {\em data contributor (DC)}, the individual or organization providing the data to the DCSP; and 3) the {\em data user (DU)}, the researcher or organization that accesses the data.  Note that often there is a fourth stakeholder: the DCSP associated with the researcher accessing the data.  In general, there will be an agreement, often called the Data Contributors Agreement (DCA), governing the terms by which the data is managed by the DCSP and the researchers accessing the data, and an agreement, often called the Data Access Agreement (DAA), governing the terms of any researcher that accesses the data.

As we describe in more detail below, we have built several data commons since 2009.  Based upon this experience we have identified six main requirements that, if followed, would enable data commons to interoperate with other data commons, with science clouds \cite{Grossman:SC12}, and with other cyber-infrastructure supporting Science as a Service.

\ph{Requirement 1: Permanent Digital IDs.} 
The first requirement is that the data commons must have a digital ID service and that datasets in the data commons must have permanent, persistent digital IDs.  Digital IDs are described in more detail in Section~\ref{section:core-services}.  Associated with digital IDs are i) access controls specifying who can access the data and ii) metadata specifying additional information about the data (see Requirement 2).   Part of this requirement is that data can be accessed from the data commons through an API by specifying its digital ID.

\ph{Requirement 2: Permanent Metadata.}
The second requirement is that there is a metadata service that for each digital ID returns the associated metadata.  Since the metadata can be indexed, this provides a basic mechanism for the data to be discoverable. 

\ph{Requirement 3: API-based Access.}
The third requirement is that data can be accessed by an application programming interface (API), not just by browsing through a portal. Part of this requirement is that there is a metadata service that can be queried that returns a list of digital IDs that can then be retrieved via the API.  For those data commons that contain controlled access data, another component of the requirement is that there is an authentication and authorization service so that users can first be authenticated and the data commons can check whether they are authorized to have access to the data.

\ph{Requirement 4: Data Portability.}
The fourth main requirement is that data be portable in the sense that a dataset that has been contributed to a data commons can be transported to another data commons so that in the future it is hosted by the second data commons.  In general, if data access is through digital IDs (versus referencing the physical location of data), then software that references data should not have to be changed when data is re-hosted by a second data commons.

\ph{Requirement 5: Data Peering.}  By data peering, we mean an agreement between two data commons service providers to transfer data {\em at no cost} so that a researcher at data commons 1 can access data stored at data commons 2.  In other words, the two data commons agree to transport research data between them with no access charges, no egress charges, and no ingress charges. 

\ph{Requirement 6: Pay for Compute.}  The final requirement co-located with the data commons is computing infrastructure that is available to researchers.  Since, in practice, researchers' demand for computing resources is larger than the available computing resources, computing resources must be rationed, either through allocations or by charging for the use of computing resources.  Notice that there is an asymmetry in how a data commons treats the storage and computing infrastructure. When data are accepted into a data commons, there is a commitment to store the data and make it available for a certain period of time, often indefinitely.   In other words, the rationing decision for initial storage is made when data are accepted.  In contrast, computing over data in a data commons is rationed in an on-going fashion, as is the working storage, and the storage required for derived data products, either by providing computing and storage allocations for this purpose or by charging for computing and storage.  For simplicity, we refer to this requirement as ``pay for computing," even though the model is more complicated than that.

Although very important for many applications, we view other services, such as services providing data provenance \cite{simmhan2005survey}, data replication \cite{chervenak2008wide}, and data collaboration \cite{alameda2007open}, as optional additional services, and not as core services.


\section{Open Science Data Cloud and OCC Data Commons}
\label{section:osdc}

\subsection{About the Open Science Data Cloud and the OCC}
The Open Science Data Cloud (OSDC) is a multi-petabyte science cloud that serves the research community by co-locating a multidisciplinary data commons containing 
approximately 1 PB of scientific data with cloud-based computing, high
performance data transport services, and virtual machine images and shareable snapshots containing common data analysis pipelines and tools.   

The OSDC is designed to provide a long term persistent home for scientific data, as well as a platform for data intensive science allowing new types of data
intensive algorithms to be developed, tested, and used over large sets of heterogeneous scientific data. Recently, OSDC researchers have logged about 2 million 
core hours each month, which translates to over \$800,000 worth of cloud 
computing services if purchased through Amazon's AWS public cloud.
This is over 12,000 core hours per user or, a 16 core machine continuously used by each researcher on average.  

OSDC researchers used a total of over 18 million core hours in 2015.  
We currently target operating OSDC computing resources at approximately 85\% of capacity and storage resources at 80\% of capacity.   Given these constraints, we determine how many researchers to support and what size allocations to provide them.  Since the OSDC specializes in supporting data intensive research projects, we have chosen to target researchers who need larger scale resources (relative to our total capacity) for data intensive science. In other words, rather than support more researchers with smaller allocations, we support fewer researchers with larger allocations.   Table \ref{tab:users} shows the number of times researchers exceeded the indicated number of core hours in a single month during 2015.  

The OSDC is developed and operated by 
the Open Commons Consortium (OCC), which is a 501(c)(3) not-for-profit supporting the 
scientific community by operating data commons and cloud computing infrastructure to 
support scientific, environmental, medical, and healthcare related research.  
OCC Members and Partners include universities (e.g., University of Chicago,
Northwestern University, University of Michigan); companies (e.g., Yahoo!, Cisco,
Infoblox); U.S. government agencies and national laboratories (e.g., NASA, NOAA); 
and international partners (e.g., Edinburgh University, University of
Amsterdam, AIST in Japan).  The OSDC is a joint project with the University of Chicago, which provides the data center used by the OSDC.   Much of the support for
the OSDC came from the Moore Foundation and from corporate donations.

%------------------------------------------------
% TABLE: DATA INTENSIVE USERS
%------------------------------------------------
\begin{table}[!ht]
\caption{Data intensive users supported by the OSDC}
\begin{center}
\begin{tabular}{|l | r|}
\hline
\# core hours during month & \# of users \\
\hline
% 10,000 & 435  \\ \hline
20,000 & 120  \\ \hline
50,000 &  34   \\ \hline
100,000 & 23  \\ \hline
200,000 &  5  \\ 
\hline 
\end{tabular}
\end{center}
\label{tab:users}
\footnotesize{The estimated cost of 100,000 core hours on a commercial cloud service provider like AWS is \$40,000 per month.}
\end{table}


\subsection{The OSDC Community}

The OSDC has a wide-reaching, multi-campus, multi-institutional,
interdisciplinary user base and has supported over 760 research
projects since its inception.  In November 2015, 186
researchers used the OSDC, out of a total of 470 allocation recipients
from 54 universities and research organizations in 14 countries who
were active sometime during 2015.  The most
computational intensive group projects in 2015 included
projects around biological sciences and genomics research, analysis of
earth science satellite imagery data, analysis of text data in historical and scientific
literature, and a computationally intensive project in sociology.  
%(As of February 2015, 47\% of active users work on research led by PIs who
%are funded with currently active NSF grants. \textit{I'm not sure about an updated
%number for this})

% Demand for computational resources from the research community is growing.
% Below we show the utilization of available compute core hours logged by OSDC users each month over time.
%------------------------------------------------
% FIGURE: computational resources growth
% this includes
%------------------------------------------------
% \begin{figure*}[!ht] 
% \centering
% \includegraphics[scale = .08]{figs/cores-used-available-perc-sullivan-dec15.png}
% \caption{The demand for computational resources on OSDC growing to max capacity on public resource "Sullivan". (Compute core usage over 80\% results in performance issues.) }
%\end{figure*}

% The number of active users and the amount of computational resources logged continued to grow until available resources maxed out, and we had to be more aggressive in our resource allocation policies.  
% We are able to currently support over 180 active users per month at current usage levels.  Subscription rates are extremely high for the public ``Sullivan'' resource, and we commonly have to turn away meritorious applicants. On average over the last year, 83\% of available cores are in use every month by the community on Sullivan.  Since inception, 827 individual resource allocations have been granted. %Additionally, starting in 2014, we started getting regular requests from the research community to provide a home to host and publicly serve their data.  It is clear that the research community is searching for a solution to a need that OSDC is working to address.


%------------------------------------------------
% TABLE: DATASETS
%------------------------------------------------
%\begin{table}[!ht]
%\caption{Summary of datasets served to the public and directly accessible to OSDC users}
% \begin{center}
% \begin{tabular}{|p{1.75in} |p{3.0in}|p{.75in}|}
% \hline
% {\bf Discipline} & {\bf Data included (e.g.)} & {\bf Size (TB)} \\ \hline
% Biological Sciences         &  NCBI Datasets, 1000 Genomes, modEncode  & 656 \\ \hline
% Earth Science, Physical Sciences, Astronomy & NOAA National Climatic Data Center, satellite data %from EO-1, Global Land Survey, Sloan Digital Sky Survey & 240 \\ \hline
% Social and Urban Sciences   & City of Chicago data, Google Books N-grams, Million Song Dataset& 19 \\ \hline
% \textbf{\emph{Total}}      & 25+ large datasets in different fields   & 915 \\
% \hline
%\end{tabular}
%\end{center}
%\label{tab:datasets}
% \end{table}

\subsection{OCC Data Commons}

\ph{OSDC Data Commons.}
We introduced our first data commons in 2009.  It currently 
holds approximately 800 TB of public open access research data, including earth science data, biological data, social science data, and digital humanities data.

\ph{Matsu Data Commons.}
The Open Cloud Consortium has collaborated with NASA since 2009 in
Project Matsu.  Project Matsu's data commons contains 6 years of EO-1
data with new data added daily, as well as selected datasets from other NASA satellites,
including MODIS and the Landsat Global Land Surveys.  

\ph{The OCC NOAA Data Commons.}
In April, 2015, NOAA announced 5 data
alliance partnerships that would have broad access to NOAA's data and
help make these data more accessible to the public.  The NOAA Data Alliance
Partners are Amazon, Google, IBM, Microsoft, and the OCC.  
Currently, only a small fraction of the over 20 PB of data that NOAA has available
in its archives is available to the public.   The NOAA Data Alliance
Partners have broader access to the data in the NOAA repositories.
The focus of the OCC data alliance is to work with the environmental research community to build 
an environmental data commons.   Currently the OCC NOAA Data Commons contains NEXRAD data.
Additional datasets will be added in 2016.


\ph{National Cancer Institute's Genomic Data Commons (GDC).}
Through a contract between NCI and the University of Chicago and in
collaboration with the OCC, we have developed a data commons for
cancer data called the Genomic Data Commons (GDC).   The GDC contains
genomic data and the associated clinical data from NCI
funded projects.  Currently the GDC contains about 2 PB of data, but this is
expected to grow rapidly over the next few years.  The GDC is in beta
testing now (January, 2016) and will be accessible to the public in May, 2016.


\ph{Bionimbus Protected Data Cloud.}
We also operate two private cloud computing platforms that are designed
to hold human genomic data and other sensitive biomedical data.  These
two clouds contain a variety of sensitive controlled access biomedical data that we make
available to the research community following the requirements of
the relevant data access committees.

\ph{Common software stack.}
The core software stack for the various data commons and
clouds described above is all open source.  Many of the components are
developed by third parties, but some key services described below are
developed and maintained by the OCC and other working groups.  Although there are some
differences between them, we try to keep the software stack as close
as possible across the various data commons. 
In practice, as we develop new
versions of the basic software stack, it usually takes a year or so
until the changes can percolate throughout our entire infrastructure.



\section{Design and Architecture}
\label{section:design}

\subsection{OSDC Architecture}
The architecture of the OSDC is shown in Figure~\ref{figure:osdc-arch}.
We are currently transitioning from version 2 of the OSDC software stack \cite{Grossman:SC12} to version
3.  Both versions of the software stack are based upon OpenStack \cite{pepple2011deploying} for Infrastructure as a Service.  Version 2 uses
GlusterFS \cite{davies2013scale} for storage,  while version 3 uses Ceph \cite{weil2006ceph} for block and object storage.  

The OSDC has a portal called the Tukey Portal, which provides a front-end web portal 
interface for users to access, launch, and manage VMs and storage.
The Tukey Portal interfaces with the Tukey Middleware, which provides a secure authentication
layer and interface between various software stacks.
OSDC uses federated login for authentication so that academic institutions with InCommon, 
CANARIE, or the UK Federation can use those credentials. We have worked with 145 
academic universities and research institutions so far to release the appropriate attributes
for authentication.  We also support Gmail and Yahoo logins, but only for approved projects, and when other authentication options are not available.

We instrument all of the resources that we operate so that we can meter and collect the data required for accounting and billing for each user. We use Salesforce.com,
one of the components of the OSDC that is not open source, to send out invoices.
Even when computing resources are allocated and no payment is required, we have found that receipt of these invoices promotes responsible usage of OSDC community resources.
% Users can also track their allocations through the web portal.
We also operate an interactive support ticketing system that tracks user support
requests and systems team responses for technical questions.
Collecting these data enables us to track usage statistics and to build a 
comprehensive assessment of how researchers use our services.

While adding to our resources, we have developed an infrastructure automation tool called Yates to simplify bringing up new computing, storage, and networking infrastructure. We also try to automate as much of the security required to operate the
OSDC as is practical.
% Our security automation application is loosely based upon OpenFISMA.

% ************ To Do ************
% mention: 
% OCM
% managed data
% do we make metadata service part of ID service or not -- need to be 
% consistent above and below
% ********** end to do **********


The core OSDC software stack is open source, enabling those interested to set up their own science cloud or data commons.  The core software stack consists of third party open source software, such as OpenStack and Ceph, and open source software developed by the OSDC community.  The latter is licensed under the open source Apache license.  The OSDC does use some proprietary software, such as Salesforce.com to do the accounting and billing, as was mentioned above.



% \begin{figure}
% \centering
% \includegraphics[scale=0.75]{figs/arch-v3.pdf}
% \caption{The OSDC architecture}
% \label{figure:osdc-arch}
% \end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.75]{figs/osdc-arch-16-v4.pdf}
\caption{The OSDC architecture}
\label{figure:osdc-arch}
\end{figure}


\subsection{OCC Digital ID and Metadata Services}
\label{section:core-services}

In this section, we describe the digital ID service and the associated metadata service used in the data commons.   The digital ID service is accessible via an API that generates digital IDs, assigns
key-value attributes to DIDs, and returns key-value attributes associated with digital IDs.
We have also developed a metadata service that is accessible
via an API and can assign and retrieve metadata associated with a digital ID. 
Users can also edit metadata associated with digital IDs if they have write access to the
digital ID.  These are described in more detail below.  Currently, due to different release schedules, there are some differences in the digital ID and metadata services between several of the data commons that we operate, but over time we plan to converge these services.


\ph{Persistent identifier strategies.} Although the necessity of assigning digital IDs to data is well
recognized  \cite{Mayernik2012data, Duerr2011utility}, there is not yet
a widely accepted service for this purpose, especially for large
datasets \cite{lagoze2014ced}.  This is in contrast to the generally accepted use of digital
object identifiers (DOI) or handles for referencing digital publications.
An alternative to a DOI is an ARK, which
is a Uniform Resource Locator (URL) that is a multi-purpose identifier for information objects of any type
\cite{kunze2003towards, Kunze:2006}.  
% 
In practice, DOIs or ARKs are generally used to assign IDs to datasets, with individual
communities sometimes developing their own IDs. 
% such as that the Global Alliance for Genomics and Health is proposing for genomic data. 
% 
DataCite is an international consortium that manages DOIs for datasets
and supports services for finding, accessing, and reusing data
\cite{Pollard2010making}.   There are also services such as EZID that
support both DOIs and ARKs \cite{starr2012collaborative}. 

Given the challenges the community is facing coming to a consensus
about which digital IDs to use, our approach has been to build an open
source digital ID service that can support multiple digital IDs and
that can scale to large datasets.  This approach also has the
advantage that we can deal with issues like ``suffix pass-through''
\cite{lagoze2014ced} and associating availability zones \cite{varia2010architecting} and support multiple access methods with the digital ID service that are helpful with petabyte-scale data archives.

\ph{Benefits of digital IDs.} From the viewpoint of researchers, the need for digital IDs associated with datasets is well appreciated \cite{ball2011cite, green2009we}.  Here we discuss some of the reasons that digital IDs are important for a data commons from an operational point of view.  

First, with digital IDs, data can be moved from one physical location or storage system within a data commons to another without the necessity of changing any code that references the data.  As the amount of data grows, moving data between zones within a data commons or between storage systems becomes more and more common and digital IDs allow this to take place without impacting researchers.   

Second, digital IDs are an important component of the data portability requirement.  More specifically, datasets can be moved between data commons, and, again, researchers do not need to change their code.
In practice, datasets can be migrated over time, with the digital IDs references updated as the migration proceeds.



\ph{Signpost digital ID service.}  Signpost is the digital ID service for the OSDC.  
%Digital IDs provide a simple, yet powerful, abstraction about how users obtain data. 
Instead of using a hardcoded URL, the primary way to access managed data 
via the OSDC is through a digital ID. Signpost is an implementation of this concept via JSON documents. 

%Signpost JSON documents look like the following:
%\begin{verbatim}
%{
%  did: “<digital id>”, 
%  rev: “<revision>”, 
%  urls: [<url>,...], 
%  acls: [<acl>,...], 
%  sha1: “sha1”
%}
%\end{verbatim}
%
%Here is some sample Python code for how Signpost could be used by a
%researcher to access data from the OSDC data commons
%given the Digital ID, in this case a UUID linked to a genome wide association study:

%\begin{verbatim}
%import requests
%import subprocess
%from urlparse import urlparse
%
%host_service = "http://signpost.service.consul/v0/did/"
%digital_id = "0007c79b-656d-4fa5-aa56-0b4c00f5995b"
%signpost_call = "".join([host_service,digital_id])
%resp_json = requests.get(signpost_call).json()
%parsed_url = urlparse(resp_json["urls"][0]) 
%destination = "/home/data/"
%cmd=["s3cmd","sync",parsed_url.path, destination]
%subprocess.call(cmd)
%\end{verbatim}

% A digital ID system for a data commons 
% requires consideration of balancing the needs of both data archiving for persistent storage, that is,
% assigning identifiers to unique pieces of information, and also active computation, that is, for 
% finding locations of data on a living system where data may be physically moved or updated.
The Signpost digital ID service integrates a mutable ID that is assigned to the data along with an immutable hash-based ID that is computed from the data.  Both IDs are accessible through a REST API interface.
With this approach, data contributors can may make updates to the data and retain the same ID, while the data commons service provider can use the hash-based ID to facilitate the management of the data.
In order to prevent unauthorized editing of digital IDs, an access control list (ACL) is kept by each
digital ID specifying the read/write permissions for different users and groups. 

The top layer utilizes \textit{user-defined identifiers}.  These are flexible, may be of any 
format, including Archival Resource Keys (ARKs) and Digital Object Identifiers (DOIs), and 
provide a layer of human readability.  These identifiers map to hashes of the identified
data objects.  
% This additionally allows for mutability by assigning different hashes and allows
% for reproducibility considerations.
The bottom layer utilizes \textit{hash-based identifiers}.  Hash-based identifiers guarantee 
immutability of the data, allow for identification of duplicated data via hash
collisions, and allow for verification upon retrieval.  These map to known locations of 
the identified data.

%The Signpost API has four essential functions applied to one or both
%of the connected layers:
%i) generate a new Signpost document (new digital ID); 
%ii) update the Signpost document;
%iii) retrieve the Signpost document; 
%iv) void/delete a Signpost document.  
%  other than resource locators.  However, 
% This is analogous to DNS in that anyone has permission to read a DNS record, but only the
% owner of the hostname is allowed to change the IP that it points to.
%  While not part of the current architecture design, if there
% becomes a requirement that read access (to the digital ID) is to be restricted, additional controls may be added to the Digital ID format.

Below, we show an implementation of the two-layer identifier service.
A researcher who has a human-readable identifier, for example, provided by a collaborating
researcher or publication or returned by a metadata search service can query Signpost to 
return the requested data.
Alternatively, a researcher who knows the exact hash of a piece of data (e.g., a scientist who is given a data file directly by a collaborator) can also query that way.

%In the following example, a curl request to the top layer of the identifier service
%queries the hash-based identifier layer (not exposed to the user) and then returns
%a list of urls to the data object identified by an ARK identifier to NOAA NEXRAD
%data in the OCC NOAA data commons.  The data were first uploaded to the S3-compatible object 
%storage. The OCC then assigned ARKs with its Name Assigning Authority Number corresponding
%to each object, and the data were then indexed via the two-layer service.

%%%  need to update the curl examples  %%%
\begin{verbatim}
curl http://<signpost>/alias/ark:/31807/DC0-7b2c1002-e3c4-41ea-8edc-8fcee4ff3f47
   {"hashes": { "md5": "1e24480435408b664b756be0822028a3" }, 
    "authority": "noaa-commons", 
    "metadata": <metadata-record-id>, 
    "name": "ark:/31807/DC0-7b2c1002-e3c4-41ea-8edc-8fcee4ff3f47", 
    "release": "public", 
    "rev": "e1407bdd", 
    "size": 45893621760, 
    "urls": [ "https://<osdc>/noaa-nexrad-l2/NWS_NEXRAD_NXL2DP_KDVN_201509_01.tar", 
       "https://<osdc>/noaa-nexrad-l2/NWS_NEXRAD_NXL2DP_KDVN_201509_02.tar", ... ] }
\end{verbatim}

In the next example, a curl request directly to the hash-based layer of the 
identifier service returns a list of urls to the data object.  

\begin{verbatim}
curl http://<signpost>/urls/?hash=md5:1e24480435408b664b756be0822028a3&size=45893621760
    { "hashes": { "md5": "1e24480435408b664b756be0822028a3" }, 
    "size": 45893621760, 
    "urls": [ "https://<osdc>/noaa-nexrad-l2/NWS_NEXRAD_NXL2DP_KDVN_201509_01.tar", 
       "https://<osdc>/noaa-nexrad-l2/NWS_NEXRAD_NXL2DP_KDVN_201509_02.tar", ... ] }
\end{verbatim}

In either case, the data can be moved and the urls updated without affecting any code that 
interfaces with data in the commons using the indexing service.  The concept is simple
and lightweight enough that it can be 
easily used in many existing environments.


\ph{OSDC metadata service.}
The OSDC metadata service is called Sightseer.  Sightseer allows users to
create, modify and access searchable JSON documents containing
metadata about digital IDs.  The primary data can be accessed using
Signpost and the digital ID.
At its core, Sightseer provides no restrictions on the JSON documents
it can store. However, it has the ability to specify metadata
types and associate them with JSON schemas. This helps prevent
unexpected errors in metadata with defined schemes. Sightseer has
similar abilities as Signpost to provide access control lists to
specify users that have write/read access to the specific JSON
document.

% \ph{OSDC download service.}
% The OSDC has several ways for downloading and transporting data.
%Users can also use standard command line tools such as curl and wget to download via HTTP(s). Initially, to obtain core functionality, we will just offer public data for download via the portal. 
%In Phase II, we will add login capabilities and the portal will be able to feed the authenticated user’s token to the GDC API so the user will be able to download protected data in the same way.
% In the main public data commons, data are available on both
% commodity internet connections and high speed Internet2 connections. 
% The public data commons are also accessible via a Globus endpoint \cite{allen2012software}.
% In addition to standard protocols for downloading data, we also
% support a number of open source UDT-based services.
% UDT (udt.sf.net) is an open source protocol designed for high performance
% transfers over wide area networks \cite{Grossman:CN2007}.
% For example, we support UDR, which is UDT enabled rsync, and  
% a command line download client based upon UDT, which uses a manifest file of
% Digital IDs generated from either the  OSDC web portal or the OSDC
% Query API.

% As we transition our current data commons from GlusterFS to Ceph-based
% object storage, we have also added support for HTTP(s),
% which allows users to download via the browser and standard
% command line tools such as curl and wget.  From a web portal, users
% perform queries until they have located the data they are
% interested in downloading. Once they click to download, the portal
% redirects the user to the download endpoint of the OSDC API which
% will then stream data via HTTP(s).  The transition from the OSDC
% version 2.0 architecture based upon GlusterFS to the OSDC version 3.0
% architecture based upon Ceph should take approximately a year.


%\begin{figure}[!ht] 
%\includegraphics[width=0.98\linewidth]{figs/architecture-dec15.png}
%\caption{OSDC design from the researcher's perspective, marked by
%orange dotted lines denoting access methods.}
%\end{figure}
% Architecture table



\section{Case Studies}
\label{section:case-studies}

\subsection{Matsu}
Project Matsu is an OCC data commons that has been operational since 2009.  
It is a collaboration between NASA and the OCC that is hosted by the University of Chicago, processes the data produced each day by NASA's Earth Observing-1
(EO-1) satellite, and makes a variety of data products available to the research community, including flood maps. The raw data, processed data, and data products are all available through OSDC. Project Matsu uses a framework called the OSDC Wheel to ingest raw data, process
and analyze it, and deliver reports with actionable information to the community in near real time \cite{mandl2013use}.

As part of Project Matsu, we host several focused analytic products with value-added data.
In Figure \ref{fig:flood dashboard} we show a screenshot from one of these focused analytic
products, the Project Matsu Namibia Flood Dashboard \cite{mandl2013use}.  
The Namibia Flood Dashboard was 
developed as a tool for aggregating and rapidly presenting data and sources of information 
about ground conditions, rainfall, and other hydrological information to 
citizens and decision makers in the flood prone areas of water basins in Namibia and 
the surrounding areas.  The tool features a bulletin system producing a short daily written
report, a geospatial data visualization display using Google Maps/Earth and OpenStreetMap, 
methods for retrieving NASA images in a region of interest, and analytics for projecting
flood potential using hydrological models.  The Namibia Flood Dashboard is an important
tool for developing better situational awareness and enabling fast decision making and
is a model for the types of focused analytics products made possible by co-locating 
related datasets with each other and with computational and analytic capabilities.  

%------------------------------------------------
% FIGURE: NAMIBIA FLOOD DASHBOARD
%------------------------------------------------
\begin{figure}[!ht] 
\centering
\includegraphics[scale = .25]{figs/flood-dashboard.pdf}
\caption{\footnotesize A screenshot of part of the Namibia Flood Dashboard from March 
14, 2014 showing water catchments (outlined and colored regions) and a one day flood 
potential forecast of the area from hydrological models using data from the Tropical 
Rainfall Measuring Mission (TRMM), a joint space mission between NASA and the
Japan Aerospace Exploration Agency (JAXA).
}
\label{fig:flood dashboard}
\end{figure}

% As a cloud service provider for the scientific community, OSDC researchers and the
% community can contribute to value-added data based on their specific research
% interests.  Users of the OSDC can build dashboards for their scientific
% communities which we can then host on the OSDC.  As we continue to grow the OSDC,
% we are working to develop methods to help scientists quickly make new 
% discoveries, including both APIs that make it easier to obtain and access relevant 
% data and data products and also analytic tools that make it easier to explore the data.

\subsection{Bionimbus}

The Bionimbus Protected Data Cloud is a petabyte scale private cloud and data commons that has been operational since March 13, 2013.  Since going online in 2013, Bionimbus has supported more than 152 allocation recipients from over 35 different projects at 29 different institutions. Each month, Bionimbus provides over 2.5 million core hours to researchers, which at standard Amazon AWS pricing would cost over \$500,000. 
One of the largest users of Bionimbus is The Cancer Genome Atlas (TCGA) / International Cancer Genome Consortium (ICGC) PanCancer Analysis of Whole Genomes working group (PCAWG).  PCAWG is currently undertaking a large scale analysis of most of the world's whole genome cancer data available to the cancer community through the TCGA and ICGA consortia using several clouds, including Bionimbus.

Bionimbus is an NIH Trusted Partner \cite{nihGDS16} that interoperates with eRA commons to authenticate researchers and with dbGaP to authorize user’s access to specific controlled access datasets, such as the TCGA dataset.

%Bionimbus specializes in supporting data intensive projects. For example, in a typical month, there are usually several computations that use over 100,000 core hours of computation. In other words, we have provided allocations of Bionimbus resources that favors a smaller number of large scale data intensive analysis over TCGA data rather than a larger number of projects that would use less resources, either less data or less core-hours. Subscription rates are extremely high for Bionimbus, and we have to turn away most applications, no matter how meritorious. Over the last year, at least 87\% of the available storage was constantly in use on Bionimbus and use of computing resources is usually at 90\% or higher.

The design and architecture for Bionimbus is described in \cite{Grossman:JAMIA-14}. The current architecture uses OpenStack for providing virtualized infrastructure, containers for providing a platform as a service capability, and object-based storage with an AWS compatible interface. See Figure~\ref{figure:osdc-arch}.


% \subsection{OSDC PIRE Project}



\section{Discussion}
\label{section:discussion}

\subsection{Data Commons and Science as a Service }

% We expect that data commons will play an increasingly important role in data science

With the appropriate services, data commons support three different, but related, functions.
First, data commons can serve as a data repository or digital library for data that is associated with published research.  Second, data commons can store data, along with computational environments in virtual machines or containers, so that computations supporting scientific discoveries can be reproducible.   Third, data commons can service as a platform enabling future discoveries, as more data, new algorithms, or new software applications are added to the commons.

Data commons fit well with a Science as a Service model: although data commons allow researchers to download data, host it themselves, and analyze it locally, they also support a Science as a Service model, in which i) current data can be reanalyzed with new methods and new tools and applications using co-located computing infrastructures; ii) new data can be uploaded for an integrated analysis; and iii) hosted data can be made available to other resources and applications using a data-as-a-service model in which data in a data commons is accessed through an API.  A data-as-a-service model is enhanced when multiple data commons and science clouds peer so that data may be moved between them at no cost.

\subsection{Challenges}

\ph{Sustainability challenges.}
Perhaps the biggest challenge for data commons, especially large scale data commons, is developing long-term sustainability models that support operations year after year. 

Over the past several years, funding agencies have required data management plans for the dissemination and sharing of research results, but, by and large, have not provided funding to support this requirement.   What this means is that there are a lot of data searching for data commons and similar infrastructure, but very little funding available to support this type of infrastructure.

% For small datasets, this is not as much as a problem as for larger datasets.  The challenge with large datasets is that it can be expensive to make large datasets available to the research community.   Although the 

\ph{Research Challenges.}
% The research challenges most relevant to the OSDC data commons including the following:
Data centers are sometimes divided into several ``pods'' to facilitate their management and build out.  For lack of better name, we sometimes use the term {\em cyberpod} to refer to the scale of a pod at a data center. A pod might contain 50 to several hundred racks of computing infrastructure.  Large scale Internet companies, such as 
Google \cite{dean2010mapreduce} and Amazon \cite{vogels2007dynamo}, have software that scales to cyberpods, but this proprietary software is not available to the research community.  
Although there are software applications, such as Hadoop \cite{white2012hadoop}, that are available to the research community and scale across multiple racks, there is not a complete open source software stack containing all the services required to build a large scale data commons, including the infrastructure automation and management services, security services, etc. \cite{barroso2013datacenter} that are required to operate a data commons at scale. 

We single out three research challenges related to building data commons at the scale of cyberpods.

\medbreak\noindent
{\em Software stacks for cyberpods.} 
The first research challenge is to develop a scalable open source software stack that the provides that infrastructure automation and monitoring, computing, storage, security services and related services required to operate computing services at the scale of a cyberpod \cite{barroso2013datacenter}.


\medbreak\noindent
{\em Datapods.} The second research challenge is to develop data management services that scale out to cyberpods.  We sometimes use the term {\em datapods} for data management infrastructure at this scale.

\medbreak\noindent
{\em AnalyticOps.} The third challenge is to develop an integrated development and operations methodology to support large scale analysis and re-analysis of data.   You might think of this as the analogy of DevOps for large scale data analysis.


\ph{Community challenges.}   The final category of challenges is the lack of consensus within the research community for a core set of standards that would support data commons.  There are not yet widely accepted standards for indexing data, APIs for accessing data, and authentication and authorization protocols for accessing controlled access data.   


\subsection{Lessons Learned}

\ph{Data re-analysis is an important capability.}  For many research projects, large datasets are periodically re-analyzed using new algorithms or new software applications.  Data commons are a convenient and cost effective way to provide this service, especially as the size of the data grows and it becomes more expensive to transfer the data.

\ph{Important discoveries are made at all computing resource levels.}  As mentioned, 
computing resources are rationed in a data commons (either directly through allocations or 
indirectly through charge backs).  Typically, there is a range of requests for computing allocations in a data commons spanning 6 to 7 or more orders of magnitude, ranging from hundreds of core hours to tens of millions of core hours.  The challenge is that important discoveries are usually made across the entire range of resource allocations, from the smallest to the largest.   This is because when large datasets, especially multiple large datasets, are co-located it is possible to make interesting discoveries even with relatively small amounts of compute.

\ph{Higher order services.}  Over the past several years, much of the research focus has been designing and operating data commons and science clouds that are scalable, contain interesting datasets, and offer computing infrastructure as a service.   We expect that as these types of Science as a Service offerings become more common, there will be a variety of more interesting higher order services, including discovery, correlation, and other analysis services that are offered within a commons or cloud and across several that interoperate. 

\ph{Cross-commons services.}  
%The last point is important enough to elaborate a bit.  
Today, web mashups are quite common, but analysis mashups in which data are left in place but continuously analyzed as a distributed service are relatively rare.  As data commons and science clouds become more common, these types of services can be more easily built.

\ph{Hybrid clouds will become the norm.}
At the scale of a several dozen racks (a cyberpod), a highly utilized data commons in a well-run data center is less expensive than using today's public clouds \cite{Grossman:JAMIA-14}.  For this reason, hybrid clouds consisting of privately run cyberpods hosting data commons that interoperate with public clouds seem to have certain advantages.   


\section{Summary and Conclusion}

In this paper, we have described a core set of requirements for data commons and some of our experiences developing and operating data commons with this design as part of the Open Science Data Cloud.  Properly designed data commons can serve several roles in Science as Service:  First, they can serve as an active, accessible, citable repository for research data in general, and research data associated with published research papers in particular.  Second, by co-locating computing resources, they can serve as a platform for reproducing research results.   Third, they can support future discoveries as more data are added to the commons, as new algorithms are developed and implemented in the commons, and as new software applications and tools are integrated into the commons.  Fourth, they can serve as a core component in an interoperable ``web of data,'' as the number of data commons begins to grow, as standards for data commons and their interoperability begin to mature, and as data commons begin to peer.

\section*{Acknowledgments}
This material is based in part upon work supported by the National Science Foundation under Grant Numbers OISE 1129076, CISE 1127316 and CISE 1251201 and by NIH/Leidos Biomedical Research, Inc. through contracts 14X050 and 13XS021 / HHSN261200800001E.



\clearpage
%\bibliographystyle{plain}
\bibliographystyle{IEEEtran}
\bibliography{bib-15}



\end{document}
osdc-cise-16-v1.bbl
% Generated by IEEEtran.bst, version: 1.13 (2008/09/30)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl

\bibitem{Grossman:SC12}
R.~L. Grossman, M.~Greenway, A.~P. Heath, R.~Powell, R.~D. Suarez, W.~Wells,
  K.~White, M.~Atkinson, I.~Klampanos, H.~L. Alvarez, C.~Harvey, and J.~J.
  Mambretti, ``The design of a community science cloud: The open science data
  cloud perspective,'' in \emph{Proceedings of the 2012 SC Companion: High
  Performance Computing, Networking Storage and Analysis}.\hskip 1em plus 0.5em
  minus 0.4em\relax IEEE Computer Society, 2012. Also available via
  arxiv.org/abs/1512.xxxxx.

\bibitem{Mell:2011}
P.~Mell and T.~Grance, ``The {NIST} definition of cloud computing, special
  publication 800-145,''
  http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf, September,
  2011.

\bibitem{dooley2012agave}
\BIBentryALTinterwordspacing
R.~Dooley, M.~Vaughn, D.~Stanzione, S.~Terry, and E.~Skidmore,
  ``Software-as-a-service: The iplant foundation api,'' in \emph{5th IEEE
  Workshop on Many-Task Computing on Grids and Supercomputers}, Nov. 2012.
  [Online]. Available: \url{http://datasys.cs.iit.edu/events/MTAGS12/p07.pdf}
\BIBentrySTDinterwordspacing

\bibitem{foster2011globus}
I.~Foster, ``Globus online: Accelerating and democratizing science through
  cloud-based services,'' \emph{IEEE Internet Computing}, no.~3, pp. 70--73,
  2011.

\bibitem{simmhan2005survey}
Y.~L. Simmhan, B.~Plale, and D.~Gannon, ``A survey of data provenance in
  e-science,'' \emph{ACM Sigmod Record}, vol.~34, no.~3, pp. 31--36, 2005.

\bibitem{chervenak2008wide}
A.~Chervenak, R.~Schuler, C.~Kesselman, S.~Koranda, and B.~Moe, ``Wide area
  data replication for scientific collaborations,'' \emph{International journal
  of high performance computing and networking}, vol.~5, no.~3, pp. 124--134,
  2008.

\bibitem{alameda2007open}
J.~Alameda, M.~Christie, G.~Fox, J.~Futrelle, D.~Gannon, M.~Hategan,
  G.~Kandaswamy, G.~von Laszewski, M.~A. Nacar, M.~Pierce \emph{et~al.}, ``The
  open grid computing environments collaboration: portlets and services for
  science gateways,'' \emph{Concurrency and Computation: Practice and
  Experience}, vol.~19, no.~6, pp. 921--942, 2007.

\bibitem{pepple2011deploying}
K.~Pepple, \emph{Deploying openstack}.\hskip 1em plus 0.5em minus 0.4em\relax
  O'Reilly Media, Inc., 2011.

\bibitem{davies2013scale}
A.~Davies and A.~Orsaria, ``Scale out with glusterfs,'' \emph{Linux Journal},
  vol. 2013, no. 235, p.~1, 2013.

\bibitem{weil2006ceph}
S.~A. Weil, S.~A. Brandt, E.~L. Miller, D.~D. Long, and C.~Maltzahn, ``Ceph: A
  scalable, high-performance distributed file system,'' in \emph{Proceedings of
  the 7th symposium on Operating systems design and implementation}.\hskip 1em
  plus 0.5em minus 0.4em\relax USENIX Association, 2006, pp. 307--320.

\bibitem{Mayernik2012data}
M.~S. Mayernik, ``Data citation initiatives and issues,'' \emph{Bulletin of the
  American Society for Information Science and Technology}, vol.~38, no.~5, pp.
  23--28, 2012.

\bibitem{Duerr2011utility}
R.~E. Duerr, R.~R. Downs, C.~Tilmes, B.~Barkstrom, W.~C. Lenhardt, J.~Glassy,
  L.~E. Bermudez, and P.~Slaughter, ``On the utility of identification schemes
  for digital earth science data: an assessment and recommendations,''
  \emph{Earth Science Informatics}, vol.~4, no.~3, pp. 139--160, 2011.

\bibitem{lagoze2014ced}
C.~Lagoze, L.~Vilhuber, J.~Williams, B.~Perry, and W.~C. Block, ``Ced 2 ar: The
  comprehensive extensible data documentation and access repository,'' in
  \emph{Digital Libraries (JCDL), 2014 IEEE/ACM Joint Conference on}.\hskip 1em
  plus 0.5em minus 0.4em\relax IEEE, 2014, pp. 267--276.

\bibitem{kunze2003towards}
J.~Kunze, ``Towards electronic persistence using ark identifiers,'' in
  \emph{Proceedings of the 3rd ECDL Workshop on Web Archives}, 2003.

\bibitem{Kunze:2006}
J.~Kunze and R.~Rodgers, ``The {ARK} identifier scheme,''
  www.ietf.org/internet-drafts/draft-kunze-ark-15.txt.

\bibitem{Pollard2010making}
T.~J. Pollard and J.~M. Wilkinson, ``Making datasets visible and accessible:
  Datacite’s first summer meeting,'' \emph{Ariadne, July}, 2010.

\bibitem{starr2012collaborative}
J.~Starr, P.~Willett, L.~Federer, C.~Horning, and M.~L. Bergstrom, ``A
  collaborative framework for data management services: The experience of the
  university of california,'' \emph{Journal of eScience Librarianship}, vol.~1,
  no.~2, p.~7, 2012.

\bibitem{varia2010architecting}
J.~Varia, ``Architecting for the cloud: Best practices,'' \emph{Amazon Web
  Services}, 2010.

\bibitem{ball2011cite}
A.~Ball and M.~Duke, \emph{How to cite datasets and link to
  publications}.\hskip 1em plus 0.5em minus 0.4em\relax Digital Curation
  Centre, 2011.

\bibitem{green2009we}
T.~Green, ``We need publishing standards for datasets and data tables,''
  \emph{Learned publishing}, vol.~22, no.~4, pp. 325--327, 2009.

\bibitem{mandl2013use}
D.~Mandl, S.~Frye, P.~Cappelaere, M.~Handy, F.~Policelli, M.~Katjizeu,
  G.~Van~Langenhove, G.~Aube, J.~Saulnier, R.~Sohlberg \emph{et~al.}, ``Use of
  the earth observing one (eo-1) satellite for the namibia sensorweb flood
  early warning pilot,'' \emph{Selected Topics in Applied Earth Observations
  and Remote Sensing, IEEE Journal of}, vol.~6, no.~2, pp. 298--308, 2013.

\bibitem{nihGDS16}
{NIH}, ``{G}enomic {D}ata {S}haring ({GDS}),'' Retrieved from
  https://gds.nih.gov on December 10, 2015.

\bibitem{Grossman:JAMIA-14}
A.~P. Heath, M.~Greenway, R.~Powell, J.~Spring, R.~Suarez, D.~Hanley,
  C.~Bandlamudi, M.~E. McNerney, K.~P. White, and R.~L. Grossman, ``Bionimbus:
  a cloud for managing, analyzing and sharing large genomics datasets,''
  \emph{Journal of the American Medical Informatics Association}, vol.~21,
  no.~6, pp. 969--975, 2014.

\bibitem{dean2010mapreduce}
J.~Dean and S.~Ghemawat, ``Mapreduce: a flexible data processing tool,''
  \emph{Communications of the ACM}, vol.~53, no.~1, pp. 72--77, 2010.

\bibitem{vogels2007dynamo}
G.~DeCandia, D.~Hastorun, M.~Jampani, G.~Kakulapati, A.~Lakshman, A.~Pilchin,
  S.~Sivasubramanian, P.~Vosshall, and W.~Vogels, ``Dynamo: amazon's highly
  available key-value store,'' in \emph{ACM SIGOPS Operating Systems Review},
  vol.~41, no.~6.\hskip 1em plus 0.5em minus 0.4em\relax ACM, 2007, pp.
  205--220.

\bibitem{white2012hadoop}
T.~White, \emph{Hadoop: The definitive guide}.\hskip 1em plus 0.5em minus
  0.4em\relax O'Reilly Media, Inc., 2012.

\bibitem{barroso2013datacenter}
L.~A. Barroso, J.~Clidaras, and U.~H{\"o}lzle, ``The datacenter as a computer:
  An introduction to the design of warehouse-scale machines,'' \emph{Synthesis
  lectures on computer architecture}, vol.~8, no.~3, pp. 1--154, 2013.

\end{thebibliography}
