Title\tnoteref{label1
Name\corref{cor1
Name\corref{cor1
Name\corref{cor1
Name\corref{cor1
Name\corref{cor1
Name\corref{cor1
Name\corref{cor1
Name\corref{cor1
Name\corref{cor1
Name\corref{cor1
Name\corref{cor1
Name\corref{cor1
%% Text of abstract
conclusion.tex
\section{\changedTwo{Competitions on ASlib}}
\label{sec:comp}


As described before and illustrated in Section~\ref{sec:experiments}, 
we \changedThree{designed ASlib to enable easy and fair comparison of different algorithm selection approaches}.
The next step to get unbiased performance comparisons of algorithm selectors is to organize competitions based on ASlib.
In this section, we will briefly describe two exemplary competition settings based on ASlib.

\paragraph{On-going Evaluation on ASlib}

In the on-going evaluation on ASlib\footnote{%
The most recent results of the on-going evaluation can be found on the ASlib homepage \url{aslib.net}.},
every participant can simply submit his/her performance for each scenario (using the provided cross validation splits) 
and the source code of their algorithm selector.
The latter will \changedThree{only be} used to verify the results in case of doubt.
The results (i.e., (penalized) average performance on each scenario) will be added in an overview table and the system will be linked. 

In this setting, every system that can read the ASlib format can easily participate and no deadlines for submission are required.
Therefore, the newest systems and results can \changedThree{always be} added on-the-fly 
such that the on-going evaluation always reflects the most recent known state-of-the-art systems and their performances.
Disadvantages of this setting are: 
\begin{enumerate}
  \item the different participants use different amounts of computational resources to compute the results -- for example, two well-performing systems in the on-going evaluation are \emph{SATzilla}~\cite{xu_satzilla_2008} and \emph{AutoFolio}~\cite{lindauer-jair15a} but it is also well-known that these two systems use a lot more computation resources (several CPU days) than other systems;
  \item since the test and training data are published, the system will tend to overfit the scenarios if we will not \changedThree{regularly provide} new scenarios to reveal such overfitting.
\end{enumerate}

\paragraph{ICON Challenge on Algorithm Selection}

The ICON Challenge on Algorithm
Selection\footnote{\url{http://challenge.icon-fet.eu/}} provided a 
comparative evaluation of state-of-the-art algorithm selection systems.
The winner of the challenge was the zilla system~\cite{xu2012satzilla2012}.
In this competition, the algorithm selectors needed to be submitted \changedThree{before} a fixed deadline and
each system was executed on the organizers' hardware with some limitations (e.g., at most $12$ CPU hours for training on one scenario).
Although the used scenarios were also already published before (i.e., ASlib
version 1.0.1),
the organizers did not reveal the training-test splits to avoid \changedThree{overly} strong overfitting on these scenarios.
Furthermore, the ICON challenge assessed the performance of the algorithm selectors based on $3$ different performance metrics (i.e., average number of solved instances, PAR10, and misclassification penalty (MCP)) which revealed some strengths and weaknesses of algorithm selectors,
e.g., systems that used an algorithm schedule had better performance on solved instances and PAR10, but wasted some \changedThree{time with respect to} MCP.
	
	
\section{Summary}
\label{sec:conclusion}


We have introduced ASlib, a benchmark library for algorithm selection, a rapidly
growing field of research with substantial impact on various sub-communities in
artificial intelligence. 
ASlib facilitates research on algorithm selection methods by 
providing a common set of benchmarks and tools for working with these. Similar to solver competitions, 
it enables principled comparative empirical performance assessment.
It also considerably lowers the otherwise rather high barrier for researchers to work on algorithm selection,
since anyone using the benchmark scenarios we provide does not have to perform
actual runs of the solvers contained in them.
Since our library provides performance data for the solvers and problem instances
included in each selection scenario, using ASlib also substantially reduces the computational burden of performance assessments. \changedThree{Otherwise, these data would have to be produced, at considerable computational cost, by anyone
working with that scenario.} We carefully selected the set of scenarios included in release version 2.0 of
ASlib to challenge algorithm selection 
methods in various ways and thus provide a solid basis for developing and assessing such methods.
%Future updates will ensure that ASlib remains useful as research on algorithm selection progresses.

Release version 2.0 of the library contains 17 algorithm selection
scenarios from six different areas with a focus on (but not a limitation to) constraint satisfaction problems.
We discussed the format of new algorithm selection scenarios and showed examples
of the automated exploratory data analysis that will run for each new scenario
submitted to our online platform 
\url{http://aslib.net/}.
Finally, exploratory studies with various algorithm selection approaches
demonstrated the performance \changedThree{that} algorithm selection systems can achieve on our
scenarios.
%We achieved performance improvements over the best single solver on all data sets, 
%often reducing penalized average runtime by a factor of 2 and in the best case by a factor of 3.

\subsection*{Acknowledgements}
\label{sec:ack}

%\note{FH}{I believe it is important to put in a thank-you like this to algorithm developers. We continue to steal their thunder and should most certainly share the glory with them.}
%\note{HH}{Agreed.}

We thank the creators of the algorithms and instance distributions used in our various algorithm selection scenarios. 
The performance of algorithm selection systems depends critically upon the ingenuity and tireless efforts of domain experts who continue to invent novel solver strategies.

FH and ML are supported by the DFG (German Research Foundation) under Emmy
Noether grant HU 1900/2-1. KLB, AF and LK were supported by an NSERC E.W.R.\
Steacie Fellowship; in addition, all of these, along with HH, were supported
under the NSERC Discovery Grant Program. Part of this research was supported by
an Azure for Research grant.

cover-resubmit.tex
\documentclass{scrlttr2}

\usepackage[english]{babel}

\setkomavar{subject}{Resubmission of manuscript ARTINT-D-14-00157}
\setkomavar{signature}{Lars Kotthoff}
\KOMAoptions{foldmarks=off,fromalign=left,fromlogo=false,addrfield=false,backaddress=false,subject=titled}
\makeatletter
\@setplength{refvpos}{5em}
\makeatother

\begin{document}

\begin{letter}

\opening{Dear editor of the AI Journal Competition Section,}

Included with this letter is the resubmission of manuscript ARTINT-D-14-00157
``ASlib: A Benchmark Library for Algorithm Selection''. The original submission
of this manuscript was for the Artificial Intelligence Journal Special Issue on
Combining Constraint Solving and Learning. The reviewers felt that the paper
could not be accepted as a standard contribution to the journal as it does not
pass the originality criteria. However, the editor in chief, Rina Dechter,
encouraged us to resubmit to the Competition Section, as a standardized
environment for the comparison of different approaches would be a valuable
service to the community and can have significant impact.

We have modified the previous version of the paper extensively to align it with
the criteria for the Competition Section; in particular we explain in detail the
relevance of the library for competitions and how it facilitates the evaluation
of different approaches in a competitive setting. The part of the paper that
provides an experimental evaluation of different approaches on the library data
has been greatly extended and can be used as a ``blueprint'' for future
competitions.

As this submission will be evaluated according to different criteria than the
previous one, we do not give a detailed list of the reviewers concerns and how
we addressed them thoroughly. Nevertheless, we have taken a lot of the feedback
we have received on board and improved the paper substantially. We would be
happy to provide such a detailed list if conducive to the review process.

\closing{Sincerely yours,}

\end{letter}
\end{document}

current_scenarios.tex
% FIXME:
% BB: This section needs an intro. Now!
% Mention where the scenarios are available on our page and give a brief overview table.
% Also match / check this intro section with the following intro sections
% Otherwise I have not checked these texts in detail.
% ML: I wrote a short intro and added a table. Please feel free to add and edit it.

\section{Algorithm Selection Scenarios Provided in ASlib Release 2.0}
\label{sec:scenarios}

% \note{Reviewer}{This section is particularly long and boring. 
% The description is not at the right level and not complete. 
% Since those are just scenarios, you do not need to explain SAT, MAXSAT, etc in full.}
% \note{ML}{I agree with the reviewer that the long description is not necessary for the paper and we could/should shorten this section quite a bit.}

% \note{Reviewer}{How is the pool of nominated algorithm selected? Further details would be nice. Further empirical evidence could be included to justify the selection.}

%\note{FH}{I reworded to remove redundant text passages.}

The set of algorithm selection scenarios in release version 2.0 of our library, shown in Table~\ref{tab:overview},
has been assembled \fh{to represent a diverse set of selection problem settings that covers a wide range of problem domains, types of algorithms, features and problem instances. Our scenarios include both problems that have been broadly studied in the context of algorithm selection techniques (such as SAT and CSP),
as well as more recent ones (such as the container pre-marshalling problem).} 
Most of our scenarios were taken from publications that
report performance improvements through algorithm selection
and consist
of algorithms where the virtual best solver (VBS)\footnote{The VBS is defined as
a solver that perfectly selects the best solver from $\mathcal{A}$ on a per-instance basis.} is 
significantly better than the single best solver.\footnote{The single best
solver has the best performance averaged across all instances.} 
Therefore, these are problems on which it makes sense to seek performance improvements via algorithm selection.
All scenarios are available on our online platform
(\url{http://www.aslib.net/}).

\fh{We now briefly describe the scenarios we included and what makes them interesting.}

%FIXME: Add Joaquins scenario when available
\begin{table}
\begin{center}
\small
\begin{tabular}{l cccccc}
\toprule
scenario 				& $\#\mathcal{I}$ 		& $\#\mathcal{A}$ 		& $\#\mathcal{F}$  		& $\#\mathcal{F}_g$  & Costs 	 & Literature \\
\midrule
 SAT$11$-HAND 	& $296$ 	 	& $15$ 	 		& $115$  		& $10$				  & $\checkmark$  & \cite{xu_satzilla_2008}\\
 SAT$11$-INDU 	& $300$			& $18$ 	 		& $115$  		& $10$				  & $\checkmark$ & \cite{xu_satzilla_2008}\\
 SAT$11$-RAND 	& $600$ 		& $9$ 	 		& $115$  		& $10$				  & $\checkmark$ & \cite{xu_satzilla_2008}\\
 SAT$12$-ALL	 	& $1614$		& $31$ 	 		& $115$  		& $10$				  & $\checkmark$ & \cite{xu2012satzilla2012}\\
 SAT$12$-HAND 	& $767$			& $31$ 	 		& $115$  		& $10$				  & $\checkmark$ & \cite{xu2012satzilla2012}\\
 SAT$12$-INDU  	& $1167$		& $31$ 	 		& $115$  		& $10$				  & $\checkmark$ & \cite{xu2012satzilla2012}\\
 SAT$12$-RAND  	& $1362$		& $31$ 	 		& $115$ 		& $10$				  & $\checkmark$ & \cite{xu2012satzilla2012}\\
 SAT$15$-INDU  	& $300$		& $28$ 	 		& $54$ 		& $1$				  & $\times$ & --\\
 QBF-$2011$  	 	&  $1368$		& $5$ 	 		& $46$  		& $1$				  & $\times$ & \cite{pulina_self-adaptive_2009}\\
 QBF-$2014$  	 	&  $1254$		& $14$ 	 		& $46$  		& $1$				  & $\times$ & --\\
 MAXSAT$12$-PMS 	& $876$			& $6$ 	 		& $37$  		& $1$				  & $\checkmark$  & \cite{malitsky_evolving_2013} \\
 MAXSAT$15$-PMS-INDU & $601$			& $29$ 	 		& $37$  		& $1$				  & $\times$  & -- \\
 CSP-$2010$  	 	& $2024$		& $2$ 	 		& $17$  		& $1$				  & $\times$ & \cite{gent_learning_2010}  \\
 CSP-MZN-$2013$		& $4642$		& $11$			& $155$			& $2$				  & $\checkmark$ & \cite{AmadiniGM14} \\
 PROTEUS-$2014$		& $4021$		& $22$			& $198$			& $4$				  & $\checkmark$ & \cite{hurley_proteus_2014}\\
 ASP-POTASSCO 	 	& $1294$		& $11$ 	 		& $138$ 		& $5$				  & $\checkmark$ & \cite{holisc14a}\\
 PREMAR-ASTAR-$2015$ & $527$ & $4$ 	 		& $22$  		& $3$				  & $\times$ & \cite{Ti14tr} \\
 %BBOB-$2013$ 	 	& $1200$		& $7$ 	 		& $53$  		& $1$				  & $\checkmark$ & $-$\\ %FIXME 
 %ML-UCI-WEKA 		& $?$			& $?$ 	 		& $?$  			& $?$				  & $?$  & \cite{?} \\ %FIXME
\bottomrule
\end{tabular}
\caption{Overview of algorithm selection scenarios in the ASLib with the number of instances $\#\mathcal{I}$, 
number of algorithms $\#\mathcal{A}$, 
number of features $\#\mathcal{F}$,
number of feature processing groups $\#\mathcal{F}_g$
and availability of feature costs.}
\label{tab:overview}
\end{center}
\end{table}

\subsection{SAT: Propositional Satisfiability}

% \note{ML/Reviewer}{I shortened this subsection further because the two reviewers complained about too much details}
% \note{KT}{Further shortened section, for datasets that do not have a corresponding TR, maybe create an appendix with the ``boring'' stuff? (e.g., SAT)}

The propositional satisfiability problem (SAT) is a classic NP-complete problem that consists of determining the existence of an assignment of values to variables of a Boolean formula such that the formula is true. It is widely studied, with many applications including formal verification \cite{prasad2005survey}, scheduling \cite{crawford1994experiment}, planning \cite{kautz1999unifying} and graph coloring \cite{van2008another}. Our SAT data mainly stems from different iterations of the SAT competition,\footnote{\url{http://www.satcompetition.org/}} \changed{which is split into three tracks: industrial (INDU), crafted (HAND), and random (RAND).}

% TODO >>> Move to Appendix?
% Both datasets are subdivided into three tracks. Industrial (INDU) tracks are encodings (both satisfiable and unsatisfiable) from real-world applications, such as hardware and software verification, bioinformatics, planning and scheduling. Crafted and handmade (CRAFTED/HAND) tracks contain custom-made instances and encodings of combinatorial problems made to challenge SAT solvers (both satisfiable and unsatisfiable). Finally, the random (RAND) tracks consist of randomly generated satisfiable instances. Solver creators determine the track or tracks in which their solvers participate, so each track has a distinct set of specialized solvers. 
% 
% In more detail, the SAT $2011$ dataset consists of all instances from the 2011 SAT 
% Competition, with solver performance data provided by the organizers. The INDU track
% contains 300 instances and 18 solvers, the CRAFTED track 296 instances and 15
% solvers, and the RAND track 600 instances and 9 solvers. In the same vein, the
% SAT $2012$ dataset is a mixture of data from all three SAT competitions and three SAT
% Races since 2006. It was used to train the successful SATzilla 2012 solver
% \cite{xu2012satzilla2012}, and algorithm executions were performed by its
% creators. Each track comprises the same 31 solvers, including 28 originating
% from various competitions, two versions of the solver \texttt{spear}~\cite{babic2008spear} configured
% for software and hardware verification~\cite{HutBabHooHu07}, and a new solver \texttt{mxc}~\cite{Bregman09thesat}. In terms
% of problem instances, there are 1\,362 instances for the RAND track, 767 instances
% for HAND track, and 1\,167 instances for the INDU track. 
% Finally, 538 instances were sampled from each of the three tracks and combined to form the general ALL track.
% \changed{We note that the SAT $2011$ scenarios cannot be combined into one scenarios 
% because the set of algorithms is different for each $2011$ scenario.} 
% 
% In addition to performance data, we provide 115 instance features for every dataset (which were developed by the SATzilla team over the years starting in 2003 \cite{Satzilla03}, and are described by \citet{nudelman_understanding_2004}, \citet{xu_satzilla_2008}, and \citet{hutter2014algorithm}). They are distinctively grouped by their origins, and can vary in extraction cost.
% Some easy instances can be solved during feature extraction with the help of probing features. 
% TODO <<<

\changed{The SAT scenarios are characterized by a high level of maturity and diversity in terms of their solvers, features and instances. Each SAT scenario involves a highly diverse set of solvers, many of which have been developed for several years.} \fh{In addition, the set of SAT features is probably the best-studied feature set among our scenarios; it includes both static and probing features that are organized into as many as ten different feature groups. The instance sets used in our various SAT scenarios range from randomly-generated ones to real-world instances submitted by industry.}

\subsection{QBF: Quantified Boolean Formula}

A quantified Boolean formula (QBF) is a formula in propositional logic with
\changed{universal or existential} quantifiers \changed{on each variable in the} formula. 
A QBF solver finds a set of variable assignments
that makes the formula true or proves that no such set can exist. This is a
\hh{PSPACE-complete} problem\changed{ for which 
solvers exhibit a wide range of} performance characteristics.
Our QBF-2011 data set comes from the QBF Solver Evaluation
%HH: sounds strange - why call it QBF-2011 and not QBF-2010?
2010\footnote{\url{http://www.qbflib.org/index_eval.php}} and consists of instances from the main, small hard, 2QBF and random tracks.
Our QBF-2014 data set comes from the application track of the QBF Gallery 2014\footnote{\url{http://qbf.satisfiability.org/gallery/}}.
The instance features were computed using the AQME system and are described in
more detail by Pulina et al.~\cite{pulina_self-adaptive_2009}.
The solvers for QBF-2011 come from the AQME system as well, whereas the solvers
for QBF-2014 are the ones submitted to the application track of the QBF Gallery.

\subsection{MAXSAT: Maximum Satisfiability}

MaxSAT is the optimization version of the previously introduced SAT problem, and aims to find a variable assignment that maximizes the number of satisfied clauses. The MaxSAT problem representation can be used to effectively encode a number of real-world problems, such as 
%timetabling~\cite{timetabling10} problems, %ML: Citation and Link was broken 
FPGA routing~\cite{XuRuSa03}, and software package installation~\cite{ArBeLyMaRa10}, among others, as it permits reasoning about both optimality and feasibility.
%
The particular scenarios focus on the partial MaxSAT (PMS) problem~\cite{SATHandbook}.
%, in which clauses in a \changed{Boolean formula are partitioned} and classified as either hard or soft. The objective of a PMS solver is to satisfy all of the ``hard'' clauses while minimizing the number of unsatisfied ``soft'' clauses. 
%The provided data is a collection of random, crafted and industrial unweighted MaxSAT instances from the 2012 MaxSAT Evaluation~\cite{MAXSATevaluations}. 
% There are a total of 876 instances, each solved with 6 state-of-the-art solvers from 2012 (akmaxsat\_ls, akmaxsat, DSWPM1\_924, pwbo2.1, qmaxsat0.21comp, qmaxsat0.21g2comp). 

% For the features, the dataset relies on a collection of 37 attributes based on the core set of SAT features described by \citet{xu_satzilla_2008}. This includes the problem size features, such as number of clauses and variables and their ratio, the variable-clause graph features, balance of positive to negative literals per variable and per clause, and the proximity to the horn formula. In addition to this standard set of SAT based features, the feature data also includes the percentage of soft clauses. 

\changed{
The MAXSAT$12$-PMS scenario is composed of a collection of random, crafted and
industrial instances from the 2012 MaxSAT Evaluation~\cite{MAXSATevaluations}.
The techniques used to solve the various instances in this scenario are very
complementary to each other, leading to a substantial performance gap between
the single best and the virtual best solver. Furthermore, \hh{because there are
only six solvers with very different performance characteristics, algorithm
selection} approaches must be very accurate in their choices, as any mistake
is heavily penalized. }

\changedTwo{
The more recent MAXSAT$15$-PMS-INDU was built on the performance data of the
industrial track on partial MAXSAT problems from the 2015 MAXSAT Evaluation.\footnote{\url{http://www.maxsat.udl.cat/15/results/index.html}}
With $29$ algorithms, it provides a larger set of solvers than MAXSAT$12$-PMS.
However, there are different parameterizations of the same solvers,
e.g., four different variants of \emph{ahms},
such that there are some subsets of strongly correlated solvers. 
The performance gap between the single best and virtual
best solver is larger in MAXSAT$12$-PMS than in MAXSAT$15$-PMS-INDU.  % is it likely that this is a consequence of multiple solvers being similar? I (KT) removed that.
}

\subsection{CSP: Constraint solving}

\changed{Constraint Satisfaction Problem (CSP; ~\cite{StuckeyFSTF14})} is concerned with finding solutions to constraint
satisfaction problems---a task that is NP-complete.
Learning in the context of constraint solving is a technique by which previously unknown
constraints that are implied by the problem specification are uncovered during
search and subsequently used to speed up the solving process.

% The data set consists of 2,024 problem instances, each of which is described by
% 86 features. There are two algorithms: one solver that employs lazy
% learning~\cite{gent_lazy_2010} and one that does not~\cite{gent_minion_2006}.
% The data set is heavily biased towards the non-learning solver, as it obtains
% better performance in the majority of cases. There are, however, some instances
% where using lazy learning improves performance by several orders of magnitude.
% More details can be found in~\cite{gent_learning_2010}.

% \paragraph{CSP-$2010$}
\changed{
The scenario CSP-$2010$ contains only two solvers: one that employs lazy
learning~\cite{gent_lazy_2010, gent_learning_2010} and one that does not~\cite{gent_minion_2006}.
The data set is heavily biased towards the non-learning solver, such that the baseline (the
single best solver) is very good already. Improving on this is a challenging
task and harder than in many of the other scenarios. Furthermore, both solvers
share a common core, which results in a scenario that directly evaluates the
efficacy of a specific technique in different contexts.% where the differences are much less obvious. %%% Too vague, created a run-on sentence. Removing.
}

% \paragraph{CSP-MZN-$2013$}
\changedTwo{The more recent scenario CSP-MZN-$2013$ provides a larger set of
instances, algorithms and instance features. Instances and algorithms come from
the MiniZinc challenge $2012$ and the International Constraint Solver
Competitions (ICSC) in $2009$. Specifically, the instances come from the
MiniZinc $1.6$ benchmark suite and the algorithms in the scenario participated
in the MiniZinc Challenge $2012$. Algorithms, instances and instance features
are described in more detail in \cite{AmadiniGM14,AmadiniGM14a}.
}

% \note{ML}{I don't know these scenarios very well. Maybe someone else can add some more?} % KT: Seems like enough information to me. Succinct is fine here.

\changedTwo{Our final CSP scenario PROTEUS comes
from}~\cite{hurley_proteus_2014} and includes an extremely diverse mix
of well-known CSP solvers alongside competition-winning SAT solvers 
that have to solve (converted) XCSP instances\footnote{The XCSP instances are taken from
\url{http://www.cril.univ-artois.fr/~lecoutre/benchmarks.html} as described
in~\cite{hurley_proteus_2014}.}. 
The SAT solvers can accept different conversions of the CSP
problem into SAT (see, e.g.,~\cite{daniel_simple_csp_to_sat,tamura_sugar,tanjo_azucar}), 
which in our format are provided as separate algorithms. 
This scenario is the only one in which
solvers are tested with varying ``views'' of the same problem. The
features of this scenario are also unique in that they include both the SAT and
CSP features for a given instance. This potentially provides additional
information to the selection approach that would normally not be available for
solving CSPs. An algorithm selection system has a very high degree of
flexibility here and may choose to perform only part of the possible
conversions, thereby reducing the set of solvers and features, but also reducing
the overhead of performing the conversions and feature computations. There are
also synergies between feature computation and algorithm runs that can be
exploited, e.g., if the same conversion is used for feature computation and to
run the chosen algorithm then the cost of performing the conversion is only incurred
once. In other cases, where features are computed on one representation and
another one is solved, conversion costs are incurred both during feature
computation and the running of the algorithm.
%\note{FH}{As a reviewer, I would ask what is meant by ``An algorithm selection system has a very high degree of
%flexibility here and may choose to perform only part of the possible
%conversions, thereby reducing the set of solvers and features''. In particular,
%where does the cost of the conversions count that need to be done for both
%instance feature computation and to run the algorithm? As part of the algorithm
%runtime, as part of the instance feature time, as part of both (you have to pay
%twice), or somehow only on demand (you have to pay once, if you use it for the
%algorithm or the features or both)?\\\\
%LK: I've tried to clarify this.}


\subsection{ASP: Answer Set Programming}

Answer Set Programming (ASP, \cite{baral02a,gekakasc12a}) is a form of declarative programming 
with roots in knowledge representation, non-monotonic reasoning and constraint solving.
In contrast to many other constraint solving domains (e.g., the satisfiability problem),
ASP provides a rich yet simple declarative modeling language 
in which problems \changed{up to $\Delta_3^{\rm p}$ (disjunctive optimization problems)} can be expressed.
\changed{ASP has proven to be efficiently applicable to many real-world applications},
e.g., product configuration~\cite{soinie99a},
decision support for NASA shuttle controllers~\cite{nobagewaba01a},
synthesis of multiprocessor systems~\cite{ismabogesc09a} 
and industrial team building~\cite{griilelirisc10a}.

%Like algorithms for many other constraint solving domains,
%ASP solvers are highly sensitive to parameter configuration
%and there is no single dominant configuration for all types of ASP instances.
%As previously shown \cite{gebser_portfolio_2011,mapuri13a},
%this can be exploited by algorithm selection.

\changed{
In contrast to the other scenarios,
the algorithms in the scenario ASP-POTASSCO were automatically constructed by an adapted version of \system{Hydra}~\cite{xu_hydra_2010},
i.e., the set of algorithms consists of complementary configurations of the solver \system{clasp}~\cite{gekasc12b}.
The instance features were generated by a light-weight version of \system{clasp},
including static and probing features organized into feature groups; they
were previously used in the algorithm selector \emph{claspfolio}~\cite{gebser_portfolio_2011,holisc14a}.}


%Our ASP data set consists of a representative and extensive set of 1\,294 grounded ASP instances in NP  
%which were collected by the Potassco group~\cite{gekakaosscsc11a}
%and include several ASP Competition instances.
%The portfolio consists of $11$ configurations of the state-of-the-art ASP solver \system{clasp}~\cite{gekasc12b};
%it was automatically constructed by an adapted version of \system{Hydra}~\cite{xu_hydra_2010} for ASP.
%The ASP feature generator \system{claspre}~\cite{gebser_portfolio_2011}, a light-weight version of \system{clasp}, was used to extract $138$ instance features,
%which includes static and dynamic features recorded during a short solving period of \system{clasp}.


% \subsection{BBOB\_2013: Black Box Optimization Benchmarks}
% 
% \note{FH}{Let's drop the information that this was part of a master thesis; we don't say for each of the other data sets that they come from PhD theses etc, and neither should we. Let's also drop ``-Master-Thesis'' from the dataset title. On the flipside, importantly, let's put in the citation for the Master thesis (for the time being, it's also OK if it's cited as ``in progress'', we just need to fix it for the CRC)!}
% 
% This data set was created during a master thesis 
% and covers the problem of finding the best solver for continuous black-box optimization problems using features derived by \system{Exploratory Landscape Analysis}~\cite{Bischl2012_2}. The original \system{Black-Box Optimization Benchmark}~\cite{hansen2010} consists of 24 continuous functions, representing various optimization problems. Due to scaling, rotation and shifting of those 24 functions, one can generate several additional problem instances. This data set used ten instances per function and each of those were replicated ten times in order to handle stochastic features and optimization algorithms. Furthermore, each of the functions can be computed for several dimensions of the decision space. The underlying data set is based on 2-, 3-, 5-, 10- and 20-dimensional problems.
% 
% In~\cite{mersmann2011} a set of high-level features was introduced, which can be used for characterizing (unknown) functions. However, as those high-level features depend on expert knowledge, they were replaced by more easily computable feature groups, with a total of 50 so-called low-level features. Those six groups are \system{convexity} (3 features), \system{curvature} (15), \system{y-distribution} (3), \system{levelset} (12), \system{local search} (10) and \system{meta-models} (7). Thus, the feature data consists of 12\,000 instances (5 dimensions, 24 functions, 10 instances, 10 replications) and 50 feature values.
% 
% The portfolio of the used optimization algorithms consists of representatives of different optimization techniques: \system{L-BFGS-B} (a Quasi-Newton method with limited memory and bound constraints~\cite{byrd1995}), \system{CMA-ES} (Covariance Matrix Adaption Evolutionary Strategy~\cite{hansen2006}), \system{DEoptim} (Differential Evolution Optimization~\cite{ardia2011}), \system{Genoud} (Genetic Optimization Using Derivatives~\cite{mebane2011}), \system{GenSA} (Generalized Simulated Annealing~\cite{xiang2013}), \system{PSO} (Particle Swarm Optimizer~\cite{qin2010}) and a very simpe random search algorithm.
% 
% The performance of the solvers was measured using the \system{Expected Runtime} (ERT~\cite{hansen2010}) based on ten runs per problem instance. As the ERT strongly depends on the dimension and complexity of the underlying problem instance, the ERT was converted into a \system{Relative Expected Runtime} (relERT~\cite{Bischl2012_2}) by standardizing the ERTs of an instance with the best ERT of that instance. In case of missing ERTs, e.g., due to timeouts, the corresponding relERT was calculated using Par10 on the values of that instance.


% \subsection{Machine Learning UCI-WEKA defaults}
% 
% Supervised classification is a classical machine learning problem, in which
% algorithms (classifiers) build a model to predict a categorical target
% variable based on a set of descriptive features. These models are evaluated by
% splitting the data into a training and a test set as part of a
% cross-validation procedure, training the model on the training data and
% testing it on the test data. For this scenario, we queried the OpenML platform \cite{openml} for the performance results of 53 classification algorithms from the WEKA workbench \cite{weka} on 88 tabular classification datasets from the UCI repository \cite{uci}.
% 
% The algorithms include
% \texttt{ADTree},
% \texttt{AODE},
% \texttt{BayesNet},
% \texttt{DecisionStump},
% \texttt{DecisionTable},
% \texttt{IBk},
% \texttt{Id3},
% \texttt{LBR},
% \texttt{LMT},
% \texttt{LWL},
% \texttt{Logistic},
% \texttt{LogitBoost},
% \texttt{NBTree},
% \texttt{NaiveBayes},
% \texttt{OneR},
% \texttt{PART},
% \texttt{Ridor},
% \texttt{ZeroR},
% \texttt{AdaBoostM1},
% \texttt{Bagging},
% \texttt{Decorate},
% \texttt{J48},
% \texttt{JRip},
% \texttt{LogitBoost},
% \texttt{MultiBoostAB},
% \texttt{MultilayerPerceptron},
% \texttt{RBFNetwork},
% \texttt{REPTree},
% \texttt{RandomForest},
% \texttt{SMO},
% \texttt{Stacking}, and
% \texttt{VotedPerceptron}. All algorithms are evaluated using a 10-fold crossvalidation procedure and a list of evaluation measures including predictive accuracy, root mean squared error (RMSE), 
% \todo{FH: if we list both predictive ``accuracy" and ``root mean squared error'', we should state how they differ; predictive accuracy is a vague term.}
% mean absolute error (MAE), cputime and memory consumption. For this scenario, only the default parameter settings of these algorithms were used.
% 
% All datasets were converted to the ARFF format, and the exact versions can be found on OpenML.org. All datasets are characterized with a set of (meta)features including statistical and information-theoretic properties of the data distribution, such as the number of instances and features, missing values, class entropy, mutual information, as well as the result of landmarkers\cite{landmarking}, i.e. the performances of simplified classifiers run on the datasets.

\subsection{PREMAR-ASTAR-2015: Container pre-marshalling}

\changed{The container pre-marshalling problem (CPMP) is an NP-hard container
stacking problem from the container terminals literature~\cite{StVo08}.
\hh{We constructed an algorithm selection scenario from two recent A* and IDA* approaches for
solving the CPMP presented in~\cite{TiPaVo14tr}, using instances from the
literature}. The scenario is described in detail in~\cite{Ti14tr}.

The pre-marshalling scenario \hh{differs from} other scenarios in that the set
of algorithms is highly homogeneous. All of the algorithms are parameterizations
of a single symmetry breaking heuristic, either using the A* or IDA* search
techniques, which stands in sharp contrast to the diversity of solvers present
in most other scenarios.
% LK: I don't think this is true anymore for the new 2015 version.
%Furthermore, the features provided are new and not as
%well tested as in the other scenarios, perhaps more accurately resembling the
%features that would be created by domain experts on their first attempt at
%modeling a problem.
The scenario represents a real-world,
time-sensitive problem from the operations research literature, where algorithm
selection techniques can have a large impact.}


eda.tex
\section{Automated Exploratory Data Analysis}
\label{sec:eda}

%\note{Reviewer}{The description here is pedestrian.
%As indicated above in section 3, a diagram illustrating the whole framework would be nice.\\
%\changed{PK: added table~\ref{tab:eda_framework} for illustrating the structure of the platform}}
%\note{ML}{Please add for each shown figure: ``lessons learned'' from the figure (or else don't show them). \changed{PK: done.}}

%\note{PK}{ Changed the boxplot figure from MAXSAT to QBF as it contains less algorithms and therefore fits better in the limited space. Also, the scatterplot matrix, as well as the density and cumulated density plots in figure~\ref{fig:qbf} were based on QBF. Now it's more consistent.}

%\note{Reviewer}{Increase font size in Figure~\ref{fig:scatter} and \ref{fig:qbf}!}

%\note{Reviewer}{Figure~\ref{fig:qbf} caption should be expanded to clarify.\\
%\changed{PK: I rearranged the 4 four plots within the two figures. Now, the scatterplot matrix is a figure itself and therefore has sufficient space to be better readable. Also, I added the boxplots to the density and cumulated density. This way, I was able to increase the visibility of all four plots. Even though there are now three plots in figure~\ref{fig:qbf} instead of two, they have a little bit more space, because they all now share the same legend. Furthermore, all three plots help to understand the distribution of the performance values within a single algorithm, whereas the scatterplot matrix looks at the correlation between them. The only downside of this approach is, that I had to modify the figures slightly (compared to the HTML pages). Is that still ok?}}

%\note{ML/Reviewer}{Number of features per feature groups not correct in Table~\ref{table:features} - I assume the complete row of PRE-featuretime is not correct. Please read format specification carefully!\\}

% LK: I've commented this for now -- it is somewhat redundant with the
% information provided in the text and I would prefer this to be included in the
% itemized list that is there already.
%\begin{table}[t]
%\centering
%{\small
%\begin{tabular}{lll}
%\toprule
% \multicolumn{3}{c}{\textbf{Overview of all Tasks}} \\ \hline
%   \rule{0pt}{9pt}\textbf{Algorithms Overview} 	& \textbf{Features Overview} 		& \textbf{Further Results / Material} \\ %\hline
%   \rule{0pt}{9pt}Summary of: 				& Summary of: 			& - readme, describing the task\\
%   - performance values 				& - feature values 		& - download the scenario files \\
%   - run status 						& - run status and costs	& - benchmark results \\
%   - dominated algorithms				& of feature steps 		& (see Section~\ref{sec:experiments}) \\
%   								& - duplicated features 	& - configurations \\
%    								& 					& - scenario validator output \\
%   \multicolumn{2}{l}{\textbf{Figures (based on performance values):}}		&  \\
%   - boxplots						&  					&  \\
%   - estimated density plots				&  					&  \\
%   \multicolumn{2}{l}{- estimated cumulative distribution functions}	&  \\
%   - scatterplot matrix  				&  					&  \\
%   \multicolumn{2}{l}{- plot of clustered correlation matrix}			&  \\
%   \bottomrule
%\end{tabular}
%}
%\caption{Framework of the Platform's Automated Exploratory Data Analysis.}
%\label{tab:eda_framework}
%\end{table}

The online platform for our benchmark repository \changedTwo{offers not only}
the scenario data files themselves. It also
provides many tables and figures that summarize them.
%Table~\ref{tab:eda_framework} gives an overview of the platform's framework.
These pages are automatically generated and currently consist (among
others) of the following parts:

\begin{itemize}
\item an overview table that lists, for example, the number of instances, algorithms
and features for all available scenarios, similar to Table~\ref{tab:overview};
\item a summary of the algorithms' performance and run status data;
\item a summary of the feature values, as well as the run status and costs of the feature groups;
\item benchmark results for standard machine learning models for each scenario; see Section~\ref{sec:experiments}.
\end{itemize}

Presenting this additional data offers the following advantages:

\begin{itemize}
\item Researchers can quickly understand which scenarios are available and select those best suited to their needs.
\item \replaced{Data can quickly be sanity-checked}{Data can be sanity-checked by eye-balling}. It is common that data collection errors occur when scenario data is gathered and submitted for the first time.
\item Interesting or challenging properties of the data sets become visible, providing the researcher with a quick and informative first impression.
\end{itemize}

The \replaced{platform's}{} \textbf{summary page for the algorithms} starts with
a table listing summary statistics
regarding their performance (e.g., mean values and standard variations) and run status 
(e.g., how many runs were successful). 
We also indicate whether one algorithm is dominated by another, i.e., an algorithm $a_1$ dominates another algorithm $a_2$ if and only if $a_1$ has performance at least equal to that of $a_2$ on all instances, and $a_1$ outperforms $a_2$ on at least one instance.
This is useful, because there is no reason to include a dominated algorithm in a portfolio.
Various visualizations, such as box plots, scatter plot matrices, correlation plots and density plots 
enable further inspection of the performance distribution
%\note{FH}{what does ``further inspection of location'' mean? Or is this a typo?}
and correlation between algorithms\changed{, allowing the reader to better understand the strengths and weaknesses of each algorithm.} \replaced{}{For display, we impute high values for the missing or \changedTwo{censored} performance values corresponding to failed runs so that they are clearly visible rather than silently discarded.} All of our plots can be configured to use log scales, which often improves visual understanding \changedTwo{of heavy-tailed distributions (e.g., runtime distributions of hard combinatorial solvers~\cite{GomesSCK00})}.
%(i.e., logarithms cannot be calculated if the data includes non-positive values).

\begin{figure}[!t]
	\begin{minipage}[!t]{0.29\textwidth}
	\includegraphics[width=\textwidth]{pics/qbf_boxplot.pdf}
	\end{minipage}
	\hfill
	\begin{minipage}[!t]{0.7\textwidth}
	\includegraphics[width=\textwidth]{pics/qbf_cdf_new.pdf}
	\end{minipage}
%	\hfill
%	\begin{minipage}[!t]{0.18\textwidth}
%	\hspace*{-0.75cm}
%	\includegraphics[width=2\textwidth]{pics/qbf_legend.pdf}
%	\end{minipage}
%	\begin{minipage}[!t]{0.5\textwidth}
%	\includegraphics[width=\textwidth]{pics/dens_qbf2.pdf}
%	\end{minipage}
%	\hfill
%	\begin{minipage}[!t]{0.5\textwidth}
%	\includegraphics[width=\textwidth]{pics/prob_qbf2.pdf}
%	\end{minipage}
	 \caption{\changed{Algorithm performance distributions of the QBF-$2011$
     scenario: Boxplots (left) and cumulative distribution functions (right);
     both on a log scale.}}
	\label{fig:qbf}
\end{figure}

%\note{HH}{I am trying this one more time: the estimated densities are misleading and should be omitted. Consider Figure~2: Most of the modes shown in the middle plot are almost certainly spurious, as can be clearly seen from the CDFs -- in particular, the large modes near the cutoff. If anyone really insists on keeping them, I insist that we add at least a warning to that effect to the text. However, I believe keeping them is a mistake: we implicitly encourage others to use a methodology which we know to be flawed in that it can (and often does) produce misleading results, while we all know well how to avoid these issues.}

Figure~\ref{fig:qbf} shows \changed{boxplots} and cumulative distribution
functions for the algorithms in the QBF-$2011$ scenario as an example.
\replaced{The boxplots
summarize the runtimes of an algorithm by drawing a box between the 25\%- and
75\%-quantile of the sample, i.e.,~the smallest values that are greater or equal
to 25\% and 75\% of the runtimes. In addition, each box contains a line showing
the median runtime, as well as so-called whiskers, i.e.,~lines that connect the
box with runtimes that are within 150\% of the interquartile range (the length
of the box) below the 25\%- or above the 75\%-quantile, respectively.
Observations with even more extreme runtimes are considered to be outliers and
are depicted by a single point per outlier. The
cumulative distribution function plots on the other hand
show runtimes on all instances for the algorithm. Each point within the plot consists of
the observed runtime on the x-axis and the corresponding cumulative density,
i.e., the percentage of instances that were solved at this or a smaller runtime,
on the y-axis.}{}

Such plots show the location of the mean, distribution spread, density
multimodality and other properties of the distribution. In addition, they reveal
how long it took an algorithm to solve the given instances. \changed{For
example, for the QBF-2011 scenario in Figure~\ref{fig:qbf}, one can see that the
algorithm \system{quantor} finds a solution very quickly on a few instances,
i.e., it solves approximately $5\%$ of the instances \changedTwo{in less than a
second.} However, if it does not succeed quickly, it often does not succeed at
all---it solved less than $30\%$ of all the instances. In contrast,
\system{sSolve} usually needs longer to find a solution, but by the time it
does, it is one of the best algorithms. Such behavior can indicate that the
algorithm \hh{requires a `warm-up'} stage, which should be considered when
deploying it.}
%\note{FH}{I believe vague language such as `might indicate' in the examples we present ultimately signals to readers and reviewers that we haven't invested much time into looking at the domains we study. While this is the case for some domains, we know others really well and we might want to use that fact in the examples we give (e.g., for the SAT example below I added some details). Does anyone know what \system{sSolve} does, e.g., from a paper describing it? Maybe its contributor? (I don't know who provided it.)}
%\note{PK}{If I remember correct, this scenario comes from Lars. But in any case, I am not sure, whether it is a bad thing to state something rather vague. I mean, the intention of this section is to describe the automated EDA, which our platform provides. This section should not aim at discussing the scenarios based on expert knowledge, it should rather inform the reader, what he/she might see by looking at our plots (without being an expert in each of the applied algorithms).}

The left panel of \changed{Figure~\ref{fig:scatter} shows pairwise scatterplots of the QBF-$2011$ scenario, allowing an easy comparison of algorithm pairs on all instances from a given scenario. 
Each point represents a problem instance within the scenario, and from the location
of the point cloud one can see whether an algorithm is dominant over the
majority of instances, or whether relative performance strongly varies across instances. The first case can be identified by a cloud that is located either in the upper-left or lower-right corner of a single scatterplot. In such a case, the dominated algorithm could be discarded from the portfolio. However, if this type of \replaced{dominance relationship}{domination} is not present, there is the potential to realize performance improvements by means of per-instance algorithm selection.}
%\note{FH}{Why did we pick this scenario? The text does not refer to the figure again, so does that mean we did not learn anything from these plots? Is there a more interesting case where we do actually learn something?}
%\note{PK}{We picked this scenario mainly because of its number of algorithms; the SAT scenarios had too many algorithms, so you barely see anything on a paper and scenarios with only two algorithms were not very interesting as well.}

%The left half of Figure~\ref{fig:scatter} shows box plots for the algorithms in the MAXSAT$12$-PMS scenario. The right half displays pairwise scatter plots for the QBF-$2011$ scenario.
%The diagonal shows histograms of the algorithms' performance and the scatter plots are enhanced with a linear regression line. Pearson's coefficient of correlation -- a measure for the linear relationship between two variables -- is also calculated for each pair of algorithms. Strong positive correlations are marked green, strong negative correlations red, mediocre correlations orange and low correlations black. 
%Detecting correlations between algorithms is interesting, as algorithms that have a (high) positive correlation are more likely to be redundant in a portfolio, whereas pairs with a (high) negative correlation are more likely to complement each other.


\begin{figure}[!t]
	\centering
	\begin{minipage}[!t]{0.55\textwidth}
	\includegraphics[width=1.05\textwidth]{pics/qbf_scatter.pdf}
	\end{minipage}
%	\newline
	\hfill
	\begin{minipage}[!t]{0.43\textwidth}
	\hspace{-0.65cm}
	\includegraphics[width=1.15\textwidth]{pics/qbf_cormatrix.pdf}
	\end{minipage}
	 \caption{\changed{Pairwise correlations among algorithms of the QBF-$2011$ scenario: A scatter plot matrix on a log scale (left) and the plot of a correlation matrix (right).}}
	\label{fig:scatter}
\end{figure}

\changed{\hh{Because detecting correlation in algorithm performance is also of
interest when analyzing the strengths and weaknesses of a given portfolio-based
solver}~\cite{xu2012evaluating},} we also present a correlation
matrix, cf. Figure \ref{fig:scatter} (right panel). \changed{Algorithms that
have a (high) positive correlation are more likely to be redundant in a
portfolio, whereas pairs with a (high) negative correlation are more likely to
complement each other.} We calculate Spearman's correlation coefficient between
ranks. Blue boxes represent positive correlation, red boxes represent negative
correlation, and shading indicates the strength of correlation. The algorithms
are also clustered according to these values (using Ward's method~\cite{ward63})
and then sorted, such that similar algorithms appear together in blocks.
\changed{This type of clustering allows the identification of algorithms with
highly correlated performance.}

\changed{\replaced{Figure~\ref{fig:cormat} shows the correlation between
algorithms for the SAT$12$-ALL scenario. The plot reveals}{The example given in
Figure \ref{fig:cormat} (the plot of a correlation matrix of the SAT$12$-ALL
scenario) shows} four groups of algorithms (\system{minisatpsm} to
\system{restartsat}, \system{sattimep} to \system{tnm}, \system{marchrw} and the
three \system{mphaseSAT}-algorithms) with high correlations within each group.
It may be desirable to include just a single representative from each group,
reducing the size of the entire portfolio from 31 to four algorithms.}

\begin{figure}[!t]
	\includegraphics[width=\textwidth]{pics/cormatrix_sat12_all.pdf}
	 \caption{Algorithm correlations on the SAT$12$-ALL scenario.}
	\label{fig:cormat}
\end{figure}


As we do with algorithm runs, we characterise the features by giving summary statistics of the
feature values, the run status and \changedThree{the} cost of the feature groups.
Table~\ref{table:features} shows the summary of the feature groups for the
SAT$12$-RAND scenario. \changed{In this scenario, all 115 features have the
feature group ``Pre'' as a requirement.
While this preprocessing group
%In addition, 109 of those 115 features need an additional feature step. 
succeeded in all cases, \fh{one other group did not: the feature group ``CG'' (which computes clause graph features) failed in 37.37\% of cases due to exceeding time or memory limits, and even for instances where it succeeded, it was quite expensive ($8.79$ seconds on average).} 
%. However, among the 1362 instances of the entire scenario, the computation of the ten features that belong to the feature step `CG', failed in 37.37\% of the cases and is relatively expensive with $8.79$ seconds on average. 
Such information is useful to understand the behavior of the features:
\changedThree{how risky is it to compute a feature group, and how much time must one invest in order to obtain the corresponding features?}}

\begin{table}[t]
\centering
{\small
\begin{tabular}{rrrrrrrrr}
\toprule
& & \multicolumn{3}{c}{runstatus [\%]} & \multicolumn{4}{c}{cost [s]} \\ 
\cmidrule(lr){3-5}\cmidrule(lr){6-9}
feature group & $\#$ & ok & $\ldots$ & crash & min & mean & max & missing [\%] \\ 
  \midrule
%Pre\_featuretime 		& 115 & 100.00	& $\ldots$ & 0.00 & 0.00 & 0.06 & 1.31 & 0.00 \\ 
%  Basic\_featuretime 		&  14 & 100.00 	& $\ldots$ & 0.00 & 0.00 & 0.00 & 0.07 & 0.00 \\ 
%  KLB\_featuretime 		&  20 & 100.00 	& $\ldots$ & 0.00 & 0.00 & 0.18 & 6.09 & 0.00 \\ 
%  CG\_featuretime 		&  10 & 62.63 	& $\ldots$ & 37.37 & 0.02 & 8.79 & 20.28 & 0.00 \\ 
%  DIAMETER\_featuretime	&   5 & 100.00 	& $\ldots$ & 0.00 & 0.00 & 0.60 & 2.11 & 0.00 \\ 
%  cl\_featuretime 		&  18 & 100.00 	& $\ldots$ & 0.00 & 0.01 & 1.99 & 2.02 & 0.00 \\ 
%  sp\_featuretime 		&  18 & 100.00 	& $\ldots$ & 0.00 & 0.01 & 0.33 & 3.05 & 0.00 \\ 
%  ls\_saps\_featuretime 	&  11 & 100.00 	& $\ldots$ & 0.00 & 1.36 & 2.12 & 2.51 & 0.00 \\ 
%  ls\_gsat\_featuretime 	&  11 & 100.00 	& $\ldots$ & 0.00 & 2.03 & 2.29 & 3.03 & 0.00 \\ 
%  lobjois\_featuretime 	&   2 & 100.00 	& $\ldots$ & 0.00 & 2.00 & 2.00 & 2.27 & 0.00 \\ 
  Pre 		& 115 & 100.00	& $\ldots$ & 0.00 & 0.00 & 0.06 & 1.31 & 0.00 \\ 
  Basic 		&  14 & 100.00 	& $\ldots$ & 0.00 & 0.00 & 0.00 & 0.07 & 0.00 \\ 
  KLB 		&  20 & 100.00 	& $\ldots$ & 0.00 & 0.00 & 0.18 & 6.09 & 0.00 \\ 
  CG 		&  10 & 62.63 	& $\ldots$ & 37.37 & 0.02 & 8.79 & 20.28 & 0.00 \\ 
  DIAMETER	&   5 & 100.00 	& $\ldots$ & 0.00 & 0.00 & 0.60 & 2.11 & 0.00 \\ 
  cl 		&  18 & 100.00 	& $\ldots$ & 0.00 & 0.01 & 1.99 & 2.02 & 0.00 \\ 
  sp 		&  18 & 100.00 	& $\ldots$ & 0.00 & 0.01 & 0.33 & 3.05 & 0.00 \\ 
  ls\_saps 	&  11 & 100.00 	& $\ldots$ & 0.00 & 1.36 & 2.12 & 2.51 & 0.00 \\ 
  ls\_gsat 	&  11 & 100.00 	& $\ldots$ & 0.00 & 2.03 & 2.29 & 3.03 & 0.00 \\ 
  lobjois 	&   2 & 100.00 	& $\ldots$ & 0.00 & 2.00 & 2.00 & 2.27 & 0.00 \\ 
   \bottomrule
\end{tabular}
}
\caption{Feature group summary for the SAT$12$-RAND scenario. The second column shows how many features
  depend on another feature group to be computed first. Percentages of runstatus
  events are followed by summary statistics for group costs.}
\label{table:features}
\end{table}

% \note{ML}{We have to decide whether we want to call it feature group or feature step. Right now, the paper is highly inconsistent wrt this aspect.}
% \note{LK}{I vote for feature group.\\
% ML: Ok, it is feature group now.}

We also check whether instances occur with exactly the same feature values, indicating that the experimenter
might have erroneously run on the same instance twice. %We furthermore list which features are constant across all instances and therefore useless for modelling, although they might result in unwanted cost. They can even result in numerical problems for some machine learning models.

All of the above tables and figures and many more were generated by our online
platform, and are also accessible through the R package \system{aslib}. The
functions are highly configurable and customisable.
\changed{We plan to extend our data analysis with additional techniques, such as
more measures of algorithm performance~\cite{SmithMiles201412}.}

%\note{Reviewer}{Maybe add some techniques from \cite{SmithMiles201412}}

experiments.tex
\section{\changedTwo{Study of Algorithm Selection Techniques}}
\label{sec:experiments}

%\note{Reviewer}{What I did not really understand in the paper is Section 6.
%Yes, it is good to know that one can use standard ML techniques, but
%would it not be more interesting to give some example exploration with
%some state-of-the-art techniques or at least techniques that are specialized
%for the context of algorithm selection?\\
%ML: Should be answered in the response letter.}

%\note{FH}{The intro to this section did not mention the new experiments at all. Rather than being too apologetic about not quite representing state-of-the-art results, I tried to also describe our experiments as one realistic way of characterizing our benchmarks.}
\fh{In this section, we present an exploratory benchmark study that gives an indication of the diversity of our benchmarks. First, we evaluate the performance
of algorithm selectors on our scenarios. We then perform a subset selection study to identify the important algorithms and instance features in each of the scenarios.
We make no claim that the presented experimental settings are exhaustive or that we achieve state-of-the-art 
algorithm selection performance; rather, we provide results that can be used as
a baseline comparison for other approaches.
These results, and our
framework in general, allow us to study which algorithm selection approaches work
well for which of our scenarios.}
%\note{LK}{``\ldots{}these results also serve to study which algorithm selection
%approaches work well for which of our scenarios.''\\
%We have explicitly avoided saying this -- this makes it easy for a reviewer to
%say that we haven't considered method X and therefore any statement about what
%works well and what doesn't is misleading. I would \emph{strongly} advocate
%going back to the previous wording to avoid any implications that this is a
%survey of different approaches to algorithm selection.}


%\pk{In} order to reach the performance of current state-of-the-art algorithm selection systems~\cite{xu2012satzilla2012,malitsky_algorithm_2013}, we would have to include various extensions, such as cost-sensitive classification and complementary techniques such as pre-solving.\footnote{A pre-solver is a default solver that is run for a small amount of time without any algorithm selection
%taking place; cf.\ \cite{xu_satzilla_2008}. The problem instances that are
%solvable in this time are solved without incurring any of the overhead that
%algorithm selection brings, such as the computation of features. This is
%relevant in practice, as the cost of computing features can be much higher than
%the cost of solving a very easy instance.}

We use the \system{LLAMA} toolkit~\cite{kotthoff_llama_2013}, version 0.9.1, in combination with the \system{aslib}
package\footnote{\url{https://github.com/coseal/aslib-r}} to
run the algorithm selection study. \system{LLAMA} is an R~\cite{R}
package
that facilitates many common algorithm selection approaches. In particular, it
enables access to classification, regression, and clustering models for algorithm
selection---the three main approaches we use in our study.
We use the \system{mlr} R
package~\cite{bischl_mlr} as an interface to the machine learning models
provided by other R packages. We parallelize all of our benchmark experiments
through the \system{BatchExperiments} \cite{Bischl2015_1} R package.

In this paper, we only present aggregated benchmark results, but the
interested reader can
access full benchmark results at \url{http://aslib.net}. Our
study is fully reproducible as the complete code to generate these
results is part of the \system{aslib} package.

We use the subset of feature groups that are recommended by the authors of the
respective scenario, called default feature set. For the feature subset selection study, we have used all
feature groups. Detailed and continuously updated information (e.g., the names
of the feature processing groups we selected and their average costs) is
provided on the ASlib website.

\subsection{Experimental setup}

We consider three approaches to algorithm selection that have been studied extensively in the
literature (cf.\ Section~\ref{sec:background:how}): 

\begin{itemize}
  \item \emph{classification} applies a multi-class classifier to directly predict the \changedTwo{best performing algorithm} of the $k$ possible algorithms;
  \item \emph{regression} predicts each algorithm's performance via a regression model and then chooses the one with the best predicted performance;
  \item \emph{clustering} groups problem instances in feature space, then
determines the cost-optimal solver for each cluster and finally assigns to each
new instance the solver associated with the instance's predicted cluster.
\end{itemize} 

\begin{table}[thb]
\centering
\small
\begin{tabular}{p{0.2em}llr}
\toprule[1pt]
& Technical Name & Algorithm and Parameter Ranges & Reference\\
\midrule
\multicolumn{2}{l}{\emph{classification}}\\
& ksvm & support vector machine & \cite{ksvm}\\
& & $C\in [2^{-12}, \; 2^{12}], \; \gamma \in [2^{-12}, \; 2^{12}]$\\
& randomForest & random forest  & \cite{rf}\\
& & $\texttt{ntree}\in [10, \; 200], \; \texttt{mtry}\in [1, \; 30]$\\
& rpart & recursive partitioning tree, CART & \cite{rpart}\\
\midrule
\multicolumn{2}{l}{\emph{regression}}\\
%mars & multivariate adaptive regression splines & \cite{mda}\\
& lm & linear regression & \cite{R}\\
& randomForest & random forest & \cite{rf}\\
& & $\texttt{ntree} \in [10, \; 200], \; \texttt{mtry} \in [1, \; 30]$\\
& rpart & recursive partitioning tree, CART & \cite{rpart}\\
\midrule
\multicolumn{2}{l}{\emph{clustering}}\\
& XMeans & extended $k$-means clustering & \cite{hall_weka_2009}\\
\bottomrule[1pt]
\end{tabular}
\caption{Machine learning algorithms \pk{and their parameter ranges} used for our study.}
\label{tab:mlalgs}
\end{table}

%\note{FH}{Not a big deal, but I find it odd that you
%(a) optimized ntree for random forests (more is always better, it's just more expensive; Bernd, I still don't believe your claim from back then that more trees can hurt in this setting ;-); (b) while you're at it did not optimize the standard RF parameters (how many random features to select at each node, and some parameter restricting depth or number of points required for further splitting); (c) while you're at it did not optimize the ridge parameter of linear regression (or was this unregularized linear regression? That should perform really poorly)}

%\note{PK}{I'll try to answer some of your comments from above: (a) I don't see an advantage of just increasing the number of trees -- even though a RF with more trees tends to be more flexible in a lot of cases, it does not guarantee better results but an increase in memory and decrease in training and prediction speed; (b) as far as I know the RF always builds complete trees, allowing each single tree to overfit completely to its training data (which does not matter because of the bagging approach behind it) -- therefore, the number of trees is the most reasonable tuning parameter of a RF (from my point of view); (c) lm is just the basic version of a linear regression -- this does not use any penalty terms or other parameters as the ridge regression does}

The specific machine learning algorithms we employed for our study are
shown in Table~\ref{tab:mlalgs}. They include
representatives of each of the three major approaches above. 

\changedTwo{The linear model we employ is the best-studied regression method. In its most basic version, it models the data using the linear function $f(x) = \beta^T x + \beta_0$; parameters are obtained by minimizing squared loss. 
The trees constructed by the CART algorithm, which can handle both classification and
regression problem, are grown in a top-down manner and divide the training
data into rectangular regions by axis-parallel splits at each interior node.
Splits are selected by considering label impurity reduction measured by an
impurity function, based, e.g., on the Gini index for classification or squared
loss for regression. Leaf nodes associate the best, but constant, label with
their feature region for prediction. 
Random forests form an ensemble of $ntree$ simpler trees by bootstrapping
multiple data sets from the original one and then fitting a tree for each.
Predictions are made through majority voting for classification and averaging for
regression. Furthermore, ensemble members are decorrelated by randomly selecting
only a few candidate features for each split point (controlled by parameter
$mtry$) in a tree and maximally growing trees without any early stopping or
pruning.
Support Vector Machines perform linear classification in a transformed feature space by maximizing the margin between the positive and negative examples. Parameter $C$ controls the trade-off between the size of the margin and classification loss.
The feature mapping is implicitly built into the algorithm by substituting the regular inner product of the Euclidean space with a so-called kernel. Parameter $\gamma$ is a property of the radial basis function kernel used here. 
% They are especially useful when learning from little data.
The XMeans clustering algorithm is the only unsupervised learning algorithm we
study. It is an extension of the well known $k$-means method to adaptively
select the number of clusters. $k$-means starts with $k$ random cluster centroids,
assigns each point to the nearest centroid, and then iteratively recomputes the
cluster centroids and cluster assignments until convergence. For further details
on all methods the reader is referred to the standard literature~\cite{Hastie2009} and for XMeans to~\cite{Pelleg2000}.
}

We tuned the hyperparameters of ksvm and randomForest (classification and
regression) within the listed parameter ranges, using random search with 250
iterations and a nested cross validation (with three internal folds) to
ensure unbiased performance results.
All other parameters were left at their default values.
For the clustering algorithm, we set the (maximum) number of clusters to 30
after some preliminary experiments; the exact number of clusters is determined
internally by XMeans.

\subsection{Data preprocessing}

%\note{FH}{This is the first time that training data is mentioned; I suggest pulling up the short paragraph on cross-validation from the end of 6.1, and then saying that we preprocessed the data in each training splits as follows. We did do that preprocessing separately for each C/V split, right (not peeking at the entire data)?}
We preprocessed the data as follows.
%ML: Since we have not repetitions in our data, we have not really did this. 
%First, if there was more than one repetition of feature computation per problem
%instance, we computed the mean over all repetitions. 
We removed constant-valued (and therefore irrelevant) features and imputed
missing feature values as the mean over all non-missing values of the
feature.\footnote{For the CSP-MZN-2013 scenario, we also removed the
\texttt{gc\_circuit} feature, which is almost constant.}
\changedTwo{For the clustering methods, we normalized the range of each feature
to the interval $[-1,1]$. The scenarios we consider in this article contain only
continuous features.} The other machine learning methods that require normalized
data perform this internally (for example the SVMs).
\pk{Missing performance values were imputed using the timeout value of the
scenario.}
%ML: Not part of our experiments
% or, if no
%timeout was given, ten times the worst performance recorded in the data set. 
%ML: This should be ensured by our format definition.
%We considered an
%algorithm run successful if the run status was ``ok'' and the
%corresponding performance value was not missing.

For each problem instance, we calculated the total feature computation cost
as the sum of the costs of all feature groups, in the order specified in the
definition of the scenario.
If the problem
instance was solved during feature computation \changedTwo{(e.g., using
SLS-probing features~\cite{xu_satzilla_2008})}, we only considered the cost of
the feature groups up
to the one that solved it. Furthermore, we set the runtime
for all algorithms to zero for instances solved during feature
computation. If the instance was not solved during feature computation, we added the feature costs computed in this way to the runtimes
of the individual algorithms on the respective instances \changedTwo{($c(f_{i}) + t(i,a)$)}. 
Given these new runtimes, we checked whether the specified timeout was now
exceeded and set the run status of any corresponding algorithm accordingly. 
Preprocessing runtimes to include feature computation time in this way allows us
to focus on an algorithm selection system's overall performance, and avoids overstating the fraction of instances that would be solved within a time budget in cases where features are expensive to compute. 

Each scenario specifies a partition into $10$ folds for cross-validation to
ensure consistent evaluation
across different methods. We used this partition in our study.
%The code used to run the experiments is
%part of the \system{aslib} package.

\subsection{Evaluation}

\changedTwo{\noindent We evaluated different algorithm selection models using three different
measures:} 

\begin{itemize}
  \item the fraction of all instances solved within the timeout;
  \item the penalized average runtime with a penalty factor of 10, i.e., a
  timeout counts as 10 times the timeout;
  \item the average misclassification penalty, which, for a given instance, is the difference between the performance of the selected algorithm and the performance of the best algorithm. 
\end{itemize}

\pk{The} performance of each algorithm selection model \pk{was compared} to the virtual best
solver (VBS) and the single best solver. The virtual best solver selects the
best solver from $\mathcal{A}$ for each instance
\changedTwo{($\forall i \in \mathcal{I}: \argmax_{a \in \mathcal{A}} m(i,a)$)}.
Note that the misclassification penalty for VBS is
zero by definition. The single best solver is the actual solver that has the
overall best performance on the data set
\changedTwo{($\argmax_{a \in \mathcal{A}} \sum_{i \in \mathcal{I}} m(i,a)$)}.
Specifically, we consider the solver
with the best PAR10 score over all problem instances in a scenario.%


\subsection{Results}

%\note{Reviewers}{Further analysis, including costs of feature selection and preprocessing, might be considered.}

Figure~\ref{fig:res-heat} presents a summary of the results of our study. In
most cases, the algorithm selection approaches performed better than the single
best solver. We expected this, as most of our data sets come from 
publications that advocate algorithm selection systems.

Nevertheless, there were significant differences between the scenarios. 
While almost all algorithm selection approaches outperformed the single
best algorithm, there are some scenarios that seem to be much harder for algorithm
selection. In particular, on the \pk{SAT12-INDU} scenario, \pk{three}
approaches were not able to achieve a performance improvement.

\begin{figure}[t]
\includegraphics[width=\textwidth]{pics/res-heat-diff}
\caption{Summary of the results of our study. 
%The shading represents the factor by which the selection model outperformed the PAR10 single best solver, measured in terms of PAR10 score and presented on a log scale. Values greater than 1 mean that the selector was better than the single best solver. Values less than 1, i.e. where the selector was worse than the single best solver, are shown in \textit{italics}. The performance for the best model for each data set is shown in \textit{\textbf{bold italics}}. The geometric mean speedup for each selection model type across all scenarios is shown on top of the heatmap.
\pk{We show how much of the gap between the single best and the virtual best
solver in terms of PAR10 score was closed by each model. That is, a value of 0
corresponds to the single best solver and a value of 1 to the virtual best.
Negative values indicate performance worse than the
single best solver. Within each data set, the best model is annotated with an
asterisk.
The shading emphasizes that comparison: green
cells correspond to values close to 1 (i.e.,\ close to the virtual best solver),
whereas red shows the models with bad performance. White shading indicates
values close to 0, i.e.\ the model has the same performance as the single best
algorithm. The arithmetic mean for each model type
across all scenarios is given in parentheses after the model name.}
}
\label{fig:res-heat}
\end{figure}

Random regression forests stood out \changedThree{quite clearly as} the best overall approach,
achieving the best performance on $13$ of the $17$ datasets. This is in line
with recent results showing the strong performance of this model for algorithm
runtime prediction~\cite{hutter2014algorithm}. The results are also consistent
with those of the original papers introducing the datasets.
% LK: This doesn't actually support the point above.
%For example,
%\citet{xu2012evaluating} reported somewhat better results for the three SAT$11$
%datasets than achieved here with our methods (which is to be expected
%since their latest SATzilla version used a cost-sensitive approach and
%pre-solving schedules).

XMeans performed worst on average.
On some scenarios, it performed well, in particular SAT11-RAND,
MAXSAT12-PMS, and PROTEUS-2014.
However, on SAT12-ALL, SAT12-INDU, and SAT15-INDU
XMeans performed worse than the single best solver. 
The default subset of instance features appears to be unfavorable for XMeans on industrial SAT instances. 


%\subsection{Summary}
%\note{FH}{I moved the following from being its own section (6.6 Summary) to here, since this only relates to these experiments not the the forward selection ones. It doesn't hurt quite as much here, but as the summary section of the entire selection experiment section it was extremely anti-climactic to say that basically the experiments are a repeat of those in \emcite{kotthoff_evaluation_2012}. I think it would be actually be much much better yet to move this right to the beginning, in Section 6.0; something along the lines of ``We follow the setup of \emcite{kotthoff_evaluation_2012} to evaluate a range of AS approaches on our algorithm selection scenarios.'' Then, readers know what's coming and don't feel fooled. The part comparing the results to those in \emcite{kotthoff_evaluation_2012} could just remain here...}

%Our focus in the experiments just presented has been on demonstrating the flexibility and potential of ASlib.
%However, we note that the results presented here are similar to those presented
%by \emcite{kotthoff_evaluation_2012}, with some overlap in data sets and
%methodology. Some important differences are that the current work considered a
%number of additional approaches, \pk{found that it is more
%challenging to improve the SAT$11$-INDU data compared to} than the SAT$11$-HAND and 
%SAT$11$-RAND data sets, and achieved better results on the CSP-$2010$ scenario.
%%
%With both, the data sets and the code for the above evaluation available, it
%is easy for interested parties to reproduce our results and compare to their own
%approaches.



\subsection{Algorithm and Feature Subset Selection}

To provide further insight into our algorithm selection scenarios,
we applied forward selection~\cite{Kohavi97} to the algorithms and features to
determine whether smaller subsets still achieve comparable performance.
We performed forward search independently for algorithms and features for each
scenario. \changedTwo{Forward selection is an iterative selection algorithm whose the first iteration starts with an empty set of algorithms and features; in each subsequent iteration, it greedily adds the 
algorithm or feature to the set which most improves the cross-validated score (PAR$10$) of the predictor.}
The selection process terminates when the score does not improve by at least
$\varepsilon$. We set $\varepsilon = 1$, which roughly corresponds to an
improvement of 1 second per instance.
In all other aspects, the experimental setup was the same as described before.

We used random regression forests\footnote{We
used random forests with default parameters, as the tuning was done for the
set of features specified by the scenario authors and the full set of solvers.}, as it was the best overall approach so far.
We note that the selection results use \changedTwo{standard cross validation rather} than the 
nested version, which may result in \fh{overconfident performance estimates for
the selected subsets~\cite{Bischl2012_3}}.
We accept this caveat since our goal here is to study the ranking of the features and the size of the selected sets,
and a more complex, nested approach would have resulted in multiple selected sets.
% We note that the presented results depend on the methodology and machine
% learning algorithm used, and the selection results use normal resampling and not the
% nested version, which may result in \fh{overconfident performance estimates for the selected subsets~\cite{Bischl2012_3}.
% We accept this caveat since our goal here is not to improve performance itself, but rather to estimate how far one can reduce the set of algorithms and features in a real-world setting.}
%\pk{Instead of improving the performance itself, we want to estimate, how far one can} reduce the set of algorithms and features in a real-world setting.

% \note{BB}{I dislike the wording here wrt to the biased results, I should reword a bit!\\FH: Reworded; please adapt as you see fit ...}

\begin{table}[thb]
\begin{center}
\begin{tabular}{lccc}
\toprule
\input{results_selection-algorithms}
\end{tabular}
\caption{PAR$10$ scores for the set of all algorithms and the reduced set, along
with the number of all algorithms and the size of the reduced portfolio.}
\label{tab:asel}
\end{center}
\end{table}    

\begin{table}[thb]
\begin{center}
\begin{tabular}{lcccc}
\toprule
\input{results_selection-features}
\end{tabular}
\caption{PAR$10$ scores for the set of all features, the default feature set and the reduced set, along
with the number of all features and the size of the reduced feature set.}
\label{tab:fsel}
\end{center}
\end{table}    

Tables~\ref{tab:asel} and~\ref{tab:fsel} present the results of forward selection for
algorithms and features on all scenarios.
Usually, the number of selected features is very small compared to the complete
feature set.
This is consistent with the observations of \changedTwo{\emcite{RobertsH09} and} \emcite{hutter_identifying_2013} 
who found in their experiments that only a few instance features are necessary
to reliably predict the runtime of their algorithms. For example, on SAT12-RAND,
the only three features selected were
a balance feature concerning the ratio of positive and negative occurrences of
each variable in each clause and
two features based on survey propagation.
%are POSNEG\_RATIO\_CLAUSE\_min, POSNEG\_RATIO\_VAR\_min, and SP\_unconstraint\_max.
%\note{FH}{I reworded to avoid giving the names of the features here without explaining what they are.
%The latter one is a feature based on survey propagation: for each variable, use probabilistic inference to estimate the probability that the variable is unconstrained (false in some satisfying assignment, true in another); then, take the max of the probabilities across the variables.
%The first and second concern balance features: the ratio of positive to negative literals in each clause (take the min over all clauses);
%and the ratio of positive to negative occurrences of each variable (take the min over all variables).
%These are not features that have come up as important frequently, so, unfortunately, I don't have a whole lot of intuition to interpret this finding ... Holger/Kevin, the only place where POSNEG\_ratio\_var\_mean has come up before was in your work with Lin on predicting solubility... do you have any intuitions from that (or other) work about the importance of POSNEG\_RATIO\_CLAUSE\_min, POSNEG\_RATIO\_VAR\_min, and SP\_unconstraint\_max?}

The number of algorithms after forward selection is also substantially reduced
on most scenarios. On the SAT scenarios, we expected to see this because the
scenarios consider a huge set of SAT solvers that were not pre-selected in any
way. \emcite{xu2012evaluating} showed  
that many SAT solvers are strongly correlated 
and make only very small contributions to the VBS, \changedTwo{a finding that is corroborated by our results (see Figure \ref{fig:cormat} in Section~\ref{sec:eda})}. For
example, on the SAT12-RAND scenario, only two solvers were selected: sparrow and
eagleup.
We did not expect the set of algorithms to be reduced on the
ASP-POTASSCO scenario, as the portfolio was automatically constructed using
algorithm configuration to obtain a set of complementary parameter settings that
are particularly amenable to portfolios; nevertheless only 5 of 11
configurations were chosen by the forward selection.

% LK: I think this is too detailed, in particular because we don't do anything
% similar for the feature selection.
%Looking in more detail at the selected algorithm configurations,
%the configurations of the last $2$ and the $5$th iteration were reduced.
%The reason for the removed configuration could be that
%\emcite{holisc14a} used a different set of instances for the configuration,
%that does not generalize perfectly on the instances used here,
%and the performance improvement was small in the last iterations of \system{Hydra}.

Our results indicate that in real-world settings, selecting the most predictive
features and the solvers that make the highest contributions can be important.
More detailed and continuously updated results can be found on the ASlib
website.

%\note{ML}{Again, add an appendix with all plots and tables?}

introduction.tex
%BB: I don't really think it helps the argument to refer to NFL. Quite the opposite. Algorithm selection models would itself be subject to NFL from a theoretical standpoint. And NFL is a useless and irrelevant theorem for what we are doing IMHO.
%ML: What is NFL? National Football League?
%BB: No Free Lunch of course. We removed that argument due to the problems above
%HH: NFL doesn't strictly apply anyway, since we are talking about problems with compact representations, not arbitrary blackbox functions

\section{Introduction}
\label{sec:intro}

%\note{Reviewers}{All three reviewer express their concerns about novelty and significance
%...too little significant new insights being gained 
%...include significant analysis of the collected examples}

Although NP-complete problems are widely believed to be intractable in the worst case, it is often possible to solve even very large instances of such problems that arise in practice. This is fortunate, because such problems are ubiquitous in Artificial Intelligence applications. There has thus emerged a large subfield of AI devoted to the advancement and analysis of heuristic algorithms for attacking hard computational problems. Indeed, quite surprisingly, this subfield has made consistent and substantial progress over the past few decades, with the newest algorithms quickly solving benchmark problems \changedTwo{that were beyond reach until recently}. The results of the international SAT competitions provide a paradigmatic example of this phenomenon. Indeed, the importance of this competition series has gone far beyond documenting the \hh{progress achieved by the SAT community in solving difficult and application-relevant SAT instances}---it has been instrumental in driving research itself, helping the community to coalesce around a shared set of benchmark instances and providing an impartial basis for determining which new ideas yield the biggest performance gains.

The central premise of events like the SAT competitions is that the research community ought to build, identify and reward single solvers that achieve strong across-the-board performance. However, this quest appears quixotic: 
%FH: de-capitlized "quixotic"; see http://en.wikipedia.org/wiki/Proper_adjective 
most hard computational problems admit multiple solution approaches, none of which dominates all alternatives across multiple problem instances. In particular, this fact has been observed to hold across a wide variety of AI applications, including
% This fact has been theoretically proven for discrete blackbox function optimization~\cite{wolpert_no_1997}, 
%In particular, this fact is empirically well supported in many sub-communities of AI, for example those dealing with 
propositional satisfiability (SAT)~\cite{xu2012evaluating}, constraint
satisfaction (CSP)~\cite{cphydra}, planning~\cite{howe_exploiting_1999,helmert_fast_2011}, and
supervised machine
learning~\cite{ThoHutHooLey13-AutoWEKA,Vanschoren2012,feurer-nips15a}. 
An alternative is to accept that no single algorithm will offer the best performance on all instances, and instead aim to identify a portfolio of complementary algorithms and a strategy for choosing between them \citep{rice_algorithm_1976}. To see the appeal of this idea, consider the results of the sequential application (SAT+UNSAT) track of the $2014$ SAT Competition.\footnote{\url{http://www.satcompetition.org/2014/results.shtml}} The best of the $35$ submitted solvers, \texttt{Lingeling ayv}~\cite{lingeling}, solved $77\%$ of the $300$ instances.
However, if we could somehow choose the best among these $35$ solvers on a per-instance basis, we would be able to solve $92\%$ of the instances. 

%\note{FH}{It felt like some text got lost here, but it was probably just suboptimal flow. I added a sentence to actually mention what algorithm selection is before talking about algorithm selectors, and now it flows OK.}
\fh{Research on this \emph{algorithm selection problem}~\citep{rice_algorithm_1976} has demonstrated the practical feasibility of using machine learning for this task.}
In fact, although practical algorithm selectors occasionally \fh{choose suboptimal algorithms}, \hh{their performance can 
	get close to that of an oracle that always makes the best choice.}
The area \hh{began to attract} considerable attention when methods based on algorithm selection began to outperform standalone solvers in SAT competitions~\cite{xu_satzilla_2008}. Algorithm selectors have since come to dominate the state of the art on many other problems, including
%~\cite{gomes_algorithm_2001} 
%HH: this reference is, IMO, unsuitable - it is 13 years old and does not tell a real success story. I've therefore removed it. (I know how you meant it, but it is easily misunderstood) The right place for this reference is in related work; I've also cited it later in the paragraph, where it fits better.
%that, typically based on per-instance algorithm selection, dominate the state of the art in SAT~\cite{xu_satzilla_2008}, 
CSP~\cite{cphydra}, planning~\cite{helmert_fast_2011}, Max-SAT~\cite{malitsky_evolving_2013}, QBF~\cite{pulina_self-adaptive_2009}, and ASP~\cite{gebser_portfolio_2011}.


%\note{FH}{I moved a discussion of Exploratory Landscape Analysis to 2.3 (Selection Enablers); not sure how it got here, it was very much out of place (commented out in source). We should, however, add continuous blackbox optimization~\cite{Kerschke2014} to the list above, no?}
%FH: moved this to 2.3:
%For continuous blackbox optimization, algorithm selection can be performed based on Exploratory Landscape Analysis
%\cite{Mersmann2013,Bischl2012_2,Kerschke2014}, 
%a technique for deriving features by numerically describing the landscapes of such optimization problems. 

%\changed{To illustrate the power and potential of per-instance algorithm selection, let us consider the concrete case of SAT solving.\footnote{While SAT is arguably the best-studied application domain for per-instance algorithm selection techniques
%and other portfolio methods, similarly substantial performance gains can be achieved on many other problems.}
%SAT is a mature field, and there exist a plethora of SAT algorithms and SAT benchmark instances. 
%\note{ML/Reviewer}{I changed the following paragraph such that we talk about the most recent results of the 2014 SAT Competition.}
%Closing this gap between the best single solver and the oracle is the aim of per-instance algorithm selection.}

% Restricting ourselves to instances used in international SAT competitions (which have been held regularly since 2002), there are roughly 4\,260 instances 
% %HH: use international convention for separators - "," is used for decimal point in some countries ;-)
% from various tracks (random, handcrafted, and industrial). With a time limit of 5\,000 seconds, the best of the 29 top-performing solvers submitted to the 2012 SAT Challenge, mphaseSAT~\cite{mphaseSAT}, solves 2\,605 of these instances ($\approx 61$\%). However, if we had access to an oracle that selects the fastest solver for 
% any given SAT instance without incurring any selection cost, we would be able to solve 3\,776 instances ($\approx 86$\%). 
% %FH: commented this sentence; the numbers don't add up?
% %This means that using only the technology available in 2012, it is possible to have a system that solves an additional 979 (20\%) instances. 
% Closing this large gap between the best single solver and the oracle solver is the task pursued by per-instance algorithm selection.\footnote{In fact, readers familiar with mphaseSAT will note that, inspired by algorithm portfolio techniques \cite{gomes_algorithm_2001}, it applies several 
% 	different phase selection heuristics which make it quite robust, albeit slower than other basic solvers (it was not the
% 	best solver for any of the 4\,260 instances reported here, but was nevertheless the best overall single solver). The best single solver not related to mphaseSAT is CCSAT~\cite{CCSAT}, which only solves 1\,861 instances 
% 	($\approx 44$\%), demonstrating the potential benefit of per-instance algorithm selection even more clearly.
% }
% While SAT is arguably the best-studied application domain for per-instance algorithm selection techniques
% and other portfolio methods, similarly substantial performance gains can be achieved on many other problems.

%The fundamental underlying reasoning driving algorithm selection is that there is no single solver that dominates on every instance. In fact, in the case of the SAT example, mphaseSATm is never the best performing solver on any instance. Instead, it seems that in aiming to be a general purpose solver, it has to sacrifice on some. While this is an extreme example, it is indicative of what happens in practice. The task of algorithm selection is therefore to use some characterizing set of features describing an instance, to select the best performing solver. 
%
%The methods used to choose the best solver for a particular instance, however, vary widely. There are techniques that fit a regression model for each solver that map the features to performance, selecting the expected best~\cite{?}. Alternatively, instances can be grouped based on their features, with a single solver assigned to each cluster. During runtime, the instance is assigned to a cluster and solved with the appropriate solver~\cite{?}. Ranking techniques can also be used to order candidate solvers for each instance~\cite{?}. Classification techniques like a random forest can be trained to rate between every pair of solvers, selecting the one voted to win most frequently~\cite{?}. Alternatively, a random forest can be trained where each tree is specifically constructed to split the training data where instances falling into each node better agree on the best solver than they did in the parent node~\cite{?}. For a thorough overview of the topic, we refer the reader to the following survey~\cite{?}.

To date, much of the progress in research on algorithm selection has been
demonstrated in algorithm competitions originally intended for
non-portfolio-based (``standalone'') solvers. This has given rise to a variety
of challenges for the field. First, benchmarks selected for such competitions tend
to emphasize problem instances that are currently hard for existing standalone
algorithms (to drive new research on solving strategies) rather than the wide
range of both easy and hard instances that would be encountered in practice
(which would be appropriately targeted by researchers developing algorithm
selectors). Relatedly, benchmark sets change from year to year, making it
difficult to assess the progress of algorithm selectors over time. Second,
although competitions often require entrants to publish their source code, none
require entries based on algorithm selectors to publish the code used to
\emph{construct} the algorithm selector (e.g., via training a machine learning
model) or to adhere to a consistent input format. Third, overwhelming competition
victories by algorithm selectors can make it more difficult for new standalone
solver designs to get the attention they deserve and can thus create resentment
among solver authors. Such concerns have led to a backlash against the
participation of portfolio-based solvers in competitions; for example,
\hh{starting in 2013 solvers that explicitly combine more than two component
algorithms have been excluded from the SAT competitions}.
\changedTwo{For similar reasons, there is a specific prize for non-portfolio solvers
in the learning track of the International Planning Competition~\cite{VallatiCGMRS15}.}

The natural solution to these challenges is to evaluate algorithm selectors on
their own terms rather than trying to shoehorn them into competitions intended
for standalone solvers. This article, written by a large set of authors active in research
on algorithm selectors, aims to advance this goal by introducing a set of
specifications and tools designed to standardize and facilitate such evaluations. Specifically, we propose a benchmark library, called ASlib, tailored to the cross-domain evaluation of algorithm selection techniques. 
%FH: The following sentence is new, to address Holger's comment.
%Indeed, our library is not limited to the algorithm selection problem specified in Definition \ref{def:algo_sel}, but also supports other related ways algorithm portfolio scenarios.
%
\changedTwo{In Section~\ref{sec:spec}, we provide a summary of the data format specification used in ASlib}
that covers a wide variety of foreseeable evaluations. To date, we have instantiated this specification with
benchmarks from six different problem domains, which we describe in Section~\ref{sec:scenarios}.
% FH: too much detail for the intro. Cut.
% This includes, but is not limited to, situations that explore the time benefits of excluding and not computing certain features. Or experiments into techniques  simultaneously pursuing multiple objectives. 
However, we intend for ASlib to grow and evolve over time.
Thus, our article is accompanied by an online repository
(\url{http://aslib.net}), which accepts submissions from any researcher. Indeed,
we \changedTwo{already} included scenarios that have been submitted by \changedTwo{contributors outside the core group of ASlib maintainers}.
%FH: I reworded this since I found it odd to say that there are people who submitted scenarios but are not involved with ASlib (submitting is a way of involvement).

Our system automatically checks newly submitted datasets to verify that they adhere to the
specifications and then provides an overview of the data, including the results
of some straightforward algorithm selection approaches based on regression, clustering and classification.
We provide some examples of these automatically-generated overviews and benchmark results in Sections \ref{sec:eda} and \ref{sec:experiments}.
All code used to parse the format files, explore the algorithm selection scenarios and run benchmark machine
learning models on them is publicly available in a new R package dubbed
\system{aslib}.\footnote{This package is currently hosted at
	\url{https://github.com/coseal/aslib-r}. We will submit it to the official R package server CRAN alongside the final version of this article.}
\changedTwo{In Section~\ref{sec:comp}, we discuss two recent examples of competition settings using ASlib, along with their advantages and disadvantages.}

Overall, our main objective in creating ASlib is the same as that of an algorithm competition: to allow
researchers to compare their algorithms systematically and fairly, without having to replicate someone else's system or to personally collect
raw data. We hope that it will help the community to obtain an unbiased understanding of
the strengths and weaknesses of different methodologies and thus to improve the current state of the art in per-instance algorithm selection.


%The methods used to attack the algorithm selection problem vary widely, ranging from explicit classification approaches that directly pick a solver for each instance to more indirect approaches, such as predicting the runtime of each solver and then selecting the one predicted to be fastest.
%
%Systems based on either of the many possible methods to solve the algorithm selection problem usually substantially outperforms the single best solver in the domain being investigated, but, as of yet, there has not been an across the board comparison of all the different methodologies. The 

%Another barrier to research on algorithm selection is the considerable upfront cost to new researchers entering the field. Currently, they first need to acquire and compile a diverse set of solvers, a feature set, and a set of problem instances of interest. They then need to measure the performance of each solver on each instance. \changed{In the SAT example above, this would involve running 35 
%solvers on 300 instances, each with a timeout of 5\,000 seconds, which, at a cost of 281 CPU days, necessitates access to substantial computing resources.} %ML: real cost data from SAT Competition website
%\note{KLB}{But the runtime results are actually on the SAT Competition website. So it's not that compelling to say that new researchers would have to rerun these experiments.}



macro.tex

\newcommand{\qm}{``?'' }
\newcommand{\filename}[1]{\textit{#1}}
\newcommand{\stringval}[1]{``#1''}
% \newcommand{\equote}[1]{``#1''}
\newcommand{\colname}[1]{``#1''}

\newcommand{\eg}{e.g.,}
\newcommand{\ie}{i.e.,}
\newcommand{\wrt}{w.\,r.\,t.\ }

\newcommand{\system}[1]{\textit{#1}}

%\newcommand{\changed}[1]{{\color{red}{#1}}}
\newcommand{\changed}[1]{{#1}}

%\newcommand{\changedTwo}[1]{{\color{red}{#1}}}
\newcommand{\changedTwo}[1]{{#1}}

%\newcommand{\changedThree}[1]{{\color{green}{#1}}}
\newcommand{\changedThree}[1]{{#1}}


\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}

paper.tex
% http://aij.ijcai.org/competition-section

%%This is a very basic article template.
%%There is just one section and two subsections.
\documentclass[]{elsarticle}

\usepackage[hyphens]{url}
\usepackage[hidelinks]{hyperref}
%\usepackage{url}
\usepackage{listings}
\usepackage[usenames,dvipsnames]{color}
%\usepackage{color}
\usepackage{csquotes}
\usepackage{dsfont}
\usepackage{caption}
\usepackage{subfig}
\usepackage{microtype}

\usepackage{booktabs}
\usepackage{pbox}
\usepackage{soul}
%\usepackage{changes}
\usepackage[final]{changes}

\biboptions{numbers,sort&compress}

\usepackage{tikz}
\usetikzlibrary{shapes.geometric}
\usetikzlibrary{positioning,shapes,shadows,arrows,calc}
\usetikzlibrary{intersections}
\pgfdeclarelayer{background}
\pgfdeclarelayer{foreground}
\pgfsetlayers{background,main,foreground}

\newcommand{\note}[2]{
	\noindent
	\vspace{0.25cm}
	\fcolorbox{Red}{Apricot}{\parbox{0.99\textwidth}{\textbf{NOTE (#1):\\} #2}}
	\vspace{0.25cm}
}
\newcommand{\emcite}[1]{\citeauthor{#1} (\citeyear{#1})}
% Uncomment the following line to hide notes.
%\renewcommand{\note}[2]{}

% \newcommand{\hh}[1]{\textcolor{blue}{#1}}
% \newcommand{\fh}[1]{\textcolor{green}{#1}}
% \newcommand{\pk}[1]{\textcolor{cyan}{#1}}
\newcommand{\hh}[1]{#1}
\newcommand{\fh}[1]{#1}
\newcommand{\pk}[1]{#1}


% Commands for theorems, lemmas, definitions, etc.
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amssymb}
\newtheorem{thm}{Theorem}%[section]
\newtheorem{lem}[thm]{Lemma}
\newtheorem{prop}[thm]{Proposition}
\newtheorem{cor}[thm]{Corollary}
\newtheorem{obs}[thm]{Observation}
%\theoremstyle{definition}
\newtheorem{define}[thm]{Definition}

\lstset{frame=lines,framesep=3pt,numbers=left,numberblanklines=false,basicstyle=\ttfamily\small}

\include{macro}

\usepackage{todonotes}

\begin{document}

\title{ASlib: A Benchmark Library for Algorithm Selection}




%core group, cooperators; alphabetically ordered

\address[muc]{LMU Munich, Germany}
\address[muenster]{University of M\"unster, Germany}
%\address[cork]{Insight Centre for Data Analytics, University College Cork, Ireland}
\address[freiburg]{University of Freiburg, Germany}
\address[vancouver]{University of British Columbia, Vancouver, Canada}
\address[paderborn]{University of Paderborn, Germany}
\address[tue]{Eindhoven Institute of Technology, Netherlands}
\address[ibm]{IBM Research, United States}


\author[muc]{Bernd Bischl}
\ead{bernd.bischl@stat.uni-muenchen.de}

\author[muenster]{Pascal Kerschke}
\ead{kerschke@uni-muenster.de}

\author[vancouver]{Lars Kotthoff}
\ead{larsko@cs.ubc.ca}

\author[freiburg]{Marius Lindauer}
\ead{lindauer@cs.uni-freiburg.de}

\author[ibm]{\mbox{Yuri Malitsky}}
\ead{yuri.malitsky@gmail.com}

%%%%%%%%%%%%%%%%%

\author[vancouver]{Alexandre Fr\'{e}chette} % (?) - helped to reformat the satzilla data
\ead{afrechet@cs.ubc.ca}

\author[vancouver]{Holger Hoos} % discussion about specification
\ead{hoos@cs.ubc.ca}

\author[freiburg]{Frank Hutter} % discussion about specification
\ead{fh@cs.uni-freiburg.de}

\author[vancouver]{\mbox{Kevin Leyton-Brown}} % discussion and writing
\ead{kevinlb@cs.ubc.ca}

\author[paderborn]{Kevin Tierney} % discussion about specification - maybe data set
\ead{tierney@dsor.de}

\author[tue]{Joaquin Vanschoren} % related work and data set about ml
\ead{j.vanschoren@tue.nl}




\begin{abstract}
% Re-draft 1 of first sentence: too clunky and long
% Different algorithms exhibit varying performance when solving a particular
% problem instance. The algorithm selection involves choosing an
% algorithm from a set of algorithms that most effectively solves a particular %%% Note: I (KBT) am explicitly avoiding algorithm schedules here to avoid complicating the abstract
% problem instance. The algorithm selection problem is attracting increasing
% attention from researchers and practitioners in AI.
The task of algorithm selection involves choosing an algorithm from a set of
algorithms on a per-instance basis in order to exploit the varying performance
of algorithms over a set of instances. The algorithm selection problem is
attracting increasing attention from researchers and practitioners in AI.
Years of fruitful applications in a number of domains have resulted in a large
amount of data, but the community lacks a standard format or repository for
this data. This situation makes it difficult to share and compare different
approaches effectively, as is done in other, more established fields. It also
unnecessarily hinders new researchers who want to work in this area.
%s
To address this problem, we introduce a standardized format for representing algorithm selection scenarios
and a repository that contains a growing number of data sets from the
literature. Our format has been designed to be able to express a wide variety of
different scenarios. \changedTwo{To demonstrate} the breadth and power of our platform, we describe a study
that builds and evaluates algorithm selection models through a common
interface. The results display the potential of algorithm selection to achieve significant performance improvements \hh{across a broad range of problems and algorithms.}
%\note{FH}{The abstract does not address the relationship to competitions -- that's fine by me, I just wanted to point it out in case that got left out.}
\end{abstract}

\begin{keyword}
algorithm selection \sep machine learning \sep empirical performance estimation 
\end{keyword}

\maketitle

\input{introduction}
\input{related_work}
\input{specification}
\input{current_scenarios}
\input{eda}
\input{experiments}
\input{conclusion}

\bibliographystyle{elsarticle-harv}
\bibliography{local}

\end{document}

as_tikz_complex.tex
\tikzstyle{activity}=[rectangle, draw=black, rounded corners, text centered, text width=9em, fill=white, drop shadow]
\tikzstyle{data}=[rectangle, draw=black, text centered, fill=black!10, text width=9em, drop shadow]
\tikzstyle{myarrow}=[->, thick]
\begin{tikzpicture}[align=center,node distance=3.7cm]
	
	\node (perf) [data] {Performance Data: $\mathcal{I} \times \mathcal{A} \to \mathbb{R}$};
	\node (feat) [data, below of=perf, yshift=2.3cm] {Feature Data: $f: \mathcal{I} \to \mathcal{F}$};
	\node (cost) [data, below of=feat, yshift=2.3cm] {Feature Costs: $c: \mathcal{P}(\mathcal{I}) \to \mathbb{R}$};
	
	\node (train) [activity, right of=feat, text width=10em] {Train\\ $\mathcal{F} \to (\phi: \mathbb{N} \to \mathcal{A} \times \mathbb{R}_{0}^+)$};
	
	\node (run) [activity, right of=train, xshift=0.5cm] {Select\\ $a_1, t_1 = \phi_{f(i)}(1)$ and apply $a_1$ for time $t_1$ to $i$};
	\node (run2) [activity, right of=run, xshift=0.5cm] {Select\\ $a_2, t_2 = \phi_{f(i)}(2)$ and apply $a_2$ for time $t_2$ to $i$};
	
	\node (feat_i) [activity, above of=run, yshift=-2.0cm] {Compute Features $f(i)$ of $i$};
	\node (inst) [data, above of=feat_i, yshift=-2.3cm] {(New) Instance $i$};
	
	\node (eva) [activity, below of=run, yshift=1.5cm] {Evaluate $m(i, \phi_{f(i)}(1))$ and $c(f(i))$};
	\node (eva2) [activity, below of=run2, yshift=1.5cm, text width=11em] {Evaluate $\sum_{j=1}^{2}m(i, \phi_{f(i)}(j))$ and $c(f(i))$};
	
	\draw[myarrow] (perf) -| (train.north);
	\draw[myarrow] (feat) -- (train.west);
	\draw[myarrow] (cost) -| (train.south);
	
	\draw[myarrow] (inst) -- (feat_i);
	\draw[myarrow] (feat_i) -- (run);
	\draw[myarrow] (train) -- (run);
	\draw[myarrow] (run) -- node [left] {success} (eva);
	\draw[myarrow] (run) -- node [above] {fail}  (run2);
	\draw[myarrow] (run2) -- node [left] {success} (eva2);
	\draw[myarrow] (run2) -- node [above] {fail} ($(run2.east)+(1.0,0)$);
	
	\path (train.east -| train.east)+(0.4,4.2) node (a) {};
    \path (train.east |- train.east)+(0.4,-3.5) node(b) {};
    \draw[dashed, color=black!60] (a) -- (b);
    \path[color=black!60] (train.east -| train.east)+(-0.3,3.8) node (a) {Offline};
    \path[color=black!60] (train.east -| train.east)+(1.0,3.8) node (a) {Online};
	
\end{tikzpicture}
as_tikz.tex
\tikzstyle{activity}=[rectangle, draw=black, rounded corners, text centered, text width=9em, fill=white, drop shadow]
\tikzstyle{data}=[rectangle, draw=black, text centered, fill=black!10, text width=9em, drop shadow]
\tikzstyle{myarrow}=[->, thick]
\begin{tikzpicture}[align=center,node distance=3.7cm]
	
	\node (perf) [data] {Performance Data: $\mathcal{I} \times \mathcal{A} \to \mathbb{R}$};
	\node (feat) [data, below of=perf, yshift=2.3cm] {Feature Data: $f: \mathcal{I} \to \mathcal{F}$};
	\node (cost) [data, below of=feat, yshift=2.3cm] {Feature Costs: $c: \mathcal{F} \to \mathbb{R}$};
	
	\node (train) [activity, right of=feat] {Train\\ $s: \mathcal{F} \to \mathcal{A} \times \mathcal{R}$};
	
	\node (run) [activity, right of=train, xshift=0.5cm] {Select $a, r = s(f(i))$ and apply $a$ to $i$ with resources $r$};
	\node (feat_i) [activity, above of=run, yshift=-2.0cm] {Compute Features $f(i) \in \mathcal{F}$ of $i$ for cost $c(f(i))$};
	\node (inst) [data, above of=feat_i, yshift=-2.3cm] {(New) Instance $i$};
	
	\node (eva) [activity, below of=run, yshift=2.3cm] {Evaluate $m(i, a)$};
	
	\draw[myarrow] (perf) -| (train.north);
	\draw[myarrow] (feat) -- (train.west);
	\draw[myarrow] (cost) -| (train.south);
	
	\draw[myarrow] (inst) -- (feat_i);
	\draw[myarrow] (feat_i) -- (run);
	\draw[myarrow] (train) -- (run);
	\draw[myarrow] (run) -- (eva);
	\draw[myarrow, dashed, color=black!60] (eva.east) -- ($(eva.east)+(0.4,0.0)$) |- node [right, yshift=-0.7cm, color=black!60] {Feedback} (run.east);
	
	\path (train.east -| train.east)+(0.4,4.2) node (a) {};
    \path (train.east |- train.east)+(0.4,-2.5) node(b) {};
    \draw[dashed, color=black!60] (a) -- (b);
    \path[color=black!60] (train.east -| train.east)+(-0.3,3.8) node (a) {Offline};
    \path[color=black!60] (train.east -| train.east)+(1.0,3.8) node (a) {Online};
	
\end{tikzpicture}

related_work.tex
\section{Background}
\label{sec:related_work}

\citet{rice_algorithm_1976} was the first to formalize the idea of selecting among different algorithms on a per-instance basis. While he referred to the problem simply as \emph{algorithm selection}, we prefer the more precise term \emph{per-instance algorithm selection}, to avoid confusion with the (simpler) task of selecting one of several given algorithms to optimize performance on a given set or distribution of instances.

%HH: I've cleaned up the definition. Note: I don't like the use of loss function here and would prefer performance metric which, IMO, is more closely aligned
%    with the general use case for algorith configuration (which goes beyond ML). I propose to change this now or in the final version.
%\note{HH/KLB}{
	%\note{HH,KLB,FH}{HH: I've changed in quite a few places to the more precise ``per-instance algorithm selection''\\
	%KLB: I preferred what it was before---Rice called it the ``algorithm selection problem''. Or at least explain in the text why you object to Rice's (at this point, pretty established) terminology, given that we're attributing the problem definition to him.\\HH: I don't see this as a big problem, but have now added 
%a brief clarification.\\FH: Actually, I also don't see the need for the `per-instance' in the problem's name -- the original `algorithm selection' is indeed pretty established by now and I haven't seen anyone be confused by it. Have you? I found the `per-instance' distracting in reading (e.g., in this quote from the following section: ``algorithm selection scenarios, i.e., instances of the per-instance algorithm selection problem''). For me, ``per-instance configuration'' absolutely makes sense, but does it for selection? What do others think? (I guess this paper establishes nomenclature we should all try to use consistently in the future.)} 
\begin{define}[Per-instance algorithm selection problem]
	\label{def:algo_sel}
	Given
	\begin{itemize}
	  \item \changedTwo{a set $\mathcal{I}$ of problem instances drawn from a distribution $\mathcal{D}$},
	  \item a space of algorithms $\mathcal{A}$, and
	  \item a performance measure $m: \mathcal{I} \times \mathcal{A} \rightarrow \mathds{R}$,
	\end{itemize}
	the \emph{per-instance algorithm selection problem}	is to 
	find a mapping
        $s: \mathcal{I} \rightarrow \mathcal{A}$ that optimizes $\mathds{E}_{i \sim \mathcal{D}}m(i, s(i))$,
	i.e., the \changedTwo{expected performance measure for instances $i$ distributed according to $\mathcal{D}$, achieved by running the selected algorithm $s(i)$ for instance $i$}.
\end{define}

% \note{Reviewer II}{
% Definition 1 is a single, hard-to-read sentence.\\
% ML: I added an itemize to structure it a bit more.
% }

In practice, the mapping $s$ is often implemented by using 
so-called instance features, i.e., 
characterizations of the instances $i \in \mathcal{I}$.
These instance features are then mapped to an algorithm using 
machine learning techniques. However, the computation of instance features incurs additional costs,
which have to be considered in the performance measure $m$.

%{\bf HH: I've had to make modifications below, because our definition of AS does not include several of the variants discussed below.}

%\note{Reviewer}{Cite and discuss also \cite{SmithMiles201412}}

There are many ways of tackling per-instance algorithm selection and related
problems. Almost all contemporary approaches use machine learning 
to build predictors of the behaviour of given algorithms as a function of instance features. 
This general strategy may involve a single learned model or a complex combination of several, which, given a new problem instance 
to solve, is used to decide which algorithm or which combination of
algorithms to choose.

\subsection{What to select and when}

It is perhaps most natural to select a single
algorithm for solving a given problem instance. This \changedTwo{approach is, e.g., used in the
SATzilla~\cite{Satzilla03,xu_satzilla_2008},
\textsc{ArgoSmArT}~\cite{nikoli_instance-based_2009},
SALSA~\cite{demmel_self-adapting_2005} and
\textsc{Eureka}~\cite{cook_maximizing_1997} systems. Its main disadvantage}
is that there is no way of mitigating a
poor selection---\changedTwo{the system cannot recover if the algorithm it chose for a problem instance exhibits poor performance.}

Alternatively, we can seek a schedule that determines an ordering and time budget according to which we run all or a subset of the algorithms in the portfolio; usually, this schedule 
is chosen in a way that reflects the expected performance of the given
algorithms (see, e.g.,
\cite{pulina_self-adaptive_2009,cphydra,kadioglu_algorithm_2011,hoos_aspeed_2014,howe_exploiting_1999}).
Under some of these approaches, the computation of the schedule is
treated as an optimization problem that aims to maximize, e.g., the number of
problem instances solved within a timeout. For stochastic algorithms, the further
question of whether and when to restart an algorithm arises, opening the possibility of schedules that contain only a single algorithm, restarted several times (see, e.g., \cite{gomes_algorithm_2001,cicirello_max_2005,streeter_restart_2007,gagliolo_learning_restart_strategies_2007}).
%\note{FH}{Added \cite{gomes_algorithm_2001} and \cite{gagliolo_learning_restart_strategies_2007}; leaving the former out of the references would've been a bit of a fault pax.}
Instead of performing algorithm selection only once before starting to solve a problem, selection can
also be carried out repeatedly while the instance is being solved, taking into account information revealed during the algorithm run. 
%Most recent approaches only consider the case where the selection step is run once and its decisions are valid for the entire solving process. There are, however, efforts to 
Such methods monitor the execution of
the chosen algorithm(s) and take remedial action if performance deviates from
what is expected~\cite{gagliolo_adaptive_2004,MUSportfolio,LeiteBV12}, or perform selection 
repeatedly for subproblems of the given instance \cite{LAG1,LAG2,arbelaez_continuous_2010,samulowitz_learning_2007}.

\subsection{How to select}\label{sec:background:how}

The kinds of decisions the selection process is asked to produce drive the choice of machine learning models that perform the selection. If only a
single algorithm should be run, we can train a classification model that makes
exactly that prediction. This renders algorithm selection conceptually quite
simple---only a single machine learning model needs to be trained and run to
determine which algorithm to choose (see, e.g., \cite{guerri_learning_2004,gent_learning_2010,malitsky_non-model-based_2011}).

There are alternatives to using a classification model to select a single algorithm to be run on
a given instance, such as using regression models to predict the performance of each algorithm in the
portfolio. This regression approach
was adopted by several systems~\cite{Satzilla03,xu_satzilla_2008,roberts_learned_2007,silverthorn_latent_2010,Mersmann2013}.
Other approaches include 
the use of clustering techniques to partition problem instances in feature space and make decisions for each partition
separately~\cite{stergiou_heuristics_2009,kadioglu_isac_2010}, hierarchical models that make a series of
decisions~\cite{xu_hierarchical_2007,hurley_proteus_2014}, cost-sensitive support vector
machines~\cite{Bischl2012_2} and cost-sensitive decision forests~\cite{xu_hydra-mip_2011}.


\subsection{Selection enablers}

%\note{Reviewer}{an explanation of instance specific features and/or exploratory landscape analysis will improve the paper: 
%what information is extracted?  are parameters important?  
%what impact does information quality have on overall oracle performance (as listed in your experiments in section 6).}

In order to make their decisions, algorithm selection systems need information
about the problem instance to solve and the performance of the algorithms in the
given portfolio. The extraction of this information---the
features used by the machine learning techniques used for selection---incurs overhead not required
when only a single algorithm is used for all instances regardless of instance characteristics. 
It is therefore desirable
to extract information as cheaply as possible, thus ensuring that the performance benefits
of using algorithm selection are not outweighed by this overhead.

Some approaches use only past performance of the algorithms in the portfolio
as a basis for selecting the one(s) to be run on a given problem
instance~\cite{gagliolo_adaptive_2004,streeter_combining_2007,silverthorn_latent_2010}. This approach has the benefit that the required data can be collected with minimal overhead as algorithms are executed. 
It can work well if the performance of the algorithms is similar
on broad ranges of problem instances. However, when this assumption is not 
satisfied (as is often the case), more informative features are needed.

%\note{Reviewer}{".. number of clauses in SAT, number of constraints in constraint problems". This is a bit awkward, clauses are constraints, while at the same time CSP/CP problems offer richers features (e.g., complexity of a constraint)}

Turning to richer instance-specific features, commonly used features include the number of variables of a problem instance and
properties of the variable domains (e.g., the list of possible assignments in
constraint problems, the number of clauses in SAT, the number of goals in
planning). 
%HH: added
Deeper analysis can involve properties of graph representations derived from the
input instance (such as the constraint graph~\cite{leyton2003portfolio,gent_learning_2010}) or properties of encodings into
different problems (such as SAT features for SAT-encoded planning
problems~\cite{fawcett2014improved}).
%\note{BB}{In the section above we should clarify that this is really only an example of features for eg SAT.
  %It sounds a bit too general, and our application domain is richer with other types of features.}

In addition, features can be extracted from
short runs of one or more solvers on the given problem instance. 
Examples of such probing features include the number of search
nodes explored within a certain time, the fraction of partial solutions that are
disallowed by a certain constraint or clause, the average depth reached
before backtracking is required, or characteristics of local minima found quickly using local search.
Probing features are usually more expensive to
compute than the features that can be obtained from shallow analysis of the instance
specification, \fh{but they can also be more powerful and have thus been used by many authors (see, e.g., 
\citep{nudelman_understanding_2004,cphydra,pulina_multi-engine_2007,xu_satzilla_2008,hutter2014algorithm}).}
%
% FH: moved the following sentence from intro to here.
%\note{Reviewers}{Discuss continuous blackbox optimization more in detail.}
For continuous blackbox optimization, algorithm selection can be performed based on Exploratory Landscape Analysis
\cite{Mersmann2013,Bischl2012_2,Kerschke2014}. 
The approach defines a set of numerical features (of different complexities and computational costs) 
to describe the landscapes of such optimization problems. Examples range from simple features that 
describe the distribution of sampled objective values to more expensive probing features based on local search. 

\changed{Finally, in the \fh{area of meta-learning (learning about the performance of machine learning algorithms; for an overview, see, e.g, \citep{Brazdil_metalearning_2008}), these} features are known as \emph{meta-features}. They include statistical and information-theoretical measures (e.g., variable entropy), landmarkers (measurements of the performance of fast algorithms~\cite{Pfahringer:2000p553}), sampling landmarkers (similar to probing features) and model-based meta-features~\cite{Vanschoren2010}. These meta-features, and the past performance measurements of many machine learning algorithms, are available from the online machine learning platform OpenML~\cite{openml2013}. In contrast to ASlib, however, OpenML is not designed to allow cross-domain evaluation of algorithm selection techniques.
}

%\note{BB}{I added the last sentences to satisfy the reviewer}

\subsection{Algorithm Selection and Algorithm Configuration}

A problem closely related to algorithm selection is the algorithm configuration problem:
given a parameterized algorithm $A$, a set of problem instances $I$ and a performance measure $m$, find a parameter setting of $A$ that optimizes $m$ on $I$ \changedTwo{(see \cite{hutter_paramils_2009} for a formal definition)}.
While algorithm selection operates on finite (usually small) sets of algorithms, algorithm configuration operates on the combinatorial space of an algorithm's parameter settings. General algorithm configuration methods, such as ParamILS~\cite{hutter_paramils_2009}, GGA~\cite{ansotegui_gender-based_2009}, I/F-Race~\cite{BirEtAl10}, and SMAC~\cite{HutHooLey11-SMAC}, 
%can handle the many discrete parameters that occur in practical algorithms.
have yielded substantial performance improvements (sometimes orders of magnitude
speedups) of state-of-the-art algorithms for several benchmarks, including
SAT-based formal verification~\cite{HutBabHooHu07}, mixed integer
programming~\cite{HutHooLey10-mipconfig}, AI planning~\cite{roberts_what_2008,Vallati13-SOCS}, the combined selection and hyperparameter optimization of machine learning algorithms~\cite{ThoHutHooLey13-AutoWEKA}, \changedTwo{and joint architecture and hyperparameter search in deep learning~\cite{domhan-ijcai15a}}.
%
Algorithm configuration and selection are complementary since configuration can
identify algorithms with peak performance for homogeneous benchmarks and
selection can then choose from among these specialized algorithms. Consequently, several possibilities exist for combining algorithm configuration and selection~\cite{hutter_performance_2006,xu_hydra_2010,kadioglu_isac_2010,xu_hydra-mip_2011,malitsky_evolving_2013,sabharwal_boosting_2013,ISAC++,feurer-aaai15a}.
The algorithm configuration counterpart of ASlib is AClib~\cite{HutEtAl14:AClib} (\url{http://aclib.net}).
\changed{In contrast to ASlib, it is infeasible in AClib to store performance data for all possible parameter configurations, which often number more than $10^{50}$.
Therefore, an experiment on AClib includes new (expensive) runs of the target algorithms with different configurations
and hence, experiments on AClib are a lot more costly than experiments on ASlib, where no new algorithm runs are necessary.\footnote{\fh{In algorithm configuration, this need for expensive runs indeed causes a problem for research. One way of mitigating it is offered by fast-to-evaluate surrogate algorithm configuration benchmarks~\cite{Eggensperger2015}.}}} \changedTwo{Furthermore, in contrast to AClib, ASlib does not include the actual instances and binaries of the algorithms.
Therefore, ASlib does not provide a way to generate new performance data, as is required in AClib as a consequence of the need to assess the performance of new target algorithm configurations arising within the configuration process. 
However, ASlib and AClib can be combined by generating actual performance data based on the resources in AClib and then creating an ASlib scenario which selects between different solver configurations on a per-instance basis.
%in a way that the actual performance data can be generated based on the resources in AClib and then converted to the ASlib format to analyze the performance of algorithm selectors.  
}

% \note{Reviewer I}{What comes to mind are the following settings that do not seem to be
% covered: approaches that combine hyper-parameter optimization and selection
% such as "Initializing Bayesian Hyper-parameter Optimization via Meta-Learning",
% Feurer et. al. and approaches that pass on instance knowledge between solver
% runs such as "Boosting Sequential Solver Portfolios: Knowledge Sharing and
% Accuracy Prediction", Malitsky et. al. The issue with the latter is that one
% actually needs to execute the solvers which is something one would not want to
% do. Not entirely sure how to combine ACLib and ASLib (maybe one needs also a
% meta-platform...), but I think it would be worthwhile to mention some settings
% that the platform do not cover.\\
% ML: literature added and some words about the limitations added.
% }


%\note{Reviewer}{What is the difference between ASlib and AClib?}

%\medskip

A full coverage of the wide literature on algorithm selection is beyond the scope of this article, but we refer the interested reader to recent survey articles on the topic~\cite{smith-miles_cross-disciplinary_2009,kotthoff_algorithm_2014,Serban:2013,VallatiCK15}.

%\subsection{Applications}
%
%\todo{LK: Not sure why this is called ``Applications'' -- it seems to introduce
%a number of specific approaches.}
%\todo{YM: The point is that this field of study is not only focused on picking the best
%solver, and that there are also other applications of the techniques we are studying.}
%
%Regardless of the specifics of the implementation, the benefits and power of 
%algorithm selection is immediately visible when
%looking at the winners of international competitions in
%SAT~\cite{xu2012evaluating}, CSP~\cite{cphydra}, AI
%planning~\cite{helmert_fast_2011}, etc. In any field where problems can be
%solved by multiple approaches, this methodology can help make the decision of
%the best approach for the given instance. For an overview of all the currently
%explored applications we refer the reader to \cite{kotthoff_algorithm_2014}. Yet
%even when only a single solver exists, there are still usually a number of
%parameters that determine its behavior. In such a scenario, multiple versions of
%the solver can be automatically configured using tools like
%SMAC~\cite{HutHooLey11-SMAC} or GGA~\cite{ansotegui_gender-based_2009} to create
%a portfolio~\cite{xu_hydra-mip_2011,malitsky_instance-specific_2012,ISAC++}.
%
%Furthermore, techniques that have been developed for algorithm selection are not
%restricted to the competition scenarios. Approaches like DASH, analyze the
%structure of the nodes of search tree, identifying the best branching heuristic
%to employ at that time~\cite{DASH}. For a given instance, systems like Proteus
%identify whether it is worthwhile transforming the instance to an alternate
%problem representation prior to selecting the most appropriate
%solver~\cite{hurley_proteus_2014}. In the realm of finding all Minimum
%Unsatisfiable Sets (MUS) in a CSP, algorithm selection was used to show that
%improved performance can be achieved by switching the applied solver every few
%minutes based on the MUSs that have already been found~\cite{MUSportfolio}. A
%related application is the creation of solver schedules which not only choose
%the best solver to apply on an instance, but can also intelligently hedge there
%bets by selecting a sequence of solvers and the amount of time that each should
%be run for~\cite{hoos_aspeed_2014,p3S}. And current research is continuing to
%take algorithm selection to new applications.
% 

results_selection-algorithms.tex
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Fri Dec 18 13:31:14 2015
Scenario & PAR10 full set & Number & PAR10 reduced set \\ 
  \midrule
SAT11-HAND & 16943.49 & 15 $\to$ 8 & 15919.09 \\ 
  SAT11-INDU & 13246.70 & 18 $\to$ 4 & 12127.05 \\ 
  SAT11-RAND & 10253.09 & 9 $\to$ 4 & 10180.39 \\ 
  SAT12-ALL & 971.45 & 31 $\to$ 11 & 979.17 \\ 
  SAT12-HAND & 4303.81 & 31 $\to$ 10 & 4187.13 \\ 
  SAT12-INDU & 2763.37 & 31 $\to$ 7 & 2775.61 \\ 
  SAT12-RAND & 3241.42 & 31 $\to$ 2 & 3153.48 \\ 
  SAT15-INDU & 3845.52 & 28 $\to$ 6 & 3604.57 \\ 
  QBF-2011 & 9232.49 & 5 $\to$ 4 & 9198.01 \\ 
  QBF-2014 & 2090.02 & 14 $\to$ 8 & 2040.33 \\ 
  MAXSAT12-PMS & 3370.22 & 6 $\to$ 3 & 3299.15 \\ 
  MAXSAT15-PMS-INDU & 1752.57 & 29 $\to$ 7 & 1479.31 \\ 
  CSP-2010 & 6541.20 & 2 $\to$ 2 & 6516.57 \\ 
  CSP-MZN-2013 & 4204.58 & 11 $\to$ 9 & 4168.35 \\ 
  PROTEUS-2014 & 5905.71 & 22 $\to$ 8 & 5725.50 \\ 
  ASP-POTASSCO & 525.55 & 11 $\to$ 5 & 509.61 \\ 
  PREMAR-ASTAR-2015 & 5154.40 & 4 $\to$ 3 & 4954.45 \\ 
   \bottomrule

results_selection-features.tex
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Fri Dec 18 13:31:14 2015
Scenario & full set & default set & Number & reduced set \\ 
  \midrule
SAT11-HAND & 17249.59 & 16943.49 & 113 $\to$ 6 & 15743.04 \\ 
  SAT11-INDU & 13111.61 & 13246.70 & 112 $\to$ 4 & 10951.00 \\ 
  SAT11-RAND & 10496.39 & 10253.09 & 112 $\to$ 3 & 9854.11 \\ 
  SAT12-ALL & 994.25 & 971.45 & 113 $\to$ 9 & 815.37 \\ 
  SAT12-HAND & 4298.00 & 4303.81 & 113 $\to$ 6 & 4092.58 \\ 
  SAT12-INDU & 2881.97 & 2763.37 & 113 $\to$ 6 & 2506.25 \\ 
  SAT12-RAND & 3196.28 & 3241.42 & 113 $\to$ 3 & 3088.72 \\ 
  SAT15-INDU & 3970.56 & 3845.52 & 51 $\to$ 3 & 3620.06 \\ 
  QBF-2011 & 9229.99 & 9232.49 & 46 $\to$ 5 & 8995.62 \\ 
  QBF-2014 & 2102.79 & 2090.02 & 46 $\to$ 4 & 2032.50 \\ 
  MAXSAT12-PMS & 3321.22 & 3370.22 & 30 $\to$ 4 & 3296.52 \\ 
  MAXSAT15-PMS-INDU & 1696.69 & 1752.57 & 29 $\to$ 5 & 1520.77 \\ 
  CSP-2010 & 6514.37 & 6541.20 & 69 $\to$ 3 & 6415.23 \\ 
  CSP-MZN-2013 & 4192.82 & 4204.58 & 115 $\to$ 5 & 4119.36 \\ 
  PROTEUS-2014 & 6120.13 & 5905.71 & 193 $\to$ 6 & 5700.05 \\ 
  ASP-POTASSCO & 516.47 & 525.55 & 134 $\to$ 4 & 472.84 \\ 
  PREMAR-ASTAR-2015 & 5289.96 & 5154.40 & 22 $\to$ 3 & 4619.49 \\ 
   \bottomrule

results_supplementary.tex
\documentclass[]{elsarticle}
\usepackage{caption}
\usepackage{booktabs}
\begin{document}% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:36 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.86 & 880.55 & 58.88 \\ 
  baseline/singleBest & 0.86 & 880.55 & 58.88 \\ 
  baseline/singleBestByPar & 0.86 & 880.55 & 58.88 \\ 
  baseline/vbs & 0.94 & 400.18 & 0.00 \\ 
  classif/ksvm & 0.89 & 693.12 & 33.14 \\ 
  classif/randomForest & 0.89 & 674.88 & 27.46 \\ 
  classif/rpart & 0.89 & 654.97 & 28.43 \\ 
  cluster/XMeans & 0.89 & 688.78 & 37.14 \\ 
  regr/lm & 0.90 & 615.05 & 26.04 \\ 
  regr/randomForest & 0.92 & 495.11 & 14.56 \\ 
  regr/rpart & 0.90 & 625.06 & 31.85 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario ASP-POTASSCO. The following feature steps were used: Static (38 features).
The cost for using the feature steps (adapted for presolving) is 1404.25, on average 1.0852 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:37 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.86 & 7201.56 & 79.14 \\ 
  baseline/singleBest & 0.86 & 7201.56 & 79.14 \\ 
  baseline/singleBestByPar & 0.86 & 7201.56 & 79.14 \\ 
  baseline/vbs & 0.88 & 6344.25 & 0.00 \\ 
  classif/ksvm & 0.87 & 6729.74 & 29.76 \\ 
  classif/randomForest & 0.87 & 6629.11 & 18.06 \\ 
  classif/rpart & 0.87 & 6488.16 & 10.51 \\ 
  cluster/XMeans & 0.86 & 7005.38 & 60.83 \\ 
  regr/lm & 0.87 & 6633.48 & 22.43 \\ 
  regr/randomForest & 0.87 & 6542.55 & 20.43 \\ 
  regr/rpart & 0.86 & 6952.25 & 52.17 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario CSP-2010. The following feature steps were used: all-feats (86 features).
} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:40 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.52 & 8789.17 & 492.88 \\ 
  baseline/singleBest & 0.52 & 8789.17 & 492.88 \\ 
  baseline/singleBestByPar & 0.52 & 8789.17 & 492.88 \\ 
  baseline/vbs & 0.80 & 3752.47 & 0.00 \\ 
  classif/ksvm & 0.75 & 4669.17 & 87.47 \\ 
  classif/randomForest & 0.74 & 4831.28 & 103.72 \\ 
  classif/rpart & 0.71 & 5355.83 & 143.84 \\ 
  cluster/XMeans & 0.56 & 7976.18 & 405.86 \\ 
  regr/lm & 0.75 & 4592.36 & 80.58 \\ 
  regr/randomForest & 0.77 & 4205.36 & 47.04 \\ 
  regr/rpart & 0.74 & 4908.38 & 115.47 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario CSP-MZN-2013. The following feature steps were used: static (144 features).
The cost for using the feature steps (adapted for presolving) is 340947, on average 73.4483 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:41 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.77 & 4893.14 & 190.90 \\ 
  baseline/singleBest & 0.77 & 4893.14 & 190.90 \\ 
  baseline/singleBestByPar & 0.77 & 4893.14 & 190.90 \\ 
  baseline/vbs & 0.85 & 3127.24 & 0.00 \\ 
  classif/ksvm & 0.82 & 3789.76 & 58.31 \\ 
  classif/randomForest & 0.82 & 3745.49 & 57.19 \\ 
  classif/rpart & 0.82 & 3795.52 & 64.07 \\ 
  cluster/XMeans & 0.82 & 3833.90 & 80.87 \\ 
  regr/lm & 0.82 & 3777.87 & 68.00 \\ 
  regr/randomForest & 0.84 & 3297.74 & 19.37 \\ 
  regr/rpart & 0.82 & 3789.72 & 79.84 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario MAXSAT12-PMS. The following feature steps were used: group-basics (37 features).
The cost for using the feature steps (adapted for presolving) is 132.54, on average 0.151301 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:41 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.88 & 2231.05 & 118.36 \\ 
  baseline/singleBest & 0.88 & 2231.05 & 118.36 \\ 
  baseline/singleBestByPar & 0.88 & 2231.05 & 118.36 \\ 
  baseline/vbs & 0.93 & 1357.95 & 0.00 \\ 
  classif/ksvm & 0.89 & 1957.04 & 59.99 \\ 
  classif/randomForest & 0.90 & 1905.59 & 62.45 \\ 
  classif/rpart & 0.89 & 1992.38 & 68.38 \\ 
  cluster/XMeans & 0.90 & 1839.30 & 77.02 \\ 
  regr/lm & 0.90 & 1830.39 & 68.12 \\ 
  regr/randomForest & 0.91 & 1669.72 & 42.23 \\ 
  regr/rpart & 0.89 & 2063.11 & 85.20 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario MAXSAT15-PMS-INDU. The following feature steps were used: ALL (37 features).
} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:41 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.81 & 7002.91 & 688.77 \\ 
  baseline/singleBest & 0.81 & 7002.91 & 688.77 \\ 
  baseline/singleBestByPar & 0.81 & 7002.91 & 688.77 \\ 
  baseline/vbs & 1.00 & 227.60 & 0.00 \\ 
  classif/ksvm & 0.86 & 5423.07 & 522.98 \\ 
  classif/randomForest & 0.84 & 5908.73 & 578.28 \\ 
  classif/rpart & 0.86 & 5437.73 & 537.64 \\ 
  cluster/XMeans & 0.85 & 5534.53 & 511.48 \\ 
  regr/lm & 0.84 & 5887.33 & 556.88 \\ 
  regr/randomForest & 0.87 & 5028.53 & 497.32 \\ 
  regr/rpart & 0.84 & 5916.12 & 585.67 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario PREMARSHALLING-ASTAR-2015. The following feature steps were used: orig (16 features).
} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:45 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.63 & 13443.35 & 957.46 \\ 
  baseline/singleBest & 0.63 & 13443.35 & 957.46 \\ 
  baseline/singleBestByPar & 0.63 & 13443.35 & 957.46 \\ 
  baseline/vbs & 0.89 & 4105.89 & 0.00 \\ 
  classif/ksvm & 0.82 & 6623.25 & 273.52 \\ 
  classif/randomForest & 0.82 & 6616.83 & 275.17 \\ 
  classif/rpart & 0.75 & 9019.04 & 509.89 \\ 
  cluster/XMeans & 0.77 & 8425.42 & 472.67 \\ 
  regr/lm & 0.83 & 6246.41 & 235.03 \\ 
  regr/randomForest & 0.84 & 5867.26 & 202.24 \\ 
  regr/rpart & 0.81 & 6848.25 & 296.95 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario PROTEUS-2014. The following feature steps were used: csp (36 features).
The cost for using the feature steps (adapted for presolving) is 26059.7, on average 6.4809 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:45 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.58 & 15330.17 & 716.76 \\ 
  baseline/singleBest & 0.58 & 15330.17 & 716.76 \\ 
  baseline/singleBestByPar & 0.58 & 15330.17 & 716.76 \\ 
  baseline/vbs & 0.77 & 8337.10 & 0.00 \\ 
  classif/ksvm & 0.71 & 10665.59 & 244.28 \\ 
  classif/randomForest & 0.73 & 9828.03 & 164.61 \\ 
  classif/rpart & 0.72 & 10293.05 & 227.01 \\ 
  cluster/XMeans & 0.66 & 12343.24 & 429.82 \\ 
  regr/lm & 0.71 & 10388.71 & 227.93 \\ 
  regr/randomForest & 0.75 & 9177.62 & 106.31 \\ 
  regr/rpart & 0.71 & 10405.54 & 244.75 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario QBF-2011. The following feature steps were used: all-feats (46 features).
} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:46 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.63 & 3401.81 & 170.95 \\ 
  baseline/singleBest & 0.63 & 3401.81 & 170.95 \\ 
  baseline/singleBestByPar & 0.63 & 3401.81 & 170.95 \\ 
  baseline/vbs & 0.81 & 1758.14 & 0.00 \\ 
  classif/ksvm & 0.71 & 2653.52 & 87.96 \\ 
  classif/randomForest & 0.72 & 2529.65 & 73.91 \\ 
  classif/rpart & 0.69 & 2783.88 & 102.06 \\ 
  cluster/XMeans & 0.64 & 3290.49 & 156.51 \\ 
  regr/lm & 0.69 & 2824.38 & 110.26 \\ 
  regr/randomForest & 0.77 & 2104.42 & 36.24 \\ 
  regr/rpart & 0.72 & 2548.49 & 86.28 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario QBF-2014. The following feature steps were used: default (46 features).
} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:46 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.50 & 25589.27 & 1434.69 \\ 
  baseline/singleBest & 0.50 & 25649.09 & 1342.48 \\ 
  baseline/singleBestByPar & 0.50 & 25589.27 & 1434.69 \\ 
  baseline/vbs & 0.74 & 13360.66 & 0.00 \\ 
  classif/ksvm & 0.67 & 16795.52 & 368.56 \\ 
  classif/randomForest & 0.65 & 17960.50 & 470.43 \\ 
  classif/rpart & 0.64 & 18329.71 & 535.66 \\ 
  cluster/XMeans & 0.52 & 24067.25 & 958.47 \\ 
  regr/lm & 0.61 & 19715.73 & 554.42 \\ 
  regr/randomForest & 0.66 & 17595.72 & 408.96 \\ 
  regr/rpart & 0.61 & 19835.52 & 674.20 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario SAT11-HAND. The following feature steps were used: Pre (6 features).
The cost for using the feature steps (adapted for presolving) is 290.88, on average 0.982703 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:47 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.72 & 14605.90 & 718.39 \\ 
  baseline/singleBest & 0.72 & 14605.90 & 718.39 \\ 
  baseline/singleBestByPar & 0.72 & 14605.90 & 718.39 \\ 
  baseline/vbs & 0.84 & 8187.52 & 0.00 \\ 
  classif/ksvm & 0.73 & 13963.88 & 577.90 \\ 
  classif/randomForest & 0.74 & 13661.59 & 568.83 \\ 
  classif/rpart & 0.73 & 13915.25 & 525.25 \\ 
  cluster/XMeans & 0.72 & 14537.56 & 692.64 \\ 
  regr/lm & 0.73 & 13756.72 & 526.94 \\ 
  regr/randomForest & 0.72 & 14545.27 & 559.57 \\ 
  regr/rpart & 0.68 & 16443.08 & 817.67 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario SAT11-INDU. The following feature steps were used: Pre (6 features).
The cost for using the feature steps (adapted for presolving) is 3332, on average 11.1067 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:47 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.60 & 19916.36 & 979.92 \\ 
  baseline/singleBest & 0.60 & 19916.36 & 979.92 \\ 
  baseline/singleBestByPar & 0.60 & 19916.36 & 979.92 \\ 
  baseline/vbs & 0.82 & 9186.44 & 0.00 \\ 
  classif/ksvm & 0.77 & 11806.90 & 205.86 \\ 
  classif/randomForest & 0.79 & 10831.69 & 130.67 \\ 
  classif/rpart & 0.79 & 10644.77 & 93.24 \\ 
  cluster/XMeans & 0.72 & 14071.00 & 447.01 \\ 
  regr/lm & 0.78 & 10979.25 & 127.61 \\ 
  regr/randomForest & 0.80 & 10260.29 & 83.75 \\ 
  regr/rpart & 0.78 & 11103.93 & 177.72 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario SAT11-RAND. The following feature steps were used: Pre (6 features).
The cost for using the feature steps (adapted for presolving) is 47.11, on average 0.0785167 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:49 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.75 & 3079.89 & 302.51 \\ 
  baseline/singleBest & 0.75 & 3079.89 & 302.51 \\ 
  baseline/singleBestByPar & 0.75 & 3079.89 & 302.51 \\ 
  baseline/vbs & 0.99 & 241.32 & 0.00 \\ 
  classif/ksvm & 0.90 & 1286.27 & 97.15 \\ 
  classif/randomForest & 0.92 & 1083.40 & 80.91 \\ 
  classif/rpart & 0.77 & 2897.47 & 241.13 \\ 
  cluster/XMeans & 0.75 & 3115.03 & 296.30 \\ 
  regr/lm & 0.86 & 1851.19 & 154.37 \\ 
  regr/randomForest & 0.92 & 1044.39 & 74.47 \\ 
  regr/rpart & 0.83 & 2160.57 & 191.10 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario SAT12-ALL. The following feature steps were used: Pre (6 features).
The cost for using the feature steps (adapted for presolving) is 9021.19, on average 5.58934 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:49 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.48 & 6338.90 & 254.76 \\ 
  baseline/singleBest & 0.48 & 6338.90 & 254.76 \\ 
  baseline/singleBestByPar & 0.48 & 6338.90 & 254.76 \\ 
  baseline/vbs & 0.70 & 3662.24 & 0.00 \\ 
  classif/ksvm & 0.61 & 4738.13 & 95.27 \\ 
  classif/randomForest & 0.63 & 4462.59 & 72.81 \\ 
  classif/rpart & 0.56 & 5349.11 & 143.27 \\ 
  cluster/XMeans & 0.50 & 6118.22 & 223.52 \\ 
  regr/lm & 0.59 & 4961.73 & 107.51 \\ 
  regr/randomForest & 0.65 & 4320.67 & 57.29 \\ 
  regr/rpart & 0.58 & 5140.14 & 131.32 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario SAT12-HAND. The following feature steps were used: Pre (6 features).
The cost for using the feature steps (adapted for presolving) is 801.95, on average 1.04557 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:50 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.74 & 3266.05 & 128.35 \\ 
  baseline/singleBest & 0.74 & 3266.05 & 128.35 \\ 
  baseline/singleBestByPar & 0.74 & 3266.05 & 128.35 \\ 
  baseline/vbs & 0.82 & 2221.50 & 0.00 \\ 
  classif/ksvm & 0.77 & 2875.11 & 55.50 \\ 
  classif/randomForest & 0.78 & 2806.37 & 41.43 \\ 
  classif/rpart & 0.72 & 3449.57 & 121.24 \\ 
  cluster/XMeans & 0.72 & 3433.01 & 118.63 \\ 
  regr/lm & 0.76 & 3015.79 & 75.81 \\ 
  regr/randomForest & 0.78 & 2770.11 & 42.38 \\ 
  regr/rpart & 0.74 & 3272.58 & 105.62 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario SAT12-INDU. The following feature steps were used: Pre (6 features).
The cost for using the feature steps (adapted for presolving) is 22200.2, on average 19.0234 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:52 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.73 & 3271.14 & 49.40 \\ 
  baseline/singleBest & 0.73 & 3271.14 & 49.40 \\ 
  baseline/singleBestByPar & 0.73 & 3271.14 & 49.40 \\ 
  baseline/vbs & 0.76 & 2872.84 & 0.00 \\ 
  classif/ksvm & 0.73 & 3272.16 & 44.50 \\ 
  classif/randomForest & 0.73 & 3231.00 & 42.96 \\ 
  classif/rpart & 0.74 & 3133.81 & 32.82 \\ 
  cluster/XMeans & 0.73 & 3231.30 & 43.25 \\ 
  regr/lm & 0.74 & 3161.09 & 36.31 \\ 
  regr/randomForest & 0.74 & 3169.67 & 37.00 \\ 
  regr/rpart & 0.72 & 3471.65 & 69.66 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario SAT12-RAND. The following feature steps were used: Pre (6 features).
The cost for using the feature steps (adapted for presolving) is 80.14, on average 0.0588399 per instance.} 
\end{table}
% latex table generated in R 3.2.3 by xtable 1.8-0 package
% Tue Dec 15 16:56:52 2015
\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
  \toprule
Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline/singleBestBySuccesses & 0.87 & 5189.36 & 525.79 \\ 
  baseline/singleBest & 0.87 & 5189.36 & 525.79 \\ 
  baseline/singleBestByPar & 0.87 & 5189.36 & 525.79 \\ 
  baseline/vbs & 0.94 & 2287.57 & 0.00 \\ 
  classif/ksvm & 0.89 & 4458.57 & 335.00 \\ 
  classif/randomForest & 0.88 & 4562.34 & 330.77 \\ 
  classif/rpart & 0.88 & 4587.28 & 355.71 \\ 
  cluster/XMeans & 0.86 & 5483.98 & 496.41 \\ 
  regr/lm & 0.87 & 4930.56 & 374.99 \\ 
  regr/randomForest & 0.91 & 3763.44 & 287.87 \\ 
  regr/rpart & 0.87 & 5119.23 & 455.66 \\ 
   \bottomrule
\end{tabular}
\caption{Full experimental results for scenario SAT15-INDU. The following feature steps were used: ALL (54 features).
} 
\end{table}
\end{document}
results.tex
\begin{table}[ht]
\subfloat[caption][ASP-POTASSCO]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{llrrr}
  \toprule
Type & Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline & singleBestByPar & 0.86 & 880.5 & 58.9 \\ 
  baseline & singleBestByPar & 0.86 & 880.5 & 58.9 \\ 
  baseline & vbs & 0.94 & 400.2 & 0.0 \\ 
  baseline & vbs & 0.94 & 400.2 & 0.0 \\ 
  classif & ksvm & 0.89 & 692.0 & 32.0 \\ 
  classif & randomForest & 0.88 & 712.2 & 31.4 \\ 
  classif & rpart & 0.89 & 655.0 & 28.4 \\ 
  cluster & XMeans & 0.89 & 701.8 & 37.7 \\ 
  regr & lm & 0.90 & 615.1 & 26.1 \\ 
  regr & randomForest & 0.92 & 509.1 & 16.0 \\ 
  regr & rpart & 0.90 & 624.9 & 31.7 \\ 
   \bottomrule
\end{tabular}
}
}
\subfloat[caption][CSP-2010]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{rrr}
  \toprule
Solved & PAR10 & MCP \\ 
  \midrule
0.86 & 7201.6 & 79.1 \\ 
  0.86 & 7201.6 & 79.1 \\ 
  0.88 & 6344.3 & 0.0 \\ 
  0.88 & 6344.3 & 0.0 \\ 
  0.87 & 6558.8 & 14.4 \\ 
  0.87 & 6677.6 & 22.1 \\ 
  0.87 & 6488.2 & 10.5 \\ 
  0.86 & 7005.4 & 60.8 \\ 
  0.87 & 6633.5 & 22.4 \\ 
  0.87 & 6466.7 & 11.3 \\ 
  0.86 & 6928.0 & 50.1 \\ 
   \bottomrule
\end{tabular}
}
}
\end{table}
\begin{table}[ht]
\subfloat[caption][MAXSAT12-PMS]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{llrrr}
  \toprule
Type & Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline & singleBestByPar & 0.77 & 4893.1 & 190.9 \\ 
  baseline & singleBestByPar & 0.77 & 4893.1 & 190.9 \\ 
  baseline & vbs & 0.85 & 3127.2 & 0.0 \\ 
  baseline & vbs & 0.85 & 3127.2 & 0.0 \\ 
  classif & ksvm & 0.83 & 3553.7 & 38.0 \\ 
  classif & randomForest & 0.82 & 3744.8 & 56.5 \\ 
  classif & rpart & 0.82 & 3795.5 & 64.1 \\ 
  cluster & XMeans & 0.82 & 3833.9 & 80.9 \\ 
  regr & lm & 0.82 & 3777.9 & 68.0 \\ 
  regr & randomForest & 0.84 & 3367.6 & 24.5 \\ 
  regr & rpart & 0.82 & 3789.7 & 79.8 \\ 
   \bottomrule
\end{tabular}
}
}
\subfloat[caption][PREMARSHALLING-ASTAR-2013]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{rrr}
  \toprule
Solved & PAR10 & MCP \\ 
  \midrule
0.81 & 7002.9 & 688.8 \\ 
  0.81 & 7002.9 & 688.8 \\ 
  1.00 & 227.6 & 0.0 \\ 
  1.00 & 227.6 & 0.0 \\ 
  0.86 & 5172.5 & 518.3 \\ 
  0.85 & 5688.9 & 542.9 \\ 
  0.88 & 4634.2 & 471.9 \\ 
  0.85 & 5756.4 & 548.9 \\ 
  0.84 & 5885.1 & 554.6 \\ 
  0.85 & 5546.5 & 523.5 \\ 
  0.85 & 5645.9 & 561.4 \\ 
   \bottomrule
\end{tabular}
}
}
\end{table}
\begin{table}[ht]
\subfloat[caption][PROTEUS-2014]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{llrrr}
  \toprule
Type & Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline & singleBestByPar & 0.63 & 13443.3 & 957.5 \\ 
  baseline & singleBestByPar & 0.63 & 13443.3 & 957.5 \\ 
  baseline & vbs & 0.89 & 4105.9 & 0.0 \\ 
  baseline & vbs & 0.89 & 4105.9 & 0.0 \\ 
  classif & ksvm & 0.81 & 6863.7 & 288.5 \\ 
  classif & randomForest & 0.81 & 6873.8 & 288.6 \\ 
  classif & rpart & 0.77 & 8527.0 & 451.9 \\ 
  cluster & XMeans & 0.78 & 8171.2 & 436.3 \\ 
  regr & lm & 0.78 & 7975.6 & 403.6 \\ 
  regr & randomForest & 0.84 & 6066.6 & 211.9 \\ 
  regr & rpart & 0.81 & 6921.3 & 302.2 \\ 
   \bottomrule
\end{tabular}
}
}
\subfloat[caption][QBF-2011]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{rrr}
  \toprule
Solved & PAR10 & MCP \\ 
  \midrule
0.58 & 15330.2 & 716.8 \\ 
  0.58 & 15330.2 & 716.8 \\ 
  0.77 & 8337.1 & 0.0 \\ 
  0.77 & 8337.1 & 0.0 \\ 
  0.71 & 10428.7 & 220.5 \\ 
  0.73 & 9776.1 & 160.1 \\ 
  0.72 & 10316.9 & 227.2 \\ 
  0.67 & 12089.1 & 412.5 \\ 
  0.71 & 10388.7 & 227.9 \\ 
  0.75 & 9075.0 & 98.4 \\ 
  0.71 & 10405.5 & 244.7 \\ 
   \bottomrule
\end{tabular}
}
}
\end{table}
\begin{table}[ht]
\subfloat[caption][SAT11-HAND]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{llrrr}
  \toprule
Type & Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline & singleBestByPar & 0.50 & 25589.3 & 1434.7 \\ 
  baseline & singleBestByPar & 0.50 & 25589.3 & 1434.7 \\ 
  baseline & vbs & 0.74 & 13360.7 & 0.0 \\ 
  baseline & vbs & 0.74 & 13360.7 & 0.0 \\ 
  classif & ksvm & 0.66 & 17446.9 & 406.1 \\ 
  classif & randomForest & 0.66 & 17655.5 & 462.7 \\ 
  classif & rpart & 0.62 & 19126.8 & 567.1 \\ 
  cluster & XMeans & 0.50 & 25272.9 & 1094.8 \\ 
  regr & lm & 0.60 & 20251.4 & 629.7 \\ 
  regr & randomForest & 0.67 & 16906.5 & 321.9 \\ 
  regr & rpart & 0.61 & 19929.2 & 611.0 \\ 
   \bottomrule
\end{tabular}
}
}
\subfloat[caption][SAT11-INDU]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{rrr}
  \toprule
Solved & PAR10 & MCP \\ 
  \midrule
0.72 & 14605.9 & 718.4 \\ 
  0.72 & 14605.9 & 718.4 \\ 
  0.84 & 8187.5 & 0.0 \\ 
  0.84 & 8187.5 & 0.0 \\ 
  0.73 & 14004.7 & 602.9 \\ 
  0.74 & 13655.1 & 541.5 \\ 
  0.72 & 14336.4 & 622.8 \\ 
  0.68 & 16299.2 & 810.8 \\ 
  0.72 & 14460.5 & 611.9 \\ 
  0.75 & 12996.6 & 494.3 \\ 
  0.69 & 15884.7 & 684.4 \\ 
   \bottomrule
\end{tabular}
}
}
\end{table}
\begin{table}[ht]
\subfloat[caption][SAT11-RAND]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{llrrr}
  \toprule
Type & Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline & singleBestByPar & 0.60 & 19916.4 & 979.9 \\ 
  baseline & singleBestByPar & 0.60 & 19916.4 & 979.9 \\ 
  baseline & vbs & 0.82 & 9186.4 & 0.0 \\ 
  baseline & vbs & 0.82 & 9186.4 & 0.0 \\ 
  classif & ksvm & 0.78 & 11164.6 & 156.0 \\ 
  classif & randomForest & 0.79 & 10838.3 & 129.8 \\ 
  classif & rpart & 0.79 & 10602.7 & 118.6 \\ 
  cluster & XMeans & 0.78 & 11337.7 & 178.9 \\ 
  regr & lm & 0.78 & 11275.4 & 191.5 \\ 
  regr & randomForest & 0.80 & 10341.2 & 82.2 \\ 
  regr & rpart & 0.76 & 12017.1 & 259.1 \\ 
   \bottomrule
\end{tabular}
}
}
\subfloat[caption][SAT12-ALL]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{rrr}
  \toprule
Solved & PAR10 & MCP \\ 
  \midrule
0.75 & 3079.9 & 302.5 \\ 
  0.75 & 3079.9 & 302.5 \\ 
  0.99 & 241.3 & 0.0 \\ 
  0.99 & 241.3 & 0.0 \\ 
  0.91 & 1195.7 & 95.4 \\ 
  0.93 & 1017.2 & 70.4 \\ 
  0.79 & 2643.6 & 219.3 \\ 
  0.74 & 3188.9 & 294.2 \\ 
  0.88 & 1517.5 & 128.8 \\ 
  0.93 & 980.3 & 65.6 \\ 
  0.83 & 2109.3 & 176.9 \\ 
   \bottomrule
\end{tabular}
}
}
\end{table}
\begin{table}[ht]
\subfloat[caption][SAT12-HAND]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{llrrr}
  \toprule
Type & Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline & singleBestByPar & 0.48 & 6338.9 & 254.8 \\ 
  baseline & singleBestByPar & 0.48 & 6338.9 & 254.8 \\ 
  baseline & vbs & 0.70 & 3662.2 & 0.0 \\ 
  baseline & vbs & 0.70 & 3662.2 & 0.0 \\ 
  classif & ksvm & 0.63 & 4552.5 & 85.9 \\ 
  classif & randomForest & 0.63 & 4511.4 & 72.9 \\ 
  classif & rpart & 0.56 & 5408.5 & 154.8 \\ 
  cluster & XMeans & 0.49 & 6149.3 & 221.0 \\ 
  regr & lm & 0.62 & 4626.4 & 89.3 \\ 
  regr & randomForest & 0.65 & 4252.9 & 53.1 \\ 
  regr & rpart & 0.56 & 5360.0 & 148.4 \\ 
   \bottomrule
\end{tabular}
}
}
\subfloat[caption][SAT12-INDU]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{rrr}
  \toprule
Solved & PAR10 & MCP \\ 
  \midrule
0.73 & 3332.1 & 148.2 \\ 
  0.73 & 3332.1 & 148.2 \\ 
  0.82 & 2221.5 & 0.0 \\ 
  0.82 & 2221.5 & 0.0 \\ 
  0.76 & 3066.0 & 71.4 \\ 
  0.77 & 2853.6 & 44.3 \\ 
  0.70 & 3677.7 & 127.9 \\ 
  0.71 & 3607.1 & 128.0 \\ 
  0.76 & 3013.8 & 74.6 \\ 
  0.77 & 2837.4 & 45.4 \\ 
  0.73 & 3399.1 & 113.1 \\ 
   \bottomrule
\end{tabular}
}
}
\end{table}
\begin{table}[ht]
\subfloat[caption][SAT12-RAND]{
\footnotesize{
% latex table generated in R 3.1.0 by xtable 1.7-4 package
% Tue Jan 27 11:27:36 2015
\begin{tabular}{llrrr}
  \toprule
Type & Model & Solved & PAR10 & MCP \\ 
  \midrule
baseline & singleBestByPar & 0.73 & 3271.1 & 49.4 \\ 
  baseline & singleBestByPar & 0.73 & 3271.1 & 49.4 \\ 
  baseline & vbs & 0.76 & 2872.8 & 0.0 \\ 
  baseline & vbs & 0.76 & 2872.8 & 0.0 \\ 
  classif & ksvm & 0.73 & 3245.4 & 42.5 \\ 
  classif & randomForest & 0.73 & 3300.3 & 50.0 \\ 
  classif & rpart & 0.74 & 3176.5 & 36.9 \\ 
  cluster & XMeans & 0.74 & 3149.6 & 33.7 \\ 
  regr & lm & 0.73 & 3254.6 & 43.7 \\ 
  regr & randomForest & 0.74 & 3210.3 & 39.1 \\ 
  regr & rpart & 0.72 & 3382.5 & 60.8 \\ 
   \bottomrule
\end{tabular}
}
}
\end{table}

results_top_tables.tex
\begin{table}[htb]
\begin{center}
\scriptsize
\begin{tabular}{lccc}
\toprule
 & 1st & 2nd & 3rd\\
\midrule
\pbox{.25\textwidth}{SAT11-HAND} & classif/ksvm & regr/randomForest & classif/randomForest\\
 & 0.656 & 0.688 & 0.702\\[.5em]
\pbox{.25\textwidth}{SAT11-INDU} & classif/randomForest & regr/lm & classif/rpart\\
 & 0.935 & 0.942 & 0.953\\[.5em]
\pbox{.25\textwidth}{SAT11-RAND} & regr/randomForest & classif/rpart & classif/randomForest\\
 & 0.515 & 0.534 & 0.544\\[.5em]
\pbox{.25\textwidth}{SAT12-ALL} & regr/randomForest & classif/randomForest & classif/ksvm\\
 & 0.339 & 0.352 & 0.418\\[.5em]
\pbox{.25\textwidth}{SAT12-HAND} & regr/randomForest & classif/randomForest & classif/ksvm\\
 & 0.682 & 0.704 & 0.747\\[.5em]
\pbox{.25\textwidth}{SAT12-INDU} & regr/randomForest & classif/randomForest & classif/ksvm\\
 & 0.848 & 0.859 & 0.88\\[.5em]
\pbox{.25\textwidth}{SAT12-RAND} & classif/rpart & regr/lm & regr/randomForest\\
 & 0.958 & 0.966 & 0.969\\[.5em]
\pbox{.25\textwidth}{SAT15-INDU} & regr/randomForest & classif/ksvm & classif/randomForest\\
 & 0.725 & 0.859 & 0.879\\[.5em]
\pbox{.25\textwidth}{QBF-2011} & regr/randomForest & classif/randomForest & classif/rpart\\
 & 0.599 & 0.641 & 0.671\\[.5em]
\pbox{.25\textwidth}{QBF-2014} & regr/randomForest & classif/randomForest & regr/rpart\\
 & 0.619 & 0.744 & 0.749\\[.5em]
\pbox{.25\textwidth}{MAXSAT12-PMS} & regr/randomForest & classif/randomForest & regr/lm\\
 & 0.674 & 0.765 & 0.772\\[.5em]
\pbox{.25\textwidth}{MAXSAT15-PMS-INDU} & regr/randomForest & regr/lm & cluster/XMeans\\
 & 0.748 & 0.82 & 0.824\\[.5em]
\pbox{.25\textwidth}{CSP-2010} & classif/rpart & regr/randomForest & classif/randomForest\\
 & 0.901 & 0.908 & 0.921\\[.5em]
\pbox{.25\textwidth}{CSP-MZN-2013} & regr/randomForest & regr/lm & classif/ksvm\\
 & 0.478 & 0.523 & 0.531\\[.5em]
\pbox{.25\textwidth}{PROTEUS-2014} & regr/randomForest & regr/lm & classif/randomForest\\
 & 0.436 & 0.465 & 0.492\\[.5em]
\pbox{.25\textwidth}{ASP-POTASSCO} & regr/randomForest & regr/lm & regr/rpart\\
 & 0.562 & 0.698 & 0.71\\[.5em]
\pbox{.25\textwidth}{PREMAR-ASTAR-2015} & \\
 & \\[.5em]
\bottomrule
\end{tabular}
\caption{Best performing approaches (by PAR10) across all data sets. The number underneath the name of the approach is the factor compared to the single best and has been rounded to three digits. That is, 0.5 means that the respective approach had a PAR10 score that was 0.5 times the score of the single best.}
\label{tab:top}
\end{center}
\end{table}

specification.tex
\section{Summary of Format Specification}
\label{sec:spec}

We propose a data format specification for algorithm selection scenarios,
i.e., instances of the per-instance algorithm selection problem.
This format and the resulting data repository allow a fair and convenient 
scientific evaluation and comparison of algorithm selectors.

The format specification assumes a generic approach to algorithm selection,
depicted in Figure~\ref{dia:as}. The general approach is as follows.

%\note{Reviewers}{You should consider providing a schematic/diagram overview of the system, before describing the format specification. 
%When I took a step back, I wanted to know more about the whole algorithm selection framework, not use the format.}
%\note{ML}{See Figure~\ref{dia:as} for a diagram; 
%HH: This is good, but I am confused about the use of $\mathcal{P(I)}$, which I don't see explained anywhere, and which, if it refers to a probability distribution over instances, does not make sense to me.
%It seems to me that feature cost should be a mapping from instances to vectors of positive reals.
%ML: $\mathcal{P}(X)$ was meant as a power set of $X$; I changed it to $2^X$ and of course, $X$ is $\mathcal{F}$ and not $\mathcal{I}$} 

%\note{ML}{
%I also considered an open issue raised by KLB that we did not considered the selection of schedules. This would also need some changes in the definition in the introduction. Right now, I have not included pre-solving schedules.
%HH: I think we should keep it that way; if we open this door too early and too wide, we are just inviting additional criticism regarding our experiments.}


\begin{figure}[tbp]
\resizebox{\textwidth}{!}{\input{pics/as_tikz.tex}}
\caption{Algorithm Selection workflow.}
\label{dia:as}
%\note{FH}{I see a couple of issues with the formalism:\\1. since $f$ maps to $\mathcal{F}$ and c maps from $2^{\mathcal{F}}$, you can't write
%$c(f(i))$ because $f(i)$ is a feature vector (an element of $\mathcal{F}$, not an element of $2^{\mathcal{F}}$). Did you mean to write $c(\mathcal{F}')$? But that also doesn't work since it would make the cost independent of the instance (only dependent on which feature subset you choose). But I think the real issue is the following:\\
%2. Unfortunately, $\mathcal{F}' \subseteq \mathcal{F}$ isn't formally clean, either: if $\mathcal{F}$ is a set of vectors of length $k$, then a subset of $\mathcal{F}$is also a set of vectors of length $k$. It's not immediately clear to me how to write it both simple and formally correct; it seems like you'd have to write $\mathcal{F}$ as the cross-product of individual dimensions, and then define $\mathcal{F}'$ as the cross-product of a subset of those dimensions.\\
%ML: I fixed it accordingly to the suggestion of Lars by not talking about feature subsets}
\end{figure}

\begin{enumerate}
  \item \changed{A vector of instance features $f(i) \in \mathcal{F}$} of $i$ is computed. 
  		Feature computation may occur in several stages,
  		each of which produces \changed{a group of (one or more) features}. Furthermore, later stages may depend on the results of earlier ones.
  	    Each feature group incurs a cost,
  	    e.g., runtime. If no features are required, the cost is $0$ (this occurs, e.g., for variants of algorithm selection that compute static schedules).
  \item A machine learning technique $s$ selects an algorithm $a \in \mathcal{A}$ based on the feature vector from Step 1.
  \item The selected algorithm $a$ is applied to $i$.
  \item Performance measure $m$ is evaluated, taking into account feature computation costs and the performance of the selected algorithm.
  \item \changed{Some algorithm selectors do not select a single algorithm, but
  compute a schedule of several algorithms:
  		they apply $a$ to $i$ for a resource budget $r \in \mathcal{R}$
        (e.g., CPU time), evaluate the performance metric, evaluate
        a stopping criterion, and repeat as necessary, taking observations made
        during the run of $a$ into account.\footnote{In \fh{principle}, the workflow can be arbitrarily more complex, e.g., alternating between computing further features and running selected algorithms.}}
\end{enumerate}
%\note{FH}{Is there any relationship between these steps and the blocks in the figure? If there could be, it would help if the blocks in the figure had matching numbers.
%ML: No, there is no perfect matching.
%}

\changed{The purpose of our library is to provide all information necessary for performing algorithm selection experiments
using the given scenario data.
The user does not need to actually run algorithms on instances, as all performance data is already \fh{precomputed}. 
\hh{This drastically reduces the time required for executing studies, i.e., 
the runtime of studies is now} dominated by the time required for learning $s$
and not by applying algorithms to instances (e.g., solving SAT problems).
It also means that results are perfectly \hh{reproducible};
for example, the runtimes of algorithms
do not depend on the hardware used;
\hh{rather, they can be simply looked up in the performance data for a} scenario.
}

Table~\ref{nutshell:spec} shows the basic structure of a scenario definition in
ASlib;
the complete specification with all details can be found in \fh{an accompanying technical report}~\cite{spec} and on our online platform.

%\note{Reviewer}{The description [of the format specification] could be simplified and improved (e.g., "A further file..").}

% All files, except the meta information and README file, follow the Attribute-Relation File Format (ARFF)
% which was developed for the \texttt{Weka} machine learning software and
% can nowadays be easily imported into many programming languages and toolboxes.
% It also offers more ways to define meta information than simple comma-separated-values (CSV) files,
% e.g., the specification of data types for each column and the definitions of missing values.
% Some information is mandatory for any algorithm selection scenario while other information is only sometimes available and useful.
% In the following, we describe these types of data.

\noindent\fbox{
\begin{minipage}[tbh]{0.96\textwidth} 
\captionof{table}{Overview of a scenario in the ASlib format.}
\label{nutshell:spec}
\paragraph{Mandatory Data}
\begin{itemize}
  	\item The \textit{meta information file} is a global description file containing general information about 
		the scenario, including the name of the scenario, performance measures, algorithms, features and 
		limitations of computational resources.
	\item The \textit{algorithm performance} file contains performance measurements \changedTwo{with possible repetitions} and completion status of the algorithm runs. \changedTwo{The performance metric can be arbitrary, e.g., runtime, solution quality, accuracy or loss.}
	\item The \textit{instance feature} file contains the feature vectors for all instances. Another file contains technical information
	about errors encountered or instances solved during feature computation.
	\item The \textit{cross-validation} file describes how to split the instance set into training and test sets
		to apply a standard machine learning approach to obtain an unbiased estimate of the performance of an algorithm selector.
    \item A human-readable \textit{README} file explains the origin and meaning of the scenario, as well as the process of data generation.
%HH: But the other files are also human-readable (as opposed to binary, right? I'd rather say "plain text" here.
\end{itemize}
\paragraph{Optional Data}
\begin{itemize}
  	\item The \textit{feature costs} file contains the costs of the feature groups\changed{, i.e., sets of features computed together.}
  	\item  The \textit{ground truth} file specifies information on the instances and their 
		respective solutions \changed{(e.g., SAT or UNSAT)}.
%HH: truths sounds awkward, so I changed it to "truth"		
	\item The \textit{literature references} file \changed{in BibTeX format} includes information on the context in which the data set was generated 
		and previous studies in which it was used.
\end{itemize}
\end{minipage}
}
 
% \note{Reviewer I}{
% Would it make sense to support performance information that is not just the
% plain runtime, but an average runtime and other statistical measures such as
% standard deviation. One does not even need to support those measures in
% the portfolios directly.\\
% ML: I added a sentence in Table~\ref{nutshell:spec} that we support arbitrary performance metrics.
% }
 
% \note{FH}{This list of what information is in which file is at a much lower level than the remainder of the paper, and it breaks the flow.
% I agree this belongs in the paper since it is already the small, high-level, excerpt of the entire specification. Nevertheless, many readers, when 
% getting to this point, would either skip the description or get bored/stop reading. I propose to put the information about the files and their contents into a table (one table each for mandatory/optional; it could also be a figure or a minipage with a visual border, anything distinct from the main text), and to pull out a little bit of text from that table into the main text. That will both make it easy for readers to skip the info and for people looking for the info to find it.}

% \subsection{Mandatory Data}
% 
% In a nutshell, an algorithm scenario has to provide meta information,
% performance of algorithm runs, feature vectors
% and a README.
% 
% \paragraph{Meta Information}
% The meta information file is a global description file containing 
% general information about the scenario.
% This is one of the few files which is not an 
% ARFF, but has its own, simple format, consisting mainly of lines with names and corresponding values.
% It includes:
% the name of the scenario;
% the measured performance metrics types of the algorithms and whether runtime or solution quality is optimized;
% the names of all available algorithms;
% the names of all available instance features;
% whether algorithms and features are stochastic or deterministic;
% resource limits of algorithm runs and feature computation.
% Furthermore, 
% instance features are normally computed in a sequence of processing steps.
% Thereby, each step can generate several features
% and a feature can depend on a list of steps.
% This n-to-m relationship can also be expressed here.
% 
% \paragraph{Algorithm Performance}
% The file about algorithm performance contains information on all evaluating runs of 
% the algorithms on the instances. 
% This includes, but is not limited to, their measured performance values and run status,
% e.g., whether an algorithm crashed or was aborted because its run hit its predefined
% cut-off time.
% In practice, certain (stochastic) algorithms may be evaluated multiple times,
% while other (deterministic) algorithms are run just once. 
% To cover both scenarios, each row of this file specifies a single repetition of an algorithm run.
% 
% \paragraph{Instance Features}
% The instance feature file contains the measured feature values for all instances. 
% It is assumed that this information 
% will be used to differentiate between different instances and to predict suitable algorithms
% by using a prognostic model.
% Also, feature generators can be stochastic 
% and therefore we allow repeated measurements of instance features.
% 
% \paragraph{Feature Computation Runstatus}
% A further file contains technical information about the feature calculation in general. 
% It states whether the execution of feature processing steps was successful, 
% if some kind of unusual abort occurred
% or the instance was solved during the feature computation.
% 
% % The file is designed to allow the user to specify what kind of a crash has transpired.
% % Only these instance features can be used where 
% % the depending feature processing steps were marked as successful in this file.
% 
% \paragraph{README}
% Last but not least, a human-readable README file must be provided,
% that explains the origin and meaning of the scenario, the process of the data generation 
% and all of peculiarities, e.g, why a certain crash has occurred or why certain values are missing.
% 
% \subsection{Optional Data}
% 
% We encourage but do not force the contributors of algorithm scenarios to also provide the following information.
% 
% \paragraph{Feature Costs}
% Although usually a minor overhead, feature computation is rarely free. In many cases this cost occurs in the form of 
% time, but no such requirement is made. Instead, this file specifies the cost of computing each feature processing step
% which was defined in the meta information file.
% The costs of a feature set is the sum of the costs of all feature steps 
% which were necessary to compute the feature set.
% If an instance was solved during a feature processing step,
% only the costs until this presolving step matter.
% 
% % We strongly recommend to provide information about feature costs.
% % However, you are allowed to omit this file if your scenario does not entail feature costs or you have 
% % absolutely no knowledge of them. 
% % \todo{FH: some parts of this aren't digested to being part of a paper yet (e.g., ``you are allowed to omit this file if your scenario'' is fine as part of a manual, but not part of an article for the wide readership of AIJ). }
% % Please note that certain analysis, \eg{} runtime improvement against best single algorithm, are not possible,
% % if feature costs are not provided. 
% 
% \paragraph{Validation}
% The cross-validation file contains information how to split the set of instances into training and test sets
% to assess the performance of an algorithm selector in an unbiased and standard machine learning fashion.
% The file should be provided to guarantee that all algorithm selectors are evaluated on the same splits.
% If it is not provided, it will be automatically generated when adding the data set to the repository.
% We support repeated cross-validation, where the process of a single cross-validation is simply
% run multiple times and performance estimators are averaged in the end.
% These repetitions are a standard technique for estimator variance reduction
% on smaller data sets or in cases 
% where other sources of instability might be present in the data.
% 
% \paragraph{Ground Truths}
% The file about ground truths is an optional file meant to specify 
% further available information about the instances and their respective solutions. 
% This can be used both as a sanity check 
% to make sure the solvers return correct solutions, 
% but also for cases where an evaluation metric depends on the optimal solution. 
% Furthermore, some selection methods can exploit structured information about the problem instances.
% For example, \texttt{satzilla}~\cite{xu_satzilla_2008} used hierarchical models 
% exploiting ground truths whether a problem instance is satisfiable or unsatisfiable in case of the 
% boolean satisfiability problem.
% 
% \paragraph{Citation}
% At last,
% there can be file which includes literature references to cite 
% in which context the data set was generated
% or where it was used.

elsarticle-template-num.tex
%%
%% Copyright 2007, 2008, 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%

%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%% SP 2008/03/01
%%
%%
%%
%% $Id: elsarticle-template-num.tex 4 2009-10-24 08:22:58Z rishi $
%%
%%
\documentclass[preprint,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%
%% \biboptions{comma,round}

% \biboptions{}


\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

%%
%% Start line numbering here if you want
%%
% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%% Usage of \cite is as follows:
%%   \cite{key}         ==>>  [#]
%%   \cite[chap. 2]{key} ==>> [#, chap. 2]
%%

%% References with bibTeX database:

\bibliographystyle{elsarticle-num}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use elsarticle-num.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have the following form:
%%   \bibitem{key}...
%%

% \bibitem{}

% \end{thebibliography}


\end{document}

%%
%% End of file `elsarticle-template-num.tex'.

elsarticle-template-5-harv.tex
%% This is file `elsarticle-template-5-harv.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with harvard style bibliographic references
%%
%% $Id: elsarticle-template-5-harv.tex 159 2009-10-08 06:08:33Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-5-harv.tex $
%%
\documentclass[preprint,authoryear,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[authoryear,preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,authoryear,1p,times]{elsarticle}
%% \documentclass[final,authoryear,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,authoryear,3p,times]{elsarticle}
%% \documentclass[final,authoryear,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,authoryear,5p,times]{elsarticle}
%% \documentclass[final,authoryear,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon (default)
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   authoryear - selects author-year citations (default)
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%   longnamesfirst  -  makes first citation full author list
%%
%% \biboptions{longnamesfirst,comma}

% \biboptions{}

\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%%
%%  \citet{key}  ==>>  Jones et al. (1990)
%%  \citep{key}  ==>>  (Jones et al., 1990)
%%
%% Multiple citations as normal:
%% \citep{key1,key2}         ==>> (Jones et al., 1990; Smith, 1989)
%%                            or  (Jones et al., 1990, 1991)
%%                            or  (Jones et al., 1990a,b)
%% \cite{key} is the equivalent of \citet{key} in author-year mode
%%
%% Full author lists may be forced with \citet* or \citep*, e.g.
%%   \citep*{key}            ==>> (Jones, Baker, and Williams, 1990)
%%
%% Optional notes as:
%%   \citep[chap. 2]{key}    ==>> (Jones et al., 1990, chap. 2)
%%   \citep[e.g.,][]{key}    ==>> (e.g., Jones et al., 1990)
%%   \citep[see][pg. 34]{key}==>> (see Jones et al., 1990, pg. 34)
%%  (Note: in standard LaTeX, only one note is allowed, after the ref.
%%   Here, one note is like the standard, two make pre- and post-notes.)
%%
%%   \citealt{key}          ==>> Jones et al. 1990
%%   \citealt*{key}         ==>> Jones, Baker, and Williams 1990
%%   \citealp{key}          ==>> Jones et al., 1990
%%   \citealp*{key}         ==>> Jones, Baker, and Williams, 1990
%%
%% Additional citation possibilities
%%   \citeauthor{key}       ==>> Jones et al.
%%   \citeauthor*{key}      ==>> Jones, Baker, and Williams
%%   \citeyear{key}         ==>> 1990
%%   \citeyearpar{key}      ==>> (1990)
%%   \citetext{priv. comm.} ==>> (priv. comm.)
%%   \citenum{key}          ==>> 11 [non-superscripted]
%% Note: full author lists depends on whether the bib style supports them;
%%       if not, the abbreviated list is printed even when full requested.
%%
%% For names like della Robbia at the start of a sentence, use
%%   \Citet{dRob98}         ==>> Della Robbia (1998)
%%   \Citep{dRob98}         ==>> (Della Robbia, 1998)
%%   \Citeauthor{dRob98}    ==>> Della Robbia


%% References with bibTeX database:

\bibliographystyle{model5-names}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model5-names.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have one of the following forms:
%%   \bibitem[Jones et al.(1990)]{key}...
%%   \bibitem[Jones et al.(1990)Jones, Baker, and Williams]{key}...
%%   \bibitem[Jones et al., 1990]{key}...
%%   \bibitem[\protect\citeauthoryear{Jones, Baker, and Williams}{Jones
%%       et al.}{1990}]{key}...
%%   \bibitem[\protect\citeauthoryear{Jones et al.}{1990}]{key}...
%%   \bibitem[\protect\astroncite{Jones et al.}{1990}]{key}...
%%   \bibitem[\protect\citename{Jones et al., }1990]{key}...
%%   \harvarditem[Jones et al.]{Jones, Baker, and Williams}{1990}{key}...
%%

% \bibitem[ ()]{}

% \end{thebibliography}

\end{document}

%%
%% End of file `elsarticle-template-5-harv.tex'.

elsarticle-template-4-harv.tex
%% This is file `elsarticle-template-4-harv.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with harvard style bibliographic references
%%
%% $Id: elsarticle-template-4-harv.tex 167 2009-10-08 07:58:22Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-4-harv.tex $
%%
\documentclass[preprint,authoryear,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[authoryear,preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,authoryear,1p,times]{elsarticle}
%% \documentclass[final,authoryear,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,authoryear,3p,times]{elsarticle}
%% \documentclass[final,authoryear,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,authoryear,5p,times]{elsarticle}
%% \documentclass[final,authoryear,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The numcompress package shorten the last page in references.
%% `nodots' option removes dots from firstnames in references.
\usepackage[nodots]{numcompress}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon (default)
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   authoryear - selects author-year citations (default)
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%   longnamesfirst  -  makes first citation full author list
%%
%% \biboptions{longnamesfirst,comma}

% \biboptions{}

\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%%
%%  \citet{key}  ==>>  Jones et al. (1990)
%%  \citep{key}  ==>>  (Jones et al., 1990)
%%
%% Multiple citations as normal:
%% \citep{key1,key2}         ==>> (Jones et al., 1990; Smith, 1989)
%%                            or  (Jones et al., 1990, 1991)
%%                            or  (Jones et al., 1990a,b)
%% \cite{key} is the equivalent of \citet{key} in author-year mode
%%
%% Full author lists may be forced with \citet* or \citep*, e.g.
%%   \citep*{key}            ==>> (Jones, Baker, and Williams, 1990)
%%
%% Optional notes as:
%%   \citep[chap. 2]{key}    ==>> (Jones et al., 1990, chap. 2)
%%   \citep[e.g.,][]{key}    ==>> (e.g., Jones et al., 1990)
%%   \citep[see][pg. 34]{key}==>> (see Jones et al., 1990, pg. 34)
%%  (Note: in standard LaTeX, only one note is allowed, after the ref.
%%   Here, one note is like the standard, two make pre- and post-notes.)
%%
%%   \citealt{key}          ==>> Jones et al. 1990
%%   \citealt*{key}         ==>> Jones, Baker, and Williams 1990
%%   \citealp{key}          ==>> Jones et al., 1990
%%   \citealp*{key}         ==>> Jones, Baker, and Williams, 1990
%%
%% Additional citation possibilities
%%   \citeauthor{key}       ==>> Jones et al.
%%   \citeauthor*{key}      ==>> Jones, Baker, and Williams
%%   \citeyear{key}         ==>> 1990
%%   \citeyearpar{key}      ==>> (1990)
%%   \citetext{priv. comm.} ==>> (priv. comm.)
%%   \citenum{key}          ==>> 11 [non-superscripted]
%% Note: full author lists depends on whether the bib style supports them;
%%       if not, the abbreviated list is printed even when full requested.
%%
%% For names like della Robbia at the start of a sentence, use
%%   \Citet{dRob98}         ==>> Della Robbia (1998)
%%   \Citep{dRob98}         ==>> (Della Robbia, 1998)
%%   \Citeauthor{dRob98}    ==>> Della Robbia


%% References with bibTeX database:

\bibliographystyle{model4-names}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model4-names.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have one of the following forms:
%%   \bibitem[Jones et al.(1990)]{key}...
%%   \bibitem[Jones et al.(1990)Jones, Baker, and Williams]{key}...
%%   \bibitem[Jones et al., 1990]{key}...
%%   \bibitem[\protect\citeauthoryear{Jones, Baker, and Williams}{Jones
%%       et al.}{1990}]{key}...
%%   \bibitem[\protect\citeauthoryear{Jones et al.}{1990}]{key}...
%%   \bibitem[\protect\astroncite{Jones et al.}{1990}]{key}...
%%   \bibitem[\protect\citename{Jones et al., }1990]{key}...
%%   \harvarditem[Jones et al.]{Jones, Baker, and Williams}{1990}{key}...
%%

% \bibitem[ ()]{}

% \end{thebibliography}

\end{document}

%%
%% End of file `elsarticle-template-4-harv.tex'.

elsarticle-template-3a-num.tex
%% This is file `elsarticle-template-3a-num.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%%
%% $Id: elsarticle-template-3a-num.tex 166 2009-10-08 07:58:13Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-3a-num.tex $
%%
\documentclass[preprint,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The numcompress package shorten the last page in references.
%% `nodots' option removes dots from firstnames in references.
%% `nocompress' option prevent shortening of last page as
%% by default it will shorten.
\usepackage[nodots,nocompress]{numcompress}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%
%% \biboptions{comma,round}

\biboptions{super}


\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

%%
%% Start line numbering here if you want
%%
% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%% Usage of \cite is as follows:
%%   \cite{key}          ==>>  [#]
%%   \cite[chap. 2]{key} ==>>  [#, chap. 2]
%%   \citet{key}         ==>>  Author [#]

%% References with bibTeX database:

\bibliographystyle{model3a-num-names}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model3a-num-names.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have the following form:
%%   \bibitem{key}...
%%

% \bibitem{}

% \end{thebibliography}


\end{document}

%%
%% End of file `elsarticle-template-3a-num.tex'.

elsarticle-template-1-num.tex
%% This is file `elsarticle-template-1-num.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%%
%% $Id: elsarticle-template-1-num.tex 149 2009-10-08 05:01:15Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-1-num.tex $
%%
\documentclass[preprint,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%
%% \biboptions{comma,round}

% \biboptions{}


\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

%%
%% Start line numbering here if you want
%%
% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%% Usage of \cite is as follows:
%%   \cite{key}          ==>>  [#]
%%   \cite[chap. 2]{key} ==>>  [#, chap. 2]
%%   \citet{key}         ==>>  Author [#]

%% References with bibTeX database:

\bibliographystyle{model1-num-names}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model1-num-names.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have the following form:
%%   \bibitem{key}...
%%

% \bibitem{}

% \end{thebibliography}


\end{document}

%%
%% End of file `elsarticle-template-1-num.tex'.

elsarticle-template-2-harv.tex
%% This is file `elsarticle-template-2-harv.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with harvard style bibliographic references
%%
%% $Id: elsarticle-template-2-harv.tex 155 2009-10-08 05:35:05Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-2-harv.tex $
%%
\documentclass[preprint,authoryear,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[authoryear,preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,authoryear,1p,times]{elsarticle}
%% \documentclass[final,authoryear,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,authoryear,3p,times]{elsarticle}
%% \documentclass[final,authoryear,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,authoryear,5p,times]{elsarticle}
%% \documentclass[final,authoryear,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon (default)
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   authoryear - selects author-year citations (default)
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%   longnamesfirst  -  makes first citation full author list
%%
%% \biboptions{longnamesfirst,comma}

% \biboptions{}

\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%%
%%  \citet{key}  ==>>  Jones et al. (1990)
%%  \citep{key}  ==>>  (Jones et al., 1990)
%%
%% Multiple citations as normal:
%% \citep{key1,key2}         ==>> (Jones et al., 1990; Smith, 1989)
%%                            or  (Jones et al., 1990, 1991)
%%                            or  (Jones et al., 1990a,b)
%% \cite{key} is the equivalent of \citet{key} in author-year mode
%%
%% Full author lists may be forced with \citet* or \citep*, e.g.
%%   \citep*{key}            ==>> (Jones, Baker, and Williams, 1990)
%%
%% Optional notes as:
%%   \citep[chap. 2]{key}    ==>> (Jones et al., 1990, chap. 2)
%%   \citep[e.g.,][]{key}    ==>> (e.g., Jones et al., 1990)
%%   \citep[see][pg. 34]{key}==>> (see Jones et al., 1990, pg. 34)
%%  (Note: in standard LaTeX, only one note is allowed, after the ref.
%%   Here, one note is like the standard, two make pre- and post-notes.)
%%
%%   \citealt{key}          ==>> Jones et al. 1990
%%   \citealt*{key}         ==>> Jones, Baker, and Williams 1990
%%   \citealp{key}          ==>> Jones et al., 1990
%%   \citealp*{key}         ==>> Jones, Baker, and Williams, 1990
%%
%% Additional citation possibilities
%%   \citeauthor{key}       ==>> Jones et al.
%%   \citeauthor*{key}      ==>> Jones, Baker, and Williams
%%   \citeyear{key}         ==>> 1990
%%   \citeyearpar{key}      ==>> (1990)
%%   \citetext{priv. comm.} ==>> (priv. comm.)
%%   \citenum{key}          ==>> 11 [non-superscripted]
%% Note: full author lists depends on whether the bib style supports them;
%%       if not, the abbreviated list is printed even when full requested.
%%
%% For names like della Robbia at the start of a sentence, use
%%   \Citet{dRob98}         ==>> Della Robbia (1998)
%%   \Citep{dRob98}         ==>> (Della Robbia, 1998)
%%   \Citeauthor{dRob98}    ==>> Della Robbia


%% References with bibTeX database:

\bibliographystyle{model2-names}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model2-names.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have one of the following forms:
%%   \bibitem[Jones et al.(1990)]{key}...
%%   \bibitem[Jones et al.(1990)Jones, Baker, and Williams]{key}...
%%   \bibitem[Jones et al., 1990]{key}...
%%   \bibitem[\protect\citeauthoryear{Jones, Baker, and Williams}{Jones
%%       et al.}{1990}]{key}...
%%   \bibitem[\protect\citeauthoryear{Jones et al.}{1990}]{key}...
%%   \bibitem[\protect\astroncite{Jones et al.}{1990}]{key}...
%%   \bibitem[\protect\citename{Jones et al., }1990]{key}...
%%   \harvarditem[Jones et al.]{Jones, Baker, and Williams}{1990}{key}...
%%

% \bibitem[ ()]{}

% \end{thebibliography}

\end{document}

%%
%% End of file `elsarticle-template-2-harv.tex'.

elsarticle-template-1a-num.tex
%% This is file `elsarticle-template-1a-num.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%%
%% $Id: elsarticle-template-1a-num.tex 151 2009-10-08 05:18:25Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-1a-num.tex $
%%
\documentclass[preprint,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%
%% \biboptions{comma,round}

% \biboptions{}


\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

%%
%% Start line numbering here if you want
%%
% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%% Usage of \cite is as follows:
%%   \cite{key}          ==>>  [#]
%%   \cite[chap. 2]{key} ==>>  [#, chap. 2]
%%   \citet{key}         ==>>  Author [#]

%% References with bibTeX database:

\bibliographystyle{model1a-num-names}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model1a-num-names.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have the following form:
%%   \bibitem{key}...
%%

% \bibitem{}

% \end{thebibliography}


\end{document}

%%
%% End of file `elsarticle-template-1a-num.tex'.

elsarticle-template-1b-num.tex
%% This is file `elsarticle-template-1b-num.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%%
%% $Id: elsarticle-template-1b-num.tex 153 2009-10-08 05:26:41Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-1b-num.tex $
%%
\documentclass[preprint,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%
%% \biboptions{comma,round}

% \biboptions{}


\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

%%
%% Start line numbering here if you want
%%
% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%% Usage of \cite is as follows:
%%   \cite{key}          ==>>  [#]
%%   \cite[chap. 2]{key} ==>>  [#, chap. 2]
%%   \citet{key}         ==>>  Author [#]

%% References with bibTeX database:

\bibliographystyle{model1b-num-names}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model1b-num-names.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have the following form:
%%   \bibitem{key}...
%%

% \bibitem{}

% \end{thebibliography}


\end{document}

%%
%% End of file `elsarticle-template-1b-num.tex'.

elsarticle-template-3-num.tex
%% This is file `elsarticle-template-3-num.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%%
%% $Id: elsarticle-template-3-num.tex 165 2009-10-08 07:58:10Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-3-num.tex $
%%
\documentclass[preprint,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The numcompress package shorten the last page in references.
%% `nodots' option removes dots from firstnames in references.
\usepackage[nodots]{numcompress}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%
%% \biboptions{comma,round}

% \biboptions{}


\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

%%
%% Start line numbering here if you want
%%
% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%% Usage of \cite is as follows:
%%   \cite{key}          ==>>  [#]
%%   \cite[chap. 2]{key} ==>>  [#, chap. 2]
%%   \citet{key}         ==>>  Author [#]

%% References with bibTeX database:

\bibliographystyle{model3-num-names}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model3-num-names.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have the following form:
%%   \bibitem{key}...
%%

% \bibitem{}

% \end{thebibliography}


\end{document}

%%
%% End of file `elsarticle-template-3-num.tex'.

elsarticle-template-harv.tex
%%
%% Copyright 2007, 2008, 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%

%% Template article for Elsevier's document class `elsarticle'
%% with harvard style bibliographic references
%% SP 2008/03/01
%%
%%
%%
%% $Id: elsarticle-template-harv.tex 4 2009-10-24 08:22:58Z rishi $
%%
%%
\documentclass[preprint,authoryear,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[authoryear,preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,authoryear,1p,times]{elsarticle}
%% \documentclass[final,authoryear,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,authoryear,3p,times]{elsarticle}
%% \documentclass[final,authoryear,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,authoryear,5p,times]{elsarticle}
%% \documentclass[final,authoryear,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon (default)
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   authoryear - selects author-year citations (default)
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%   longnamesfirst  -  makes first citation full author list
%%
%% \biboptions{longnamesfirst,comma}

% \biboptions{}

\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%%
%%  \citet{key}  ==>>  Jones et al. (1990)
%%  \citep{key}  ==>>  (Jones et al., 1990)
%%
%% Multiple citations as normal:
%% \citep{key1,key2}         ==>> (Jones et al., 1990; Smith, 1989)
%%                            or  (Jones et al., 1990, 1991)
%%                            or  (Jones et al., 1990a,b)
%% \cite{key} is the equivalent of \citet{key} in author-year mode
%%
%% Full author lists may be forced with \citet* or \citep*, e.g.
%%   \citep*{key}            ==>> (Jones, Baker, and Williams, 1990)
%%
%% Optional notes as:
%%   \citep[chap. 2]{key}    ==>> (Jones et al., 1990, chap. 2)
%%   \citep[e.g.,][]{key}    ==>> (e.g., Jones et al., 1990)
%%   \citep[see][pg. 34]{key}==>> (see Jones et al., 1990, pg. 34)
%%  (Note: in standard LaTeX, only one note is allowed, after the ref.
%%   Here, one note is like the standard, two make pre- and post-notes.)
%%
%%   \citealt{key}          ==>> Jones et al. 1990
%%   \citealt*{key}         ==>> Jones, Baker, and Williams 1990
%%   \citealp{key}          ==>> Jones et al., 1990
%%   \citealp*{key}         ==>> Jones, Baker, and Williams, 1990
%%
%% Additional citation possibilities
%%   \citeauthor{key}       ==>> Jones et al.
%%   \citeauthor*{key}      ==>> Jones, Baker, and Williams
%%   \citeyear{key}         ==>> 1990
%%   \citeyearpar{key}      ==>> (1990)
%%   \citetext{priv. comm.} ==>> (priv. comm.)
%%   \citenum{key}          ==>> 11 [non-superscripted]
%% Note: full author lists depends on whether the bib style supports them;
%%       if not, the abbreviated list is printed even when full requested.
%%
%% For names like della Robbia at the start of a sentence, use
%%   \Citet{dRob98}         ==>> Della Robbia (1998)
%%   \Citep{dRob98}         ==>> (Della Robbia, 1998)
%%   \Citeauthor{dRob98}    ==>> Della Robbia


%% References with bibTeX database:

\bibliographystyle{elsarticle-harv}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use elsarticle-harv.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have one of the following forms:
%%   \bibitem[Jones et al.(1990)]{key}...
%%   \bibitem[Jones et al.(1990)Jones, Baker, and Williams]{key}...
%%   \bibitem[Jones et al., 1990]{key}...
%%   \bibitem[\protect\citeauthoryear{Jones, Baker, and Williams}{Jones
%%       et al.}{1990}]{key}...
%%   \bibitem[\protect\citeauthoryear{Jones et al.}{1990}]{key}...
%%   \bibitem[\protect\astroncite{Jones et al.}{1990}]{key}...
%%   \bibitem[\protect\citename{Jones et al., }1990]{key}...
%%   \harvarditem[Jones et al.]{Jones, Baker, and Williams}{1990}{key}...
%%

% \bibitem[ ()]{}

% \end{thebibliography}

\end{document}

%%
%% End of file `elsarticle-template-harv.tex'.

elsarticle-template-1c-num.tex
%% This is file `elsarticle-template-1c-num.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%%
%% $Id: elsarticle-template-1c-num.tex 154 2009-10-08 05:30:27Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-1c-num.tex $
%%
\documentclass[preprint,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%
%% \biboptions{comma,round}

% \biboptions{}


\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

%%
%% Start line numbering here if you want
%%
% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%% Usage of \cite is as follows:
%%   \cite{key}          ==>>  [#]
%%   \cite[chap. 2]{key} ==>>  [#, chap. 2]
%%   \citet{key}         ==>>  Author [#]

%% References with bibTeX database:

\bibliographystyle{model1c-num-names}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model1c-num-names.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have the following form:
%%   \bibitem{key}...
%%

% \bibitem{}

% \end{thebibliography}


\end{document}

%%
%% End of file `elsarticle-template-1c-num.tex'.

elsarticle-template-6-num.tex
%% This is file `elsarticle-template-6-num.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%%
%% $Id: elsarticle-template-6-num.tex 169 2009-10-08 09:17:45Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-6-num.tex $
%%
\documentclass[preprint,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The numcompress package shorten the last page in references.
%% `nodots' option removes dots from firstnames in references.
%% `nocompress' option prevent shortening of last page as
%% by default it will shorten.
\usepackage[nodots,nocompress]{numcompress}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%
%% \biboptions{comma,round}

\biboptions{super}


\journal{Nuclear Physics B}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{}

\address{}

\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

%%
%% Start line numbering here if you want
%%
% \linenumbers

%% main text
\section{}
\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%% Usage of \cite is as follows:
%%   \cite{key}          ==>>  [#]
%%   \cite[chap. 2]{key} ==>>  [#, chap. 2]
%%   \citet{key}         ==>   Author [#]

%% References with bibTeX database:

\bibliographystyle{model6-num-names}
\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model6-num-names.bst.

%% References without bibTeX database:

% \begin{thebibliography}{00}

%% \bibitem must have the following form:
%%   \bibitem{key}...
%%

% \bibitem{}

% \end{thebibliography}


\end{document}

%%
%% End of file `elsarticle-template-6-num.tex'.

paper.bbl
\begin{thebibliography}{121}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\href}[2]{#2}
\providecommand{\path}[1]{#1}
\providecommand{\DOIprefix}{doi:}
\providecommand{\ArXivprefix}{arXiv:}
\providecommand{\URLprefix}{URL: }
\providecommand{\Pubmedprefix}{pmid:}
\providecommand{\doi}[1]{\href{http://dx.doi.org/#1}{\path{#1}}}
\providecommand{\Pubmed}[1]{\href{pmid:#1}{\path{#1}}}
\providecommand{\bibinfo}[2]{#2}
\ifx\xfnm\relax \def\xfnm[#1]{\unskip,\space#1}\fi
%Type = Inproceedings
\bibitem[{Amadini et~al.(2014a)Amadini, Gabbrielli and Mauro}]{AmadiniGM14a}
\bibinfo{author}{Amadini, R.}, \bibinfo{author}{Gabbrielli, M.},
  \bibinfo{author}{Mauro, J.}, \bibinfo{year}{2014}a.
\newblock \bibinfo{title}{An enhanced features extractor for a portfolio of
  constraint solvers}, in: \bibinfo{booktitle}{Symposium on Applied Computing,
  {SAC} 2014, Gyeongju, Republic of Korea - March 24 - 28, 2014}, pp.
  \bibinfo{pages}{1357--1359}.
%Type = Article
\bibitem[{Amadini et~al.(2014b)Amadini, Gabbrielli and Mauro}]{AmadiniGM14}
\bibinfo{author}{Amadini, R.}, \bibinfo{author}{Gabbrielli, M.},
  \bibinfo{author}{Mauro, J.}, \bibinfo{year}{2014}b.
\newblock \bibinfo{title}{{SUNNY:} a lazy portfolio approach for constraint
  solving}.
\newblock \bibinfo{journal}{{TPLP}} \bibinfo{volume}{14},
  \bibinfo{pages}{509--524}.
%Type = Inproceedings
\bibitem[{Ans\'{o}tegui et~al.(2014)Ans\'{o}tegui, Malitsky and
  Sellmann}]{ISAC++}
\bibinfo{author}{Ans\'{o}tegui, C.}, \bibinfo{author}{Malitsky, Y.},
  \bibinfo{author}{Sellmann, M.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{{MaxSAT by Improved Instance-Specific Algorithm
  Configuration}}, in: \bibinfo{booktitle}{Proceedings of the {Twenty-Eighth}
  National Conference on Artificial Intelligence}, pp.
  \bibinfo{pages}{2594--260}.
%Type = Inproceedings
\bibitem[{Ans{\'o}tegui et~al.(2009)Ans{\'o}tegui, Sellmann and
  Tierney}]{ansotegui_gender-based_2009}
\bibinfo{author}{Ans{\'o}tegui, C.}, \bibinfo{author}{Sellmann, M.},
  \bibinfo{author}{Tierney, K.}, \bibinfo{year}{2009}.
\newblock \bibinfo{title}{A gender-based genetic algorithm for the automatic
  configuration of algorithms}, in: \bibinfo{booktitle}{Proceedings of the
  Fifteenth International Conference on Principles and Practice of Constraint
  Programming (CP'09)}, pp. \bibinfo{pages}{142--157}.
%Type = Inproceedings
\bibitem[{Arbelaez et~al.(2010)Arbelaez, Hamadi and
  Sebag}]{arbelaez_continuous_2010}
\bibinfo{author}{Arbelaez, A.}, \bibinfo{author}{Hamadi, Y.},
  \bibinfo{author}{Sebag, M.}, \bibinfo{year}{2010}.
\newblock \bibinfo{title}{Continuous search in constraint programming}, in:
  \bibinfo{booktitle}{Proceedings of the {Twenty-Second} {IEEE} International
  Conference on Tools with Artificial Intelligence}, pp.
  \bibinfo{pages}{53--60}.
%Type = Inproceedings
\bibitem[{Argelich et~al.(2010)Argelich, Berre, Lynce, Marques-Silva and
  Rapicault}]{ArBeLyMaRa10}
\bibinfo{author}{Argelich, J.}, \bibinfo{author}{Berre, D.L.},
  \bibinfo{author}{Lynce, I.}, \bibinfo{author}{Marques-Silva, J.},
  \bibinfo{author}{Rapicault, P.}, \bibinfo{year}{2010}.
\newblock \bibinfo{title}{Solving linux upgradeability problems using boolean
  optimization}, in: \bibinfo{booktitle}{Proceedings of the International
  Workshop on Logics for Component Configuration}, pp. \bibinfo{pages}{11--22}.
%Type = Misc
\bibitem[{Argelich et~al.(2012)Argelich, Li, Many{\`a} and
  Planes}]{MAXSATevaluations}
\bibinfo{author}{Argelich, J.}, \bibinfo{author}{Li, C.},
  \bibinfo{author}{Many{\`a}, F.}, \bibinfo{author}{Planes, J.},
  \bibinfo{year}{2012}.
\newblock \bibinfo{title}{{Seventh MaxSAT Evaluation}}.
\newblock \bibinfo{note}{\url{http://www.maxsat.udl.cat/12/}}.
%Type = Book
\bibitem[{Baral(2003)}]{baral02a}
\bibinfo{author}{Baral, C.}, \bibinfo{year}{2003}.
\newblock \bibinfo{title}{Knowledge Representation, Reasoning and Declarative
  Problem Solving}.
\newblock \bibinfo{publisher}{Cambridge University Press}.
%Type = Inproceedings
\bibitem[{Biere(2014)}]{lingeling}
\bibinfo{author}{Biere, A.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Yet another local search solver and {Lingeling} and
  friends entering the {SAT} competition 2014}, in:
  \bibinfo{booktitle}{Proceedings of {SAT} Competition 2014: Solver and
  Benchmark Descriptions}, pp. \bibinfo{pages}{39--40}.
%Type = Book
\bibitem[{Biere et~al.(2009)Biere, Heule, {van Maaren} and Walsh}]{SATHandbook}
\bibinfo{editor}{Biere, A.}, \bibinfo{editor}{Heule, M.}, \bibinfo{editor}{{van
  Maaren}, H.}, \bibinfo{editor}{Walsh, T.} (Eds.), \bibinfo{year}{2009}.
\newblock \bibinfo{title}{Handbook of Satisfiability}. volume
  \bibinfo{volume}{185} of \textit{\bibinfo{series}{Frontiers in Artificial
  Intelligence and Applications}}.
\newblock \bibinfo{publisher}{IOS Press}.
%Type = Incollection
\bibitem[{Birattari et~al.(2010)Birattari, Yuan, Balaprakash and
  St\"{u}tzle}]{BirEtAl10}
\bibinfo{author}{Birattari, M.}, \bibinfo{author}{Yuan, Z.},
  \bibinfo{author}{Balaprakash, P.}, \bibinfo{author}{St\"{u}tzle, T.},
  \bibinfo{year}{2010}.
\newblock in: \bibinfo{booktitle}{Empirical Methods for the Analysis of
  Optimization Algorithms}. \bibinfo{publisher}{Springer}. chapter
  \bibinfo{chapter}{{F-race and iterated F-race: {An} overview}}.
%Type = Techreport
\bibitem[{Bischl et~al.(2014)Bischl, Kotthoff, Lindauer, Malitsky, Fr\'echette,
  Hoos, Hutter, Kerschke, Leyton-Brown and Vanschoren}]{spec}
\bibinfo{author}{Bischl, B.}, \bibinfo{author}{Kotthoff, L.},
  \bibinfo{author}{Lindauer, M.}, \bibinfo{author}{Malitsky, Y.},
  \bibinfo{author}{Fr\'echette, A.}, \bibinfo{author}{Hoos, H.},
  \bibinfo{author}{Hutter, F.}, \bibinfo{author}{Kerschke, P.},
  \bibinfo{author}{Leyton-Brown, K.}, \bibinfo{author}{Vanschoren, J.},
  \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Algorithm Selection Format Specification}.
\newblock \bibinfo{type}{Technical Report}.
\newblock \bibinfo{note}{Available at \url{http://www.aslib.net/}}.
%Type = Article
\bibitem[{Bischl et~al.(2015a)Bischl, Lang, Mersmann, Rahnenf{\"u}hrer and
  Weihs}]{Bischl2015_1}
\bibinfo{author}{Bischl, B.}, \bibinfo{author}{Lang, M.},
  \bibinfo{author}{Mersmann, O.}, \bibinfo{author}{Rahnenf{\"u}hrer, J.},
  \bibinfo{author}{Weihs, C.}, \bibinfo{year}{2015}a.
\newblock \bibinfo{title}{{BatchJobs} and {BatchExperiments}: Abstraction
  mechanisms for using {R} in batch environments}.
\newblock \bibinfo{journal}{Journal of Statistical Software}
  \bibinfo{volume}{64}, \bibinfo{pages}{1--25}.
%Type = Misc
\bibitem[{Bischl et~al.(2015b)Bischl, Lang, Richter, Bossek, Judt, Kuehn,
  Studerus, Kotthoff and Jones}]{bischl_mlr}
\bibinfo{author}{Bischl, B.}, \bibinfo{author}{Lang, M.},
  \bibinfo{author}{Richter, J.}, \bibinfo{author}{Bossek, J.},
  \bibinfo{author}{Judt, L.}, \bibinfo{author}{Kuehn, T.},
  \bibinfo{author}{Studerus, E.}, \bibinfo{author}{Kotthoff, L.},
  \bibinfo{author}{Jones, Z.}, \bibinfo{year}{2015}b.
\newblock \bibinfo{title}{{mlr: Machine Learning in R}}.
\newblock \bibinfo{note}{{R package version 2.7.
  \url{https://github.com/mlr-org/mlr}}}.
%Type = Inproceedings
\bibitem[{Bischl et~al.(2012a)Bischl, Mersmann, Trautmann and
  Preuss}]{Bischl2012_2}
\bibinfo{author}{Bischl, B.}, \bibinfo{author}{Mersmann, O.},
  \bibinfo{author}{Trautmann, H.}, \bibinfo{author}{Preuss, M.},
  \bibinfo{year}{2012}a.
\newblock \bibinfo{title}{Algorithm selection based on exploratory landscape
  analysis and cost-sensitive learning}, in: \bibinfo{booktitle}{Proceedings of
  the Fourteenth Annual Conference on Genetic and Evolutionary Computation},
  pp. \bibinfo{pages}{313--320}.
%Type = Article
\bibitem[{Bischl et~al.(2012b)Bischl, Mersmann, Trautmann and
  Weihs}]{Bischl2012_3}
\bibinfo{author}{Bischl, B.}, \bibinfo{author}{Mersmann, O.},
  \bibinfo{author}{Trautmann, H.}, \bibinfo{author}{Weihs, C.},
  \bibinfo{year}{2012}b.
\newblock \bibinfo{title}{Resampling methods for meta-model validation with
  recommendations for evolutionary computation}.
\newblock \bibinfo{journal}{Evolutionary Computation} \bibinfo{volume}{20},
  \bibinfo{pages}{249--275}.
%Type = Book
\bibitem[{Brazdil et~al.(2008)Brazdil, Giraud-Carrier, Soares and
  Vilalta}]{Brazdil_metalearning_2008}
\bibinfo{author}{Brazdil, P.}, \bibinfo{author}{Giraud-Carrier, C.},
  \bibinfo{author}{Soares, C.}, \bibinfo{author}{Vilalta, R.},
  \bibinfo{year}{2008}.
\newblock \bibinfo{title}{Metalearning: Applications to Data Mining}.
\newblock \bibinfo{edition}{1st} ed., \bibinfo{publisher}{Springer}.
%Type = Inproceedings
\bibitem[{Cicirello and Smith(2005)}]{cicirello_max_2005}
\bibinfo{author}{Cicirello, V.A.}, \bibinfo{author}{Smith, S.F.},
  \bibinfo{year}{2005}.
\newblock \bibinfo{title}{The max k-armed bandit: A new model of exploration
  applied to search heuristic selection}, in: \bibinfo{booktitle}{Proceedings
  of the Twentieth National Conference on Artificial Intelligence},
  \bibinfo{publisher}{{AAAI} Press}. pp. \bibinfo{pages}{1355--1361}.
%Type = Inproceedings
\bibitem[{Cook and Varnell(1997)}]{cook_maximizing_1997}
\bibinfo{author}{Cook, D.J.}, \bibinfo{author}{Varnell, R.C.},
  \bibinfo{year}{1997}.
\newblock \bibinfo{title}{Maximizing the benefits of parallel search using
  machine learning}, in: \bibinfo{booktitle}{Proceedings of the Fourteenth
  National Conference on Artificial Intelligence}, \bibinfo{publisher}{{AAAI}
  Press}. pp. \bibinfo{pages}{559--564}.
%Type = Inproceedings
\bibitem[{Crawford and Baker(1994)}]{crawford1994experiment}
\bibinfo{author}{Crawford, J.M.}, \bibinfo{author}{Baker, A.B.},
  \bibinfo{year}{1994}.
\newblock \bibinfo{title}{Experimental results on the application of
  satisfiability algorithms to scheduling problems}, in: \bibinfo{booktitle}{In
  Proceedings of the Twelfth National Conference on Artificial Intelligence},
  pp. \bibinfo{pages}{1092--1097}.
%Type = Inproceedings
\bibitem[{Dan~Pelleg(2000)}]{Pelleg2000}
\bibinfo{author}{Dan~Pelleg, A.M.}, \bibinfo{year}{2000}.
\newblock \bibinfo{title}{X-means: Extending k-means with efficient estimation
  of the number of clusters}, in: \bibinfo{booktitle}{Proceedings of the
  Seventeenth International Conference on Machine Learning},
  \bibinfo{publisher}{Morgan Kaufmann}, \bibinfo{address}{San Francisco}. pp.
  \bibinfo{pages}{727--734}.
%Type = Article
\bibitem[{Demmel et~al.(2005)Demmel, Dongarra, Eijkhout, Fuentes, Petitet,
  Vuduc, Whaley and Yelick}]{demmel_self-adapting_2005}
\bibinfo{author}{Demmel, J.}, \bibinfo{author}{Dongarra, J.},
  \bibinfo{author}{Eijkhout, V.}, \bibinfo{author}{Fuentes, E.},
  \bibinfo{author}{Petitet, A.}, \bibinfo{author}{Vuduc, R.},
  \bibinfo{author}{Whaley, R.C.}, \bibinfo{author}{Yelick, K.},
  \bibinfo{year}{2005}.
\newblock \bibinfo{title}{{Self-Adapting} linear algebra algorithms and
  software}.
\newblock \bibinfo{journal}{Proceedings of the {IEEE}} \bibinfo{volume}{93},
  \bibinfo{pages}{293--312}.
%Type = Inproceedings
\bibitem[{Domhan et~al.(2015)Domhan, Springenberg and Hutter}]{domhan-ijcai15a}
\bibinfo{author}{Domhan, T.}, \bibinfo{author}{Springenberg, J.T.},
  \bibinfo{author}{Hutter, F.}, \bibinfo{year}{2015}.
\newblock \bibinfo{title}{Speeding up automatic hyperparameter optimization of
  deep neural networks by extrapolation of learning curves}, in:
  \bibinfo{booktitle}{Proceedings of the 24th International Joint Conference on
  Artificial Intelligence (IJCAI)}.
%Type = Inproceedings
\bibitem[{Eggensperger et~al.(2015)Eggensperger, Hutter, Hoos and
  Leyton-Brown}]{Eggensperger2015}
\bibinfo{author}{Eggensperger, K.}, \bibinfo{author}{Hutter, F.},
  \bibinfo{author}{Hoos, H.H.}, \bibinfo{author}{Leyton-Brown, K.},
  \bibinfo{year}{2015}.
\newblock \bibinfo{title}{Efficient benchmarking of hyperparameter optimizers
  via surrogates}, in: \bibinfo{booktitle}{Proceedings of the Twenty-Ninth AAAI
  Conference on Artificial Intelligence}.
%Type = Inproceedings
\bibitem[{Fawcett et~al.(2014)Fawcett, Vallati, Hutter, Hoffmann, Hoos and
  Leyton-Brown}]{fawcett2014improved}
\bibinfo{author}{Fawcett, C.}, \bibinfo{author}{Vallati, M.},
  \bibinfo{author}{Hutter, F.}, \bibinfo{author}{Hoffmann, J.},
  \bibinfo{author}{Hoos, H.}, \bibinfo{author}{Leyton-Brown, K.},
  \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Improved features for runtime prediction of
  domain-independent planners}, in: \bibinfo{booktitle}{Proceedings of the
  International Conference on Automated Planning and Scheduling}.
%Type = Inproceedings
\bibitem[{Feurer et~al.(2015a)Feurer, Klein, Eggensperger, Springenberg, Blum
  and Hutter}]{feurer-nips15a}
\bibinfo{author}{Feurer, M.}, \bibinfo{author}{Klein, A.},
  \bibinfo{author}{Eggensperger, K.}, \bibinfo{author}{Springenberg, J.},
  \bibinfo{author}{Blum, M.}, \bibinfo{author}{Hutter, F.},
  \bibinfo{year}{2015}a.
\newblock \bibinfo{title}{Efficient and robust automated machine leraning}, in:
  \bibinfo{booktitle}{Advances in Neural Information Processing Systems 28},
  pp. \bibinfo{pages}{2944--2952}.
%Type = Inproceedings
\bibitem[{Feurer et~al.(2015b)Feurer, Springenberg and Hutter}]{feurer-aaai15a}
\bibinfo{author}{Feurer, M.}, \bibinfo{author}{Springenberg, J.T.},
  \bibinfo{author}{Hutter, F.}, \bibinfo{year}{2015}b.
\newblock \bibinfo{title}{Initializing {Bayesian} hyperparameter optimization
  via meta-learning}, in: \bibinfo{editor}{Bonet, B.}, \bibinfo{editor}{Koenig,
  S.} (Eds.), \bibinfo{booktitle}{Proceedings of the Twenty-Ninth {AAAI}
  Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas,
  {USA.}}, \bibinfo{publisher}{{AAAI} Press}. pp. \bibinfo{pages}{1128--1135}.
%Type = Inproceedings
\bibitem[{Gagliolo and
  Schmidhuber(2007)}]{gagliolo_learning_restart_strategies_2007}
\bibinfo{author}{Gagliolo, M.}, \bibinfo{author}{Schmidhuber, J.},
  \bibinfo{year}{2007}.
\newblock \bibinfo{title}{Learning restart strategies}, in:
  \bibinfo{booktitle}{Proceedings of the {Twentieth} International Joint
  Conference on Artificial Intelligence ({IJCAI})}, pp.
  \bibinfo{pages}{792--797}.
%Type = Inproceedings
\bibitem[{Gagliolo et~al.(2004)Gagliolo, Zhumatiy and
  Schmidhuber}]{gagliolo_adaptive_2004}
\bibinfo{author}{Gagliolo, M.}, \bibinfo{author}{Zhumatiy, V.},
  \bibinfo{author}{Schmidhuber, J.}, \bibinfo{year}{2004}.
\newblock \bibinfo{title}{Adaptive online time allocation to search
  algorithms}, in: \bibinfo{booktitle}{Proceedings of European Conference on
  Machine Learning}, \bibinfo{publisher}{Springer}. pp.
  \bibinfo{pages}{134--143}.
%Type = Book
\bibitem[{Gebser et~al.(2012a)Gebser, Kaminski, Kaufmann and
  Schaub}]{gekakasc12a}
\bibinfo{author}{Gebser, M.}, \bibinfo{author}{Kaminski, R.},
  \bibinfo{author}{Kaufmann, B.}, \bibinfo{author}{Schaub, T.},
  \bibinfo{year}{2012}a.
\newblock \bibinfo{title}{Answer Set Solving in Practice}.
\newblock Synthesis Lectures on Artificial Intelligence and Machine Learning,
  \bibinfo{publisher}{Morgan and Claypool Publishers}.
%Type = Inproceedings
\bibitem[{Gebser et~al.(2011)Gebser, Kaminski, Kaufmann, Schaub, Schneider and
  Ziller}]{gebser_portfolio_2011}
\bibinfo{author}{Gebser, M.}, \bibinfo{author}{Kaminski, R.},
  \bibinfo{author}{Kaufmann, B.}, \bibinfo{author}{Schaub, T.},
  \bibinfo{author}{Schneider, M.T.}, \bibinfo{author}{Ziller, S.},
  \bibinfo{year}{2011}.
\newblock \bibinfo{title}{A portfolio solver for answer set programming:
  preliminary report}, in: \bibinfo{booktitle}{Eleventh International
  Conference on Logic Programming and Nonmonotonic Reasoning},
  \bibinfo{publisher}{{Springer}}. pp. \bibinfo{pages}{352--357}.
%Type = Article
\bibitem[{Gebser et~al.(2012b)Gebser, Kaufmann and Schaub}]{gekasc12b}
\bibinfo{author}{Gebser, M.}, \bibinfo{author}{Kaufmann, B.},
  \bibinfo{author}{Schaub, T.}, \bibinfo{year}{2012}b.
\newblock \bibinfo{title}{Multi-threaded {ASP} solving with clasp}.
\newblock \bibinfo{journal}{Theory and Practice of Logic Programming}
  \bibinfo{volume}{12}, \bibinfo{pages}{525--545}.
%Type = Inproceedings
\bibitem[{Gent et~al.(2010a)Gent, Jefferson, Kotthoff, Miguel, Moore,
  Nightingale and Petrie}]{gent_learning_2010}
\bibinfo{author}{Gent, I.}, \bibinfo{author}{Jefferson, C.},
  \bibinfo{author}{Kotthoff, L.}, \bibinfo{author}{Miguel, I.},
  \bibinfo{author}{Moore, N.}, \bibinfo{author}{Nightingale, P.},
  \bibinfo{author}{Petrie, K.}, \bibinfo{year}{2010}a.
\newblock \bibinfo{title}{Learning when to use lazy learning in constraint
  solving}, in: \bibinfo{booktitle}{Proceedings of the Nineteenth European
  Conference on Artificial Intelligence}, \bibinfo{publisher}{{IOS} Press}. pp.
  \bibinfo{pages}{873--878}.
%Type = Inproceedings
\bibitem[{Gent et~al.(2006)Gent, Jefferson and Miguel}]{gent_minion_2006}
\bibinfo{author}{Gent, I.P.}, \bibinfo{author}{Jefferson, C.A.},
  \bibinfo{author}{Miguel, I.}, \bibinfo{year}{2006}.
\newblock \bibinfo{title}{{MINION}: A fast, scalable, constraint solver}, in:
  \bibinfo{booktitle}{Proceedings of the European Conference on Artificial
  Intelligence}, pp. \bibinfo{pages}{98--102}.
%Type = Inproceedings
\bibitem[{Gent et~al.(2010b)Gent, Miguel and Moore}]{gent_lazy_2010}
\bibinfo{author}{Gent, I.P.}, \bibinfo{author}{Miguel, I.},
  \bibinfo{author}{Moore, N.C.A.}, \bibinfo{year}{2010}b.
\newblock \bibinfo{title}{Lazy explanations for constraint propagators}, in:
  \bibinfo{booktitle}{Proceedings of the Twelfth International Symposium on
  Practical Aspects of Declarative Languages}, pp. \bibinfo{pages}{217--233}.
%Type = Article
\bibitem[{Gomes et~al.(2000)Gomes, Selman, Crato and Kautz}]{GomesSCK00}
\bibinfo{author}{Gomes, C.}, \bibinfo{author}{Selman, B.},
  \bibinfo{author}{Crato, N.}, \bibinfo{author}{Kautz, H.},
  \bibinfo{year}{2000}.
\newblock \bibinfo{title}{Heavy-tailed phenomena in satisfiability and
  constraint satisfaction problems}.
\newblock \bibinfo{journal}{Journal of Automated Reasoning}
  \bibinfo{volume}{24}, \bibinfo{pages}{67--100}.
%Type = Article
\bibitem[{Gomes and Selman(2001)}]{gomes_algorithm_2001}
\bibinfo{author}{Gomes, C.P.}, \bibinfo{author}{Selman, B.},
  \bibinfo{year}{2001}.
\newblock \bibinfo{title}{Algorithm portfolios}.
\newblock \bibinfo{journal}{Artificial Intelligence} \bibinfo{volume}{126},
  \bibinfo{pages}{43--62}.
%Type = Inproceedings
\bibitem[{Grasso et~al.(2010)Grasso, Iiritano, Leone, Lio, Ricca and
  Scalise}]{griilelirisc10a}
\bibinfo{author}{Grasso, G.}, \bibinfo{author}{Iiritano, S.},
  \bibinfo{author}{Leone, N.}, \bibinfo{author}{Lio, V.},
  \bibinfo{author}{Ricca, F.}, \bibinfo{author}{Scalise, F.},
  \bibinfo{year}{2010}.
\newblock \bibinfo{title}{An {ASP}-based system for team-building in the
  {G}ioia-{T}auro seaport}, in: \bibinfo{booktitle}{Proceedings of the Twelfth
  International Symposium on Practical Aspects of Declarative Languages}, pp.
  \bibinfo{pages}{40--42}.
%Type = Inproceedings
\bibitem[{Guerri and Milano(2004)}]{guerri_learning_2004}
\bibinfo{author}{Guerri, A.}, \bibinfo{author}{Milano, M.},
  \bibinfo{year}{2004}.
\newblock \bibinfo{title}{Learning techniques for automatic algorithm portfolio
  selection}, in: \bibinfo{booktitle}{Proceedings of the Sixteenth Eureopean
  Conference on Artificial Intelligence}, \bibinfo{publisher}{IOS Press}. pp.
  \bibinfo{pages}{475--479}.
%Type = Article
\bibitem[{Hall et~al.(2009)Hall, Frank, Holmes, Pfahringer, Reutemann and
  Witten}]{hall_weka_2009}
\bibinfo{author}{Hall, M.}, \bibinfo{author}{Frank, E.},
  \bibinfo{author}{Holmes, G.}, \bibinfo{author}{Pfahringer, B.},
  \bibinfo{author}{Reutemann, P.}, \bibinfo{author}{Witten, I.H.},
  \bibinfo{year}{2009}.
\newblock \bibinfo{title}{The {WEKA} data mining software: An update}.
\newblock \bibinfo{journal}{{SIGKDD} Explorations} \bibinfo{volume}{11},
  \bibinfo{pages}{10--18}.
%Type = Book
\bibitem[{Hastie et~al.(2009)Hastie, Tibshirani and Friedman}]{Hastie2009}
\bibinfo{author}{Hastie, T.J.}, \bibinfo{author}{Tibshirani, R.J.},
  \bibinfo{author}{Friedman, J.H.}, \bibinfo{year}{2009}.
\newblock \bibinfo{title}{The elements of statistical learning : data mining,
  inference, and prediction}.
\newblock Springer series in statistics, \bibinfo{publisher}{Springer},
  \bibinfo{address}{New York}.
%Type = Inproceedings
\bibitem[{Helmert et~al.(2011)Helmert, R\"oger and Karpas}]{helmert_fast_2011}
\bibinfo{author}{Helmert, M.}, \bibinfo{author}{R\"oger, G.},
  \bibinfo{author}{Karpas, E.}, \bibinfo{year}{2011}.
\newblock \bibinfo{title}{Fast downward stone soup: A baseline for building
  planner portfolios}, in: \bibinfo{booktitle}{Proceedings of the Workshop on
  Planning and Learning at the {Twenty-First} International Conference on
  Automated Planning and Scheduling}, pp. \bibinfo{pages}{28--35}.
%Type = Article
\bibitem[{Hoos et~al.(2014a)Hoos, Lindauer and Schaub}]{holisc14a}
\bibinfo{author}{Hoos, H.}, \bibinfo{author}{Lindauer, M.},
  \bibinfo{author}{Schaub, T.}, \bibinfo{year}{2014}a.
\newblock \bibinfo{title}{claspfolio 2: Advances in algorithm selection for
  answer set programming}.
\newblock \bibinfo{journal}{Theory and Practice of Logic Programming} ,
  \bibinfo{pages}{569--585}.
%Type = Article
\bibitem[{Hoos et~al.(2014b)Hoos, Kaminski, Lindauer and
  Schaub}]{hoos_aspeed_2014}
\bibinfo{author}{Hoos, H.H.}, \bibinfo{author}{Kaminski, R.},
  \bibinfo{author}{Lindauer, M.}, \bibinfo{author}{Schaub, T.},
  \bibinfo{year}{2014}b.
\newblock \bibinfo{title}{aspeed: Solver scheduling via answer set
  programming}.
\newblock \bibinfo{journal}{Theory and Practice of Logic Programming} ,
  \bibinfo{pages}{1--26}.
%Type = Inproceedings
\bibitem[{Howe et~al.(1999)Howe, Dahlman, Hansen, Scheetz and von
  Mayrhauser}]{howe_exploiting_1999}
\bibinfo{author}{Howe, A.E.}, \bibinfo{author}{Dahlman, E.},
  \bibinfo{author}{Hansen, C.}, \bibinfo{author}{Scheetz, M.},
  \bibinfo{author}{von Mayrhauser, A.}, \bibinfo{year}{1999}.
\newblock \bibinfo{title}{Exploiting competitive planner performance}, in:
  \bibinfo{booktitle}{Proceedings of the Fifth European Conference on
  Planning}, \bibinfo{publisher}{Springer}. pp. \bibinfo{pages}{62--72}.
%Type = Inproceedings
\bibitem[{Hurley et~al.(2014)Hurley, Kotthoff, Malitsky and
  {O'Sullivan}}]{hurley_proteus_2014}
\bibinfo{author}{Hurley, B.}, \bibinfo{author}{Kotthoff, L.},
  \bibinfo{author}{Malitsky, Y.}, \bibinfo{author}{{O'Sullivan}, B.},
  \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Proteus: A hierarchical portfolio of solvers and
  transformations}, in: \bibinfo{booktitle}{Proceedings of the Eleventh
  International Conference on Integration of {AI} and {OR} Techniques in
  Constraint Programming for Combinatorial Optimization Problems}, pp.
  \bibinfo{pages}{301--317}.
%Type = Inproceedings
\bibitem[{Hutter et~al.(2007)Hutter, Babi\'c, Hoos and Hu}]{HutBabHooHu07}
\bibinfo{author}{Hutter, F.}, \bibinfo{author}{Babi\'c, D.},
  \bibinfo{author}{Hoos, H.H.}, \bibinfo{author}{Hu, A.J.},
  \bibinfo{year}{2007}.
\newblock \bibinfo{title}{{Boosting Verification by Automatic Tuning of
  Decision Procedures}}, in: \bibinfo{booktitle}{Formal Methods in Computer
  Aided Design}, \bibinfo{publisher}{IEEE Computer Society}. pp.
  \bibinfo{pages}{27--34}.
%Type = Inproceedings
\bibitem[{Hutter et~al.(2006)Hutter, Hamadi, Hoos and
  {Leyton-Brown}}]{hutter_performance_2006}
\bibinfo{author}{Hutter, F.}, \bibinfo{author}{Hamadi, Y.},
  \bibinfo{author}{Hoos, H.H.}, \bibinfo{author}{{Leyton-Brown}, K.},
  \bibinfo{year}{2006}.
\newblock \bibinfo{title}{Performance prediction and automated tuning of
  randomized and parametric algorithms}, in: \bibinfo{booktitle}{Proceedings of
  the Twelfth International Conference on Principles and Practice of Constraint
  Programming}, pp. \bibinfo{pages}{213--228}.
%Type = Inproceedings
\bibitem[{Hutter et~al.(2010)Hutter, Hoos and
  Leyton-Brown}]{HutHooLey10-mipconfig}
\bibinfo{author}{Hutter, F.}, \bibinfo{author}{Hoos, H.H.},
  \bibinfo{author}{Leyton-Brown, K.}, \bibinfo{year}{2010}.
\newblock \bibinfo{title}{Automated configuration of mixed integer programming
  solvers}, in: \bibinfo{booktitle}{Integration of {AI} and {OR} Techniques in
  Constraint Programming for Combinatorial Optimization Problems}, pp.
  \bibinfo{pages}{186--202}.
%Type = Inproceedings
\bibitem[{Hutter et~al.(2011)Hutter, Hoos and Leyton-Brown}]{HutHooLey11-SMAC}
\bibinfo{author}{Hutter, F.}, \bibinfo{author}{Hoos, H.H.},
  \bibinfo{author}{Leyton-Brown, K.}, \bibinfo{year}{2011}.
\newblock \bibinfo{title}{Sequential model-based optimization for general
  algorithm configuration}, in: \bibinfo{booktitle}{Proceedings of the
  International Conference on Learning and Intelligent Optimization}, pp.
  \bibinfo{pages}{507--523}.
%Type = Inproceedings
\bibitem[{Hutter et~al.(2013)Hutter, Hoos and
  Leyton-Brown}]{hutter_identifying_2013}
\bibinfo{author}{Hutter, F.}, \bibinfo{author}{Hoos, H.H.},
  \bibinfo{author}{Leyton-Brown, K.}, \bibinfo{year}{2013}.
\newblock \bibinfo{title}{Identifying key algorithm parameters and instance
  features using forward selection}, in: \bibinfo{booktitle}{{LION} 7}.
%Type = Article
\bibitem[{Hutter et~al.(2009)Hutter, Hoos, {Leyton-Brown} and
  St\"utzle}]{hutter_paramils_2009}
\bibinfo{author}{Hutter, F.}, \bibinfo{author}{Hoos, H.H.},
  \bibinfo{author}{{Leyton-Brown}, K.}, \bibinfo{author}{St\"utzle, T.},
  \bibinfo{year}{2009}.
\newblock \bibinfo{title}{{ParamILS:} an automatic algorithm configuration
  framework}.
\newblock \bibinfo{journal}{Journal of Artificial Intelligence Research (JAIR)}
  \bibinfo{volume}{36}, \bibinfo{pages}{267--306}.
%Type = Inproceedings
\bibitem[{Hutter et~al.(2014a)Hutter, L\'{o}pez-Ib\'{a}\~{n}ez, Fawcett,
  Lindauer, Hoos, Leyton-Brown and St\"utzle}]{HutEtAl14:AClib}
\bibinfo{author}{Hutter, F.}, \bibinfo{author}{L\'{o}pez-Ib\'{a}\~{n}ez, M.},
  \bibinfo{author}{Fawcett, C.}, \bibinfo{author}{Lindauer, M.},
  \bibinfo{author}{Hoos, H.}, \bibinfo{author}{Leyton-Brown, K.},
  \bibinfo{author}{St\"utzle, T.}, \bibinfo{year}{2014}a.
\newblock \bibinfo{title}{Aclib: a benchmark library for algorithm
  configuration}, in: \bibinfo{booktitle}{Proceedings of the International
  Conference on Learning and Intelligent Optimization}, pp.
  \bibinfo{pages}{36--40}.
%Type = Article
\bibitem[{Hutter et~al.(2014b)Hutter, Xu, Hoos and
  Leyton-Brown}]{hutter2014algorithm}
\bibinfo{author}{Hutter, F.}, \bibinfo{author}{Xu, L.}, \bibinfo{author}{Hoos,
  H.H.}, \bibinfo{author}{Leyton-Brown, K.}, \bibinfo{year}{2014}b.
\newblock \bibinfo{title}{Algorithm runtime prediction: Methods \& evaluation}.
\newblock \bibinfo{journal}{Artificial Intelligence} \bibinfo{volume}{206},
  \bibinfo{pages}{79--111}.
%Type = Article
\bibitem[{Ishebabi et~al.(2009)Ishebabi, Mahr, Bobda, Gebser and
  Schaub}]{ismabogesc09a}
\bibinfo{author}{Ishebabi, H.}, \bibinfo{author}{Mahr, P.},
  \bibinfo{author}{Bobda, C.}, \bibinfo{author}{Gebser, M.},
  \bibinfo{author}{Schaub, T.}, \bibinfo{year}{2009}.
\newblock \bibinfo{title}{Answer set vs.\ integer linear programming for
  automatic synthesis of multiprocessor systems from real-time parallel
  programs}.
\newblock \bibinfo{journal}{Journal of Reconfigurable Computing} .
%Type = Incollection
\bibitem[{Kadioglu et~al.(2011)Kadioglu, Malitsky, Sabharwal, Samulowitz and
  Sellmann}]{kadioglu_algorithm_2011}
\bibinfo{author}{Kadioglu, S.}, \bibinfo{author}{Malitsky, Y.},
  \bibinfo{author}{Sabharwal, A.}, \bibinfo{author}{Samulowitz, H.},
  \bibinfo{author}{Sellmann, M.}, \bibinfo{year}{2011}.
\newblock \bibinfo{title}{Algorithm selection and scheduling}, in:
  \bibinfo{booktitle}{Proceedings of the International Conference on Principles
  and Practice of Constraint Programming}. \bibinfo{publisher}{Springer}.
  volume \bibinfo{volume}{6876} of \textit{\bibinfo{series}{Lecture Notes in
  Computer Science}}, pp. \bibinfo{pages}{454--469}.
%Type = Inproceedings
\bibitem[{Kadioglu et~al.(2010)Kadioglu, Malitsky, Sellmann and
  Tierney}]{kadioglu_isac_2010}
\bibinfo{author}{Kadioglu, S.}, \bibinfo{author}{Malitsky, Y.},
  \bibinfo{author}{Sellmann, M.}, \bibinfo{author}{Tierney, K.},
  \bibinfo{year}{2010}.
\newblock \bibinfo{title}{{ISAC} – {Instance-Specific Algorithm
  Configuration}}, in: \bibinfo{booktitle}{Proceedings of Nineteenth European
  Conference on Artificial Intelligence}, \bibinfo{publisher}{{IOS} Press}. pp.
  \bibinfo{pages}{751--756}.
%Type = Article
\bibitem[{Karatzoglou et~al.(2004)Karatzoglou, Smola, Hornik and
  Zeileis}]{ksvm}
\bibinfo{author}{Karatzoglou, A.}, \bibinfo{author}{Smola, A.},
  \bibinfo{author}{Hornik, K.}, \bibinfo{author}{Zeileis, A.},
  \bibinfo{year}{2004}.
\newblock \bibinfo{title}{kernlab -- an {S4} package for kernel methods in
  {R}}.
\newblock \bibinfo{journal}{Journal of Statistical Software}
  \bibinfo{volume}{11}, \bibinfo{pages}{1--20}.
%Type = Inproceedings
\bibitem[{Kautz and Selman(1999)}]{kautz1999unifying}
\bibinfo{author}{Kautz, H.}, \bibinfo{author}{Selman, B.},
  \bibinfo{year}{1999}.
\newblock \bibinfo{title}{Unifying {SAT}-based and graph-based planning}, in:
  \bibinfo{booktitle}{Proceedings of the Sixteenth International Joint
  Conference on Artifical Intelligence}, \bibinfo{publisher}{Morgan Kaufmann}.
  pp. \bibinfo{pages}{318--325}.
%Type = Inproceedings
\bibitem[{Kerschke et~al.(2014)Kerschke, Preuss, Hern{\'a}ndez, Sch{\"u}tze,
  Sun, Grimme, Rudolph, Bischl and Trautmann}]{Kerschke2014}
\bibinfo{author}{Kerschke, P.}, \bibinfo{author}{Preuss, M.},
  \bibinfo{author}{Hern{\'a}ndez, C.}, \bibinfo{author}{Sch{\"u}tze, O.},
  \bibinfo{author}{Sun, J.Q.}, \bibinfo{author}{Grimme, C.},
  \bibinfo{author}{Rudolph, G.}, \bibinfo{author}{Bischl, B.},
  \bibinfo{author}{Trautmann, H.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Cell mapping techniques for exploratory landscape
  analysis}, in: \bibinfo{booktitle}{Proceedings of the {EVOLVE 2014}: {A}
  Bridge between Probability, Set Oriented Numerics, and Evolutionary
  Computation}, \bibinfo{publisher}{Springer}. pp. \bibinfo{pages}{115--131}.
%Type = Article
\bibitem[{Kohavi and John(1997)}]{Kohavi97}
\bibinfo{author}{Kohavi, R.}, \bibinfo{author}{John, G.H.},
  \bibinfo{year}{1997}.
\newblock \bibinfo{title}{Wrappers for feature subset selection}.
\newblock \bibinfo{journal}{Artificial Intelligence} \bibinfo{volume}{97},
  \bibinfo{pages}{273--324}.
%Type = Techreport
\bibitem[{Kotthoff(2013)}]{kotthoff_llama_2013}
\bibinfo{author}{Kotthoff, L.}, \bibinfo{year}{2013}.
\newblock \bibinfo{title}{{LLAMA:} Leveraging Learning to Automatically Manage
  Algorithms}.
\newblock \bibinfo{type}{Technical Report} \bibinfo{number}{{arXiv:1306.1031}}.
  \bibinfo{address}{{arXiv}}.
\newblock \bibinfo{note}{\url{http://arxiv.org/abs/1306.1031}}.
%Type = Article
\bibitem[{Kotthoff(2014)}]{kotthoff_algorithm_2014}
\bibinfo{author}{Kotthoff, L.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Algorithm selection for combinatorial search
  problems: A survey}.
\newblock \bibinfo{journal}{{AI} Magazine} \bibinfo{volume}{35},
  \bibinfo{pages}{48--60}.
%Type = Inproceedings
\bibitem[{Lagoudakis and Littman(2000)}]{LAG1}
\bibinfo{author}{Lagoudakis, M.}, \bibinfo{author}{Littman, M.},
  \bibinfo{year}{2000}.
\newblock \bibinfo{title}{Algorithm selection using reinforcement learning},
  in: \bibinfo{booktitle}{Proceedings of the Seventeenth International
  Conference on Machine Learning}, pp. \bibinfo{pages}{511--518}.
%Type = Inproceedings
\bibitem[{Lagoudakis and Littman(2001)}]{LAG2}
\bibinfo{author}{Lagoudakis, M.}, \bibinfo{author}{Littman, M.},
  \bibinfo{year}{2001}.
\newblock \bibinfo{title}{Learning to select branching rules in the {DPLL}
  procedure for satisfiability}, in: \bibinfo{booktitle}{Proceedings of the
  International Conference on Satisfiability}, pp. \bibinfo{pages}{344--359}.
%Type = Inproceedings
\bibitem[{Le~Berre and Lynce(2008)}]{daniel_simple_csp_to_sat}
\bibinfo{author}{Le~Berre, D.}, \bibinfo{author}{Lynce, I.},
  \bibinfo{year}{2008}.
\newblock \bibinfo{title}{{CSP2SAT4J: A simple {CSP} to {SAT} translator}}, in:
  \bibinfo{booktitle}{Proceedings of the Second International {CSP} Solver
  Competition}, pp. \bibinfo{pages}{43--54}.
%Type = Inproceedings
\bibitem[{Leite et~al.(2012)Leite, Brazdil and Vanschoren}]{LeiteBV12}
\bibinfo{author}{Leite, R.}, \bibinfo{author}{Brazdil, P.},
  \bibinfo{author}{Vanschoren, J.}, \bibinfo{year}{2012}.
\newblock \bibinfo{title}{Selecting classification algorithms with active
  testing}, in: \bibinfo{booktitle}{Machine Learning and Data Mining in Pattern
  Recognition}, \bibinfo{publisher}{Springer}. pp. \bibinfo{pages}{117--131}.
%Type = Inproceedings
\bibitem[{Leyton-Brown et~al.(2003)Leyton-Brown, Nudelman, Andrew, McFadden and
  Shoham}]{leyton2003portfolio}
\bibinfo{author}{Leyton-Brown, K.}, \bibinfo{author}{Nudelman, E.},
  \bibinfo{author}{Andrew, G.}, \bibinfo{author}{McFadden, J.},
  \bibinfo{author}{Shoham, Y.}, \bibinfo{year}{2003}.
\newblock \bibinfo{title}{A portfolio approach to algorithm selection}, in:
  \bibinfo{booktitle}{Proceedings of the Eighteenth International Joint
  Conference on Artificial Intelligence}, \bibinfo{publisher}{Morgan Kaufmann}.
  pp. \bibinfo{pages}{1542--1543}.
%Type = Article
\bibitem[{Liaw and Wiener(2002)}]{rf}
\bibinfo{author}{Liaw, A.}, \bibinfo{author}{Wiener, M.}, \bibinfo{year}{2002}.
\newblock \bibinfo{title}{Classification and regression by {randomForest}}.
\newblock \bibinfo{journal}{R News} \bibinfo{volume}{2},
  \bibinfo{pages}{18--22}.
%Type = Article
\bibitem[{Lindauer et~al.(2015)Lindauer, Hoos, Hutter and
  Schaub}]{lindauer-jair15a}
\bibinfo{author}{Lindauer, M.}, \bibinfo{author}{Hoos, H.},
  \bibinfo{author}{Hutter, F.}, \bibinfo{author}{Schaub, T.},
  \bibinfo{year}{2015}.
\newblock \bibinfo{title}{Autofolio: An automatically configured algorithm
  selector}.
\newblock \bibinfo{journal}{Journal of Artificial Intelligence}
  \bibinfo{volume}{53}, \bibinfo{pages}{745--778}.
%Type = Inproceedings
\bibitem[{Malitsky et~al.(2013)Malitsky, Mehta and
  {O'Sullivan}}]{malitsky_evolving_2013}
\bibinfo{author}{Malitsky, Y.}, \bibinfo{author}{Mehta, D.},
  \bibinfo{author}{{O'Sullivan}, B.}, \bibinfo{year}{2013}.
\newblock \bibinfo{title}{Evolving instance specific algorithm configuration},
  in: \bibinfo{booktitle}{The Sixth Annual Symposium on Combinatorial Search}.
%Type = Inproceedings
\bibitem[{Malitsky et~al.(2014)Malitsky, O'Sullivan, Previti and
  Marques-Silva}]{MUSportfolio}
\bibinfo{author}{Malitsky, Y.}, \bibinfo{author}{O'Sullivan, B.},
  \bibinfo{author}{Previti, A.}, \bibinfo{author}{Marques-Silva, J.},
  \bibinfo{year}{2014}.
\newblock \bibinfo{title}{A portfolio approach to enumerating minimal
  correction subsets for satisfiability problems}, in:
  \bibinfo{booktitle}{Proceedings of the Eleventh International Conference on
  Integration of Artificical Intelligence and Operations Research Techniques in
  Constraint Programming}.
%Type = Inproceedings
\bibitem[{Malitsky et~al.(2011)Malitsky, Sabharwal, Samulowitz and
  Sellmann}]{malitsky_non-model-based_2011}
\bibinfo{author}{Malitsky, Y.}, \bibinfo{author}{Sabharwal, A.},
  \bibinfo{author}{Samulowitz, H.}, \bibinfo{author}{Sellmann, M.},
  \bibinfo{year}{2011}.
\newblock \bibinfo{title}{Non-model-based algorithm portfolios for {SAT}}, in:
  \bibinfo{booktitle}{Proceedings of the Fourteenth International Conference on
  Theory and Applications of Satisfiability Testing},
  \bibinfo{publisher}{Springer}. pp. \bibinfo{pages}{369--370}.
%Type = Article
\bibitem[{Mersmann et~al.(2013)Mersmann, Bischl, Trautmann, Wagner, Bossek and
  Neumann}]{Mersmann2013}
\bibinfo{author}{Mersmann, O.}, \bibinfo{author}{Bischl, B.},
  \bibinfo{author}{Trautmann, H.}, \bibinfo{author}{Wagner, M.},
  \bibinfo{author}{Bossek, J.}, \bibinfo{author}{Neumann, F.},
  \bibinfo{year}{2013}.
\newblock \bibinfo{title}{A novel feature-based approach to characterize
  algorithm performance for the traveling salesperson problem}.
\newblock \bibinfo{journal}{Annals of Mathematics and Artificial Intelligence}
  , \bibinfo{pages}{1--32}.
%Type = Inproceedings
\bibitem[{Nikoli\'c et~al.(2009)Nikoli\'c, Mari\'c and
  Jani\v{c}i\'c}]{nikoli_instance-based_2009}
\bibinfo{author}{Nikoli\'c, M.}, \bibinfo{author}{Mari\'c, F.},
  \bibinfo{author}{Jani\v{c}i\'c, P.}, \bibinfo{year}{2009}.
\newblock \bibinfo{title}{Instance-based selection of policies for {SAT}
  solvers}, in: \bibinfo{booktitle}{Proceedings of the Twelfth International
  Conference on Theory and Applications of Satisfiability Testing},
  \bibinfo{publisher}{{Springer}}. pp. \bibinfo{pages}{326--340}.
%Type = Inproceedings
\bibitem[{Nogueira et~al.(2001)Nogueira, Balduccini, Gelfond, Watson and
  Barry}]{nobagewaba01a}
\bibinfo{author}{Nogueira, M.}, \bibinfo{author}{Balduccini, M.},
  \bibinfo{author}{Gelfond, M.}, \bibinfo{author}{Watson, R.},
  \bibinfo{author}{Barry, M.}, \bibinfo{year}{2001}.
\newblock \bibinfo{title}{An {A}-prolog decision support system for the space
  shuttle}, in: \bibinfo{booktitle}{Proceedings of the Third International
  Symposium on Practical Aspects of Declarative Languages},
  \bibinfo{publisher}{Springer}. pp. \bibinfo{pages}{169--183}.
%Type = Unpublished
\bibitem[{Nudelman et~al.(2003)Nudelman, Leyton-Brown, Andrew, Gomes, McFadden,
  Selman and Shoham}]{Satzilla03}
\bibinfo{author}{Nudelman, E.}, \bibinfo{author}{Leyton-Brown, K.},
  \bibinfo{author}{Andrew, G.}, \bibinfo{author}{Gomes, C.},
  \bibinfo{author}{McFadden, J.}, \bibinfo{author}{Selman, B.},
  \bibinfo{author}{Shoham, Y.}, \bibinfo{year}{2003}.
\newblock \bibinfo{title}{{Satzilla} 0.9}.
%Type = Inproceedings
\bibitem[{Nudelman et~al.(2004)Nudelman, {Leyton-Brown}, Hoos, Devkar and
  Shoham}]{nudelman_understanding_2004}
\bibinfo{author}{Nudelman, E.}, \bibinfo{author}{{Leyton-Brown}, K.},
  \bibinfo{author}{Hoos, H.H.}, \bibinfo{author}{Devkar, A.},
  \bibinfo{author}{Shoham, Y.}, \bibinfo{year}{2004}.
\newblock \bibinfo{title}{Understanding random {SAT:} beyond the
  {Clauses-to-Variables} ratio}, in: \bibinfo{booktitle}{Principles and
  Practice of Constraint Programming – {CP} 2004},
  \bibinfo{publisher}{Springer}. pp. \bibinfo{pages}{438--452}.
%Type = Inproceedings
\bibitem[{O'Mahony et~al.(2008)O'Mahony, Hebrard, Holland, Nugent and
  O'Sullivan}]{cphydra}
\bibinfo{author}{O'Mahony, E.}, \bibinfo{author}{Hebrard, E.},
  \bibinfo{author}{Holland, A.}, \bibinfo{author}{Nugent, C.},
  \bibinfo{author}{O'Sullivan, B.}, \bibinfo{year}{2008}.
\newblock \bibinfo{title}{Using case-based reasoning in an algorithm portfolio
  for constraint solving}, in: \bibinfo{booktitle}{Proceedings of the
  Nineteenth Irish Conference on Artificial Intelligence and Cognitive
  Science}.
%Type = Article
\bibitem[{Pfahringer et~al.(2000)Pfahringer, Bensusan and
  Giraud-Carrier}]{Pfahringer:2000p553}
\bibinfo{author}{Pfahringer, B.}, \bibinfo{author}{Bensusan, H.},
  \bibinfo{author}{Giraud-Carrier, C.}, \bibinfo{year}{2000}.
\newblock \bibinfo{title}{Meta-learning by landmarking various learning
  algorithms}.
\newblock \bibinfo{journal}{Proceedings of the Seventeenth International
  Conference on Machine Learning} , \bibinfo{pages}{743--750}.
%Type = Article
\bibitem[{Prasad et~al.(2005)Prasad, Biere and Gupta}]{prasad2005survey}
\bibinfo{author}{Prasad, M.R.}, \bibinfo{author}{Biere, A.},
  \bibinfo{author}{Gupta, A.}, \bibinfo{year}{2005}.
\newblock \bibinfo{title}{A survey of recent advances in {SAT}-based formal
  verification}.
\newblock \bibinfo{journal}{International Journal on Software Tools for
  Technology Transfer} \bibinfo{volume}{7}, \bibinfo{pages}{156--173}.
%Type = Inproceedings
\bibitem[{Pulina and Tacchella(2007)}]{pulina_multi-engine_2007}
\bibinfo{author}{Pulina, L.}, \bibinfo{author}{Tacchella, A.},
  \bibinfo{year}{2007}.
\newblock \bibinfo{title}{A multi-engine solver for quantified boolean
  formulas}, in: \bibinfo{booktitle}{Proceedings of the Thirteenth
  International Conference on Principles and Practice of Constraint
  Programming}, \bibinfo{publisher}{{Springer}}. pp. \bibinfo{pages}{574--589}.
%Type = Article
\bibitem[{Pulina and Tacchella(2009)}]{pulina_self-adaptive_2009}
\bibinfo{author}{Pulina, L.}, \bibinfo{author}{Tacchella, A.},
  \bibinfo{year}{2009}.
\newblock \bibinfo{title}{A self-adaptive multi-engine solver for quantified
  boolean formulas}.
\newblock \bibinfo{journal}{Constraints} \bibinfo{volume}{14},
  \bibinfo{pages}{80--116}.
%Type = Manual
\bibitem[{{R Core Team}(2014)}]{R}
\bibinfo{author}{{R Core Team}}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{R: A Language and Environment for Statistical
  Computing}.
\newblock \bibinfo{organization}{R Foundation for Statistical Computing}.
  \bibinfo{address}{Vienna, Austria}.
\newblock \URLprefix \url{http://www.R-project.org/}.
%Type = Article
\bibitem[{Rice(1976)}]{rice_algorithm_1976}
\bibinfo{author}{Rice, J.R.}, \bibinfo{year}{1976}.
\newblock \bibinfo{title}{The algorithm selection problem}.
\newblock \bibinfo{journal}{Advances in Computers} \bibinfo{volume}{15},
  \bibinfo{pages}{65--118}.
%Type = Article
\bibitem[{Roberts and Howe(2009)}]{RobertsH09}
\bibinfo{author}{Roberts, M.}, \bibinfo{author}{Howe, A.},
  \bibinfo{year}{2009}.
\newblock \bibinfo{title}{Learning from planner performance}.
\newblock \bibinfo{journal}{Artificial Intelligence Journal}
  \bibinfo{volume}{173}, \bibinfo{pages}{536--561}.
%Type = Inproceedings
\bibitem[{Roberts and Howe(2007)}]{roberts_learned_2007}
\bibinfo{author}{Roberts, M.}, \bibinfo{author}{Howe, A.E.},
  \bibinfo{year}{2007}.
\newblock \bibinfo{title}{Learned models of performance for many planners}, in:
  \bibinfo{booktitle}{Proceedings of the Workshop on {AI} Planning and Learning
  at the Seventeenth International Conference on Automated Planning and
  Scheduling.}
%Type = Inproceedings
\bibitem[{Roberts et~al.(2008)Roberts, Howe, Wilson and
  {desJardins}}]{roberts_what_2008}
\bibinfo{author}{Roberts, M.}, \bibinfo{author}{Howe, A.E.},
  \bibinfo{author}{Wilson, B.}, \bibinfo{author}{{desJardins}, M.},
  \bibinfo{year}{2008}.
\newblock \bibinfo{title}{What makes planners predictable?}, in:
  \bibinfo{booktitle}{{ICAPS}}, pp. \bibinfo{pages}{288--295}.
%Type = Inproceedings
\bibitem[{Sabharwal et~al.(2013)Sabharwal, Samulowitz, Sellmann and
  Malitsky}]{sabharwal_boosting_2013}
\bibinfo{author}{Sabharwal, A.}, \bibinfo{author}{Samulowitz, H.},
  \bibinfo{author}{Sellmann, M.}, \bibinfo{author}{Malitsky, Y.},
  \bibinfo{year}{2013}.
\newblock \bibinfo{title}{Boosting sequential solver portfolios: Knowledge
  sharing and accuracy prediction}, in: \bibinfo{booktitle}{{LION} 7}.
%Type = Inproceedings
\bibitem[{Samulowitz and Memisevic(2007)}]{samulowitz_learning_2007}
\bibinfo{author}{Samulowitz, H.}, \bibinfo{author}{Memisevic, R.},
  \bibinfo{year}{2007}.
\newblock \bibinfo{title}{Learning to solve {QBF}}, in:
  \bibinfo{booktitle}{Proceedings of the {Twenty-Second} National Conference on
  Artificial Intelligence}, \bibinfo{publisher}{{AAAI} Press}. pp.
  \bibinfo{pages}{255--260}.
%Type = Article
\bibitem[{Serban et~al.(2013)Serban, Vanschoren, Kietz and
  Bernstein}]{Serban:2013}
\bibinfo{author}{Serban, F.}, \bibinfo{author}{Vanschoren, J.},
  \bibinfo{author}{Kietz, J.U.}, \bibinfo{author}{Bernstein, A.},
  \bibinfo{year}{2013}.
\newblock \bibinfo{title}{A survey of intelligent assistants for data
  analysis}.
\newblock \bibinfo{journal}{ACM Comput. Surv.} \bibinfo{volume}{45},
  \bibinfo{pages}{1--35}.
%Type = Inproceedings
\bibitem[{Silverthorn and Miikkulainen(2010)}]{silverthorn_latent_2010}
\bibinfo{author}{Silverthorn, B.}, \bibinfo{author}{Miikkulainen, R.},
  \bibinfo{year}{2010}.
\newblock \bibinfo{title}{Latent class models for algorithm portfolio
  methods.}, in: \bibinfo{booktitle}{Proceedings of the {Twenty-Fourth} AAAI
  Conference on Artificial Intelligence}, pp. \bibinfo{pages}{167--172}.
%Type = Article
\bibitem[{{Smith-Miles}(2008)}]{smith-miles_cross-disciplinary_2009}
\bibinfo{author}{{Smith-Miles}, K.A.}, \bibinfo{year}{2008}.
\newblock \bibinfo{title}{Cross-disciplinary perspectives on meta-learning for
  algorithm selection}.
\newblock \bibinfo{journal}{{ACM} Computing Surveys} \bibinfo{volume}{41},
  \bibinfo{pages}{6:1--6:25}.
%Type = Article
\bibitem[{Smith-Miles et~al.(2014)Smith-Miles, Baatar, Wreford and
  Lewis}]{SmithMiles201412}
\bibinfo{author}{Smith-Miles, K.A.}, \bibinfo{author}{Baatar, D.},
  \bibinfo{author}{Wreford, B.J.}, \bibinfo{author}{Lewis, R.},
  \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Towards objective measures of algorithm performance
  across instance space}.
\newblock \bibinfo{journal}{Computers \& Operations Research}
  \bibinfo{volume}{45}, \bibinfo{pages}{12--24}.
%Type = Inproceedings
\bibitem[{Soininen and Niemel{\"a}(1999)}]{soinie99a}
\bibinfo{author}{Soininen, T.}, \bibinfo{author}{Niemel{\"a}, I.},
  \bibinfo{year}{1999}.
\newblock \bibinfo{title}{Developing a declarative rule language for
  applications in product configuration}, in: \bibinfo{booktitle}{Proceedings
  of the First International Workshop on Practical Aspects of Declarative
  Languages}, \bibinfo{publisher}{Springer}. pp. \bibinfo{pages}{305--319}.
%Type = Article
\bibitem[{Stahlbock and Vo{\ss}(2008)}]{StVo08}
\bibinfo{author}{Stahlbock, R.}, \bibinfo{author}{Vo{\ss}, S.},
  \bibinfo{year}{2008}.
\newblock \bibinfo{title}{Operations research at container terminals: a
  literature update}.
\newblock \bibinfo{journal}{OR Spectrum} \bibinfo{volume}{30},
  \bibinfo{pages}{1--52}.
%Type = Article
\bibitem[{Stergiou(2009)}]{stergiou_heuristics_2009}
\bibinfo{author}{Stergiou, K.}, \bibinfo{year}{2009}.
\newblock \bibinfo{title}{Heuristics for dynamically adapting propagation in
  constraint satisfaction problems}.
\newblock \bibinfo{journal}{{AI} Communications} \bibinfo{volume}{22},
  \bibinfo{pages}{125--141}.
%Type = Inproceedings
\bibitem[{Streeter et~al.(2007a)Streeter, Golovin and
  Smith}]{streeter_combining_2007}
\bibinfo{author}{Streeter, M.J.}, \bibinfo{author}{Golovin, D.},
  \bibinfo{author}{Smith, S.F.}, \bibinfo{year}{2007}a.
\newblock \bibinfo{title}{Combining multiple heuristics online}, in:
  \bibinfo{booktitle}{Proceedings of the {Twenty-Second} National Conference on
  Artificial Intelligence}, \bibinfo{publisher}{{AAAI} Press}. pp.
  \bibinfo{pages}{1197--1203}.
%Type = Inproceedings
\bibitem[{Streeter et~al.(2007b)Streeter, Golovin and
  Smith}]{streeter_restart_2007}
\bibinfo{author}{Streeter, M.J.}, \bibinfo{author}{Golovin, D.},
  \bibinfo{author}{Smith, S.F.}, \bibinfo{year}{2007}b.
\newblock \bibinfo{title}{Restart schedules for ensembles of problem
  instances}, in: \bibinfo{booktitle}{Proceedings of the {Twenty-Second}
  National Conference on Artificial Intelligence}, \bibinfo{publisher}{{AAAI}
  Press}. pp. \bibinfo{pages}{1204--1210}.
%Type = Article
\bibitem[{Stuckey et~al.(2014)Stuckey, Feydy, Schutt, Tack and
  Fischer}]{StuckeyFSTF14}
\bibinfo{author}{Stuckey, P.J.}, \bibinfo{author}{Feydy, T.},
  \bibinfo{author}{Schutt, A.}, \bibinfo{author}{Tack, G.},
  \bibinfo{author}{Fischer, J.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{The {MiniZinc} challenge 2008-2013}.
\newblock \bibinfo{journal}{AI Magazine} \bibinfo{volume}{35},
  \bibinfo{pages}{55--60}.
%Type = Inproceedings
\bibitem[{Tamura et~al.(2008)Tamura, Tanjo and Banbara}]{tamura_sugar}
\bibinfo{author}{Tamura, N.}, \bibinfo{author}{Tanjo, T.},
  \bibinfo{author}{Banbara, M.}, \bibinfo{year}{2008}.
\newblock \bibinfo{title}{System description of a {SAT}-based {CSP} solver
  sugar}, in: \bibinfo{booktitle}{Proceedings of the Third International {CSP}
  Solver Competition}, pp. \bibinfo{pages}{71--75}.
%Type = Incollection
\bibitem[{Tanjo et~al.(2012)Tanjo, Tamura and Banbara}]{tanjo_azucar}
\bibinfo{author}{Tanjo, T.}, \bibinfo{author}{Tamura, N.},
  \bibinfo{author}{Banbara, M.}, \bibinfo{year}{2012}.
\newblock \bibinfo{title}{Azucar: a {SAT}-based {CSP} solver using compact
  order encoding}, in: \bibinfo{booktitle}{Theory and Applications of
  Satisfiability Testing -- SAT 2012}. \bibinfo{publisher}{Springer}, pp.
  \bibinfo{pages}{456--462}.
%Type = Manual
\bibitem[{Therneau et~al.(2014)Therneau, Atkinson and Ripley}]{rpart}
\bibinfo{author}{Therneau, T.}, \bibinfo{author}{Atkinson, B.},
  \bibinfo{author}{Ripley, B.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{rpart: Recursive Partitioning and Regression Trees}.
\newblock \URLprefix \url{http://CRAN.R-project.org/package=rpart}.
  \bibinfo{note}{r package version 4.1-8}.
%Type = Inproceedings
\bibitem[{Thornton et~al.(2013)Thornton, Hutter, Hoos and
  Leyton-Brown}]{ThoHutHooLey13-AutoWEKA}
\bibinfo{author}{Thornton, C.}, \bibinfo{author}{Hutter, F.},
  \bibinfo{author}{Hoos, H.}, \bibinfo{author}{Leyton-Brown, K.},
  \bibinfo{year}{2013}.
\newblock \bibinfo{title}{Auto-{WEKA}: Combined selection and hyperparameter
  optimization of classification algorithms}, in:
  \bibinfo{booktitle}{Proceedings of the Nineteenth ACM SIGKDD International
  Conference on Knowledge Discovery and Data Mining}, \bibinfo{publisher}{ACM}.
  pp. \bibinfo{pages}{847--855}.
%Type = Inproceedings
\bibitem[{Tierney and Malitsky(2015)}]{Ti14tr}
\bibinfo{author}{Tierney, K.}, \bibinfo{author}{Malitsky, Y.},
  \bibinfo{year}{2015}.
\newblock \bibinfo{title}{An algorithm selection benchmark of the container
  pre-marshalling problem}, in: \bibinfo{editor}{Dhaenens, C.},
  \bibinfo{editor}{Jourdan, L.}, \bibinfo{editor}{Marmion, M.E.} (Eds.),
  \bibinfo{booktitle}{Learning and Intelligent Optimization},
  \bibinfo{publisher}{Springer International Publishing}. pp.
  \bibinfo{pages}{17--22}.
%Type = Techreport
\bibitem[{Tierney et~al.(2014)Tierney, Pacino and Vo{\ss}}]{TiPaVo14tr}
\bibinfo{author}{Tierney, K.}, \bibinfo{author}{Pacino, D.},
  \bibinfo{author}{Vo{\ss}, S.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Solving the Pre-Marshalling Problem to Optimality
  with {A*} and {IDA*}}.
\newblock \bibinfo{type}{Technical Report} \bibinfo{number}{Working Paper
  \#1401}. Decision Support \& Optimization Lab, University of Paderborn.
%Type = Article
\bibitem[{Vallati et~al.(2015a)Vallati, Chrpa, Grzes, McCluskey, Roberts and
  Sanner}]{VallatiCGMRS15}
\bibinfo{author}{Vallati, M.}, \bibinfo{author}{Chrpa, L.},
  \bibinfo{author}{Grzes, M.}, \bibinfo{author}{McCluskey, T.L.},
  \bibinfo{author}{Roberts, M.}, \bibinfo{author}{Sanner, S.},
  \bibinfo{year}{2015}a.
\newblock \bibinfo{title}{The 2014 international planning competition: Progress
  and trends}.
\newblock \bibinfo{journal}{{AI} Magazine} \bibinfo{volume}{36},
  \bibinfo{pages}{90--98}.
%Type = Article
\bibitem[{Vallati et~al.(2015b)Vallati, Chrpa and Kitchin}]{VallatiCK15}
\bibinfo{author}{Vallati, M.}, \bibinfo{author}{Chrpa, L.},
  \bibinfo{author}{Kitchin, D.}, \bibinfo{year}{2015}b.
\newblock \bibinfo{title}{Portfolio-based planning: State of the art, common
  practice and open challenges}.
\newblock \bibinfo{journal}{{AI} Communincations} \bibinfo{volume}{28},
  \bibinfo{pages}{717--733}.
%Type = Inproceedings
\bibitem[{Vallati et~al.(2013)Vallati, Fawcett, Gerevini, Hoos and
  Saetti}]{Vallati13-SOCS}
\bibinfo{author}{Vallati, M.}, \bibinfo{author}{Fawcett, C.},
  \bibinfo{author}{Gerevini, A.}, \bibinfo{author}{Hoos, H.H.},
  \bibinfo{author}{Saetti, A.}, \bibinfo{year}{2013}.
\newblock \bibinfo{title}{Automatic generation of efficient domain-optimized
  planners from generic parametrized planners}, in:
  \bibinfo{booktitle}{International Symposium on Combinatorial Search (SoCS)}.
%Type = Article
\bibitem[{Van~Gelder(2008)}]{van2008another}
\bibinfo{author}{Van~Gelder, A.}, \bibinfo{year}{2008}.
\newblock \bibinfo{title}{Another look at graph coloring via propositional
  satisfiability}.
\newblock \bibinfo{journal}{Discrete Applied Mathematics}
  \bibinfo{volume}{156}, \bibinfo{pages}{230--243}.
%Type = Phdthesis
\bibitem[{Vanschoren(2010)}]{Vanschoren2010}
\bibinfo{author}{Vanschoren, J.}, \bibinfo{year}{2010}.
\newblock \bibinfo{title}{Understanding Machine Learning Performance with
  Experiment Databases}.
\newblock Ph.D. thesis. University of Leuven.
%Type = Article
\bibitem[{Vanschoren et~al.(2012)Vanschoren, Blockeel, Pfahringer and
  Holmes}]{Vanschoren2012}
\bibinfo{author}{Vanschoren, J.}, \bibinfo{author}{Blockeel, H.},
  \bibinfo{author}{Pfahringer, B.}, \bibinfo{author}{Holmes, G.},
  \bibinfo{year}{2012}.
\newblock \bibinfo{title}{Experiment databases. {A} new way to share, organize
  and learn from experiments}.
\newblock \bibinfo{journal}{Machine Learning} \bibinfo{volume}{87},
  \bibinfo{pages}{127--158}.
%Type = Article
\bibitem[{Vanschoren et~al.(2013)Vanschoren, van Rijn, Bischl and
  Torgo}]{openml2013}
\bibinfo{author}{Vanschoren, J.}, \bibinfo{author}{van Rijn, J.N.},
  \bibinfo{author}{Bischl, B.}, \bibinfo{author}{Torgo, L.},
  \bibinfo{year}{2013}.
\newblock \bibinfo{title}{{OpenML}: Networked science in machine learning}.
\newblock \bibinfo{journal}{SIGKDD Explorations} \bibinfo{volume}{15},
  \bibinfo{pages}{49--60}.
%Type = Article
\bibitem[{Ward(1963)}]{ward63}
\bibinfo{author}{Ward, J.}, \bibinfo{year}{1963}.
\newblock \bibinfo{title}{Hierarchical grouping to optimize an objective
  function}.
\newblock \bibinfo{journal}{Journal of the American Statistical Association}
  \bibinfo{volume}{22}, \bibinfo{pages}{236--244}.
%Type = Inproceedings
\bibitem[{Xu et~al.(2003)Xu, Rutenbar and Sakallah}]{XuRuSa03}
\bibinfo{author}{Xu, H.}, \bibinfo{author}{Rutenbar, R.},
  \bibinfo{author}{Sakallah, K.}, \bibinfo{year}{2003}.
\newblock \bibinfo{title}{Sub-{SAT}: A formulation for relaxed boolean
  satisfiability with applications in routing}, in: \bibinfo{booktitle}{IEEE
  Transactions on Computer-Aided Design of Integrated Circuits and Systems},
  pp. \bibinfo{pages}{814--820}.
%Type = Inproceedings
\bibitem[{Xu et~al.(2007)Xu, Hoos and {Leyton-Brown}}]{xu_hierarchical_2007}
\bibinfo{author}{Xu, L.}, \bibinfo{author}{Hoos, H.H.},
  \bibinfo{author}{{Leyton-Brown}, K.}, \bibinfo{year}{2007}.
\newblock \bibinfo{title}{Hierarchical hardness models for {SAT}}, in:
  \bibinfo{booktitle}{Proceedings of the Thirteenth International Conference on
  Principles and Practice of Constraint Programming},
  \bibinfo{publisher}{Springer}. pp. \bibinfo{pages}{696--711}.
%Type = Inproceedings
\bibitem[{Xu et~al.(2010)Xu, Hoos and {Leyton-Brown}}]{xu_hydra_2010}
\bibinfo{author}{Xu, L.}, \bibinfo{author}{Hoos, H.H.},
  \bibinfo{author}{{Leyton-Brown}, K.}, \bibinfo{year}{2010}.
\newblock \bibinfo{title}{Hydra: Automatically configuring algorithms for
  {Portfolio-Based} selection}, in: \bibinfo{booktitle}{Proceedings of the
  {Twenty-Fourth} AAAI Conference on Artificial Intelligence},
  \bibinfo{publisher}{AAAI Press}. pp. \bibinfo{pages}{210--216}.
%Type = Article
\bibitem[{Xu et~al.(2008)Xu, Hutter, Hoos and
  {Leyton-Brown}}]{xu_satzilla_2008}
\bibinfo{author}{Xu, L.}, \bibinfo{author}{Hutter, F.}, \bibinfo{author}{Hoos,
  H.H.}, \bibinfo{author}{{Leyton-Brown}, K.}, \bibinfo{year}{2008}.
\newblock \bibinfo{title}{{SATzilla:} portfolio-based algorithm selection for
  {SAT}}.
\newblock \bibinfo{journal}{Journal of Artificial Intelligence Research}
  \bibinfo{volume}{32}, \bibinfo{pages}{565--606}.
%Type = Inproceedings
\bibitem[{Xu et~al.(2011)Xu, Hutter, Hoos and
  {Leyton-Brown}}]{xu_hydra-mip_2011}
\bibinfo{author}{Xu, L.}, \bibinfo{author}{Hutter, F.}, \bibinfo{author}{Hoos,
  H.H.}, \bibinfo{author}{{Leyton-Brown}, K.}, \bibinfo{year}{2011}.
\newblock \bibinfo{title}{{Hydra-MIP:} automated algorithm configuration and
  selection for mixed integer programming}, in: \bibinfo{booktitle}{Proceedings
  of the {RCRA} Workshop on Experimental Evaluation of Algorithms for Solving
  Problems with Combinatorial Explosion at the {Twenty-Second} International
  Joint Conference on Artificial Intelligence}.
%Type = Inproceedings
\bibitem[{Xu et~al.(2012a)Xu, Hutter, Hoos and
  {Leyton-Brown}}]{xu2012evaluating}
\bibinfo{author}{Xu, L.}, \bibinfo{author}{Hutter, F.}, \bibinfo{author}{Hoos,
  H.H.}, \bibinfo{author}{{Leyton-Brown}, K.}, \bibinfo{year}{2012}a.
\newblock \bibinfo{title}{Evaluating component solver contributions to
  {Portfolio-Based} algorithm selectors}, in: \bibinfo{booktitle}{Proceedings
  of the Fifteenth International Conference on Theory and Applications of
  Satisfiability Testing}, \bibinfo{publisher}{Springer}. pp.
  \bibinfo{pages}{228--241}.
%Type = Inproceedings
\bibitem[{Xu et~al.(2012b)Xu, Hutter, Shen, Hoos and
  Leyton-Brown}]{xu2012satzilla2012}
\bibinfo{author}{Xu, L.}, \bibinfo{author}{Hutter, F.}, \bibinfo{author}{Shen,
  J.}, \bibinfo{author}{Hoos, H.H.}, \bibinfo{author}{Leyton-Brown, K.},
  \bibinfo{year}{2012}b.
\newblock \bibinfo{title}{Satzilla2012: Improved algorithm selection based on
  cost-sensitive classification models}, in: \bibinfo{booktitle}{Proceedings of
  {SAT} Challenge 2012: Solver and Benchmark Descriptions}, pp.
  \bibinfo{pages}{57--58}.

\end{thebibliography}
