Navigate, Understand, Communicate:\\ How Developers Locate Performance Bugs
\IEEEauthorblockN{Sebastian Baltes
Oliver Moseler
Stephan Diehl
\IEEEauthorblockN{ Sebastian Baltes\IEEEauthorrefmark{1
%\boldmath Background: Performance bugs can lead to severe issues regarding computation efficiency, power consumption, and user experience. Locating these bugs is a difficult task because developers have to judge for every costly operation whether runtime is consumed necessarily or unnecessarily. Objective: We wanted to investigate how developers, when locating performance bugs, navigate through the code, understand the program, and communicate the detected issues. Method: We performed a qualitative user study observing twelve developers trying to fix documented performance bugs in two open source projects. The developers worked with a profiling and analysis tool that visually depicts runtime information in a list representation and embedded into the source code view. Results: We identified typical navigation strategies developers used for pinpointing the bug, for instance, following method calls based on runtime consumption. The integration of visualization and code helped developers to understand the bug. Sketches visualizing data structures and algorithms turned out to be valuable for externalizing and communicating the comprehension process for complex bugs. Conclusion: Fixing a performance bug is a code comprehension and navigation problem. Flexible navigation features based on executed methods and a close integration of source code and performance information support the process.
esem15-debugging-arXiv.tex

%% bare_conf.tex
%% V1.3
%% 2007/01/11
%% by Michael Shell
%% See:
%% http://www.michaelshell.org/
%% for current contact information.
%%
%% This is a skeleton file demonstrating the use of IEEEtran.cls
%% (requires IEEEtran.cls version 1.7 or later) with an IEEE conference paper.
%%
%% Support sites:
%% http://www.michaelshell.org/tex/ieeetran/
%% http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/
%% and
%% http://www.ieee.org/

%%*************************************************************************
%% Legal Notice:
%% This code is offered as-is without any warranty either expressed or
%% implied; without even the implied warranty of MERCHANTABILITY or
%% FITNESS FOR A PARTICULAR PURPOSE! 
%% User assumes all risk.
%% In no event shall IEEE or any contributor to this code be liable for
%% any damages or losses, including, but not limited to, incidental,
%% consequential, or any other damages, resulting from the use or misuse
%% of any information contained here.
%%
%% All comments are the opinions of their respective authors and are not
%% necessarily endorsed by the IEEE.
%%
%% This work is distributed under the LaTeX Project Public License (LPPL)
%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used,
%% distributed and modified. A copy of the LPPL, version 1.3, is included
%% in the base LaTeX documentation of all distributions of LaTeX released
%% 2003/12/01 or later.
%% Retain all contribution notices and credits.
%% ** Modified files should be clearly indicated as such, including  **
%% ** renaming them and changing author support contact information. **
%%
%% File list of work: IEEEtran.cls, IEEEtran_HOWTO.pdf, bare_adv.tex,
%%                    bare_conf.tex, bare_jrnl.tex, bare_jrnl_compsoc.tex
%%*************************************************************************

% *** Authors should verify (and, if needed, correct) their LaTeX system  ***
% *** with the testflow diagnostic prior to trusting their LaTeX platform ***
% *** with production work. IEEE's font choices can trigger bugs that do  ***
% *** not appear when using other class files.                            ***
% The testflow support page is at:
% http://www.michaelshell.org/tex/testflow/



% Note that the a4paper option is mainly intended so that authors in
% countries using A4 can easily print to A4 and see how their papers will
% look in print - the typesetting of the document will not typically be
% affected with changes in paper size (but the bottom and side margins will).
% Use the testflow package mentioned above to verify correct handling of
% both paper sizes by the user's LaTeX system.
%
% Also note that the "draftcls" or "draftclsnofoot", not "draft", option
% should be used if it is desired that the figures are to be displayed in
% draft mode.
%
\documentclass[conference]{IEEEtran}
% Add the compsoc option for Computer Society conferences.
%
% If IEEEtran.cls has not been installed into the LaTeX system files,
% manually specify the path to it like:
% \documentclass[conference]{../sty/IEEEtran}

% Set letter paper size:
\setlength{\paperheight}{11in}
\setlength{\paperwidth}{8.5in}
\usepackage[
  pass,% keep layout unchanged 
  % showframe,% show the layout
]{geometry}

% Some very useful LaTeX packages include:
% (uncomment the ones you want to load)


% *** MISC UTILITY PACKAGES ***
%
%\usepackage{ifpdf}
% Heiko Oberdiek's ifpdf.sty is very useful if you need conditional
% compilation based on whether the output is pdf or dvi.
% usage:
% \ifpdf
%   % pdf code
% \else
%   % dvi code
% \fi
% The latest version of ifpdf.sty can be obtained from:
% http://www.ctan.org/tex-archive/macros/latex/contrib/oberdiek/
% Also, note that IEEEtran.cls V1.7 and later provides a builtin
% \ifCLASSINFOpdf conditional that works the same way.
% When switching from latex to pdflatex and vice-versa, the compiler may
% have to be run twice to clear warning/error messages.






% *** CITATION PACKAGES ***
%
\usepackage{cite}
\usepackage[numbers]{natbib} % Sebastian: to be able to use \citeauthor
% cite.sty was written by Donald Arseneau
% V1.6 and later of IEEEtran pre-defines the format of the cite.sty package
% \cite{} output to follow that of IEEE. Loading the cite package will
% result in citation numbers being automatically sorted and properly
% "compressed/ranged". e.g., [1], [9], [2], [7], [5], [6] without using
% cite.sty will become [1], [2], [5]--[7], [9] using cite.sty. cite.sty's
% \cite will automatically add leading space, if needed. Use cite.sty's
% noadjust option (cite.sty V3.8 and later) if you want to turn this off.
% cite.sty is already installed on most LaTeX systems. Be sure and use
% version 4.0 (2003-05-27) and later if using hyperref.sty. cite.sty does
% not currently provide for hyperlinked citations.
% The latest version can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/cite/
% The documentation is contained in the cite.sty file itself.






% *** GRAPHICS RELATED PACKAGES ***
%
\ifCLASSINFOpdf
  % \usepackage[pdftex]{graphicx}
  % declare the path(s) where your graphic files are
  % \graphicspath{{../pdf/}{../jpeg/}}
  % and their extensions so you won't have to specify these with
  % every instance of \includegraphics
  % \DeclareGraphicsExtensions{.pdf,.jpeg,.png}
\else
  % or other class option (dvipsone, dvipdf, if not using dvips). graphicx
  % will default to the driver specified in the system graphics.cfg if no
  % driver is specified.
  % \usepackage[dvips]{graphicx}
  % declare the path(s) where your graphic files are
  % \graphicspath{{../eps/}}
  % and their extensions so you won't have to specify these with
  % every instance of \includegraphics
  % \DeclareGraphicsExtensions{.eps}
\fi
% graphicx was written by David Carlisle and Sebastian Rahtz. It is
% required if you want graphics, photos, etc. graphicx.sty is already
% installed on most LaTeX systems. The latest version and documentation can
% be obtained at: 
% http://www.ctan.org/tex-archive/macros/latex/required/graphics/
% Another good source of documentation is "Using Imported Graphics in
% LaTeX2e" by Keith Reckdahl which can be found as epslatex.ps or
% epslatex.pdf at: http://www.ctan.org/tex-archive/info/
%
% latex, and pdflatex in dvi mode, support graphics in encapsulated
% postscript (.eps) format. pdflatex in pdf mode supports graphics
% in .pdf, .jpeg, .png and .mps (metapost) formats. Users should ensure
% that all non-photo figures use a vector format (.eps, .pdf, .mps) and
% not a bitmapped formats (.jpeg, .png). IEEE frowns on bitmapped formats
% which can result in "jaggedy"/blurry rendering of lines and letters as
% well as large increases in file sizes.
%
% You can find documentation about the pdfTeX application at:
% http://www.tug.org/applications/pdftex





% *** MATH PACKAGES ***
%
%\usepackage[cmex10]{amsmath}
% A popular package from the American Mathematical Society that provides
% many useful and powerful commands for dealing with mathematics. If using
% it, be sure to load this package with the cmex10 option to ensure that
% only type 1 fonts will utilized at all point sizes. Without this option,
% it is possible that some math symbols, particularly those within
% footnotes, will be rendered in bitmap form which will result in a
% document that can not be IEEE Xplore compliant!
%
% Also, note that the amsmath package sets \interdisplaylinepenalty to 10000
% thus preventing page breaks from occurring within multiline equations. Use:
%\interdisplaylinepenalty=2500
% after loading amsmath to restore such page breaks as IEEEtran.cls normally
% does. amsmath.sty is already installed on most LaTeX systems. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/required/amslatex/math/





% *** SPECIALIZED LIST PACKAGES ***
%
%\usepackage{algorithmic}
% algorithmic.sty was written by Peter Williams and Rogerio Brito.
% This package provides an algorithmic environment fo describing algorithms.
% You can use the algorithmic environment in-text or within a figure
% environment to provide for a floating algorithm. Do NOT use the algorithm
% floating environment provided by algorithm.sty (by the same authors) or
% algorithm2e.sty (by Christophe Fiorio) as IEEE does not use dedicated
% algorithm float types and packages that provide these will not provide
% correct IEEE style captions. The latest version and documentation of
% algorithmic.sty can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/algorithms/
% There is also a support site at:
% http://algorithms.berlios.de/index.html
% Also of interest may be the (relatively newer and more customizable)
% algorithmicx.sty package by Szasz Janos:
% http://www.ctan.org/tex-archive/macros/latex/contrib/algorithmicx/




% *** ALIGNMENT PACKAGES ***
%
%\usepackage{array}
% Frank Mittelbach's and David Carlisle's array.sty patches and improves
% the standard LaTeX2e array and tabular environments to provide better
% appearance and additional user controls. As the default LaTeX2e table
% generation code is lacking to the point of almost being broken with
% respect to the quality of the end results, all users are strongly
% advised to use an enhanced (at the very least that provided by array.sty)
% set of table tools. array.sty is already installed on most systems. The
% latest version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/required/tools/


%\usepackage{mdwmath}
%\usepackage{mdwtab}
% Also highly recommended is Mark Wooding's extremely powerful MDW tools,
% especially mdwmath.sty and mdwtab.sty which are used to format equations
% and tables, respectively. The MDWtools set is already installed on most
% LaTeX systems. The lastest version and documentation is available at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/mdwtools/


% IEEEtran contains the IEEEeqnarray family of commands that can be used to
% generate multiline equations as well as matrices, tables, etc., of high
% quality.


%\usepackage{eqparbox}
% Also of notable interest is Scott Pakin's eqparbox package for creating
% (automatically sized) equal width boxes - aka "natural width parboxes".
% Available at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/eqparbox/





% *** SUBFIGURE PACKAGES ***
%\usepackage[tight,footnotesize]{subfigure}
% subfigure.sty was written by Steven Douglas Cochran. This package makes it
% easy to put subfigures in your figures. e.g., "Figure 1a and 1b". For IEEE
% work, it is a good idea to load it with the tight package option to reduce
% the amount of white space around the subfigures. subfigure.sty is already
% installed on most LaTeX systems. The latest version and documentation can
% be obtained at:
% http://www.ctan.org/tex-archive/obsolete/macros/latex/contrib/subfigure/
% subfigure.sty has been superceeded by subfig.sty.



%\usepackage[caption=false]{caption}
%\usepackage[font=footnotesize]{subfig}
% subfig.sty, also written by Steven Douglas Cochran, is the modern
% replacement for subfigure.sty. However, subfig.sty requires and
% automatically loads Axel Sommerfeldt's caption.sty which will override
% IEEEtran.cls handling of captions and this will result in nonIEEE style
% figure/table captions. To prevent this problem, be sure and preload
% caption.sty with its "caption=false" package option. This is will preserve
% IEEEtran.cls handing of captions. Version 1.3 (2005/06/28) and later 
% (recommended due to many improvements over 1.2) of subfig.sty supports
% the caption=false option directly:
%\usepackage[caption=false,font=footnotesize]{subfig}
%
% The latest version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/subfig/
% The latest version and documentation of caption.sty can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/caption/




% *** FLOAT PACKAGES ***
%
%\usepackage{fixltx2e}
% fixltx2e, the successor to the earlier fix2col.sty, was written by
% Frank Mittelbach and David Carlisle. This package corrects a few problems
% in the LaTeX2e kernel, the most notable of which is that in current
% LaTeX2e releases, the ordering of single and double column floats is not
% guaranteed to be preserved. Thus, an unpatched LaTeX2e can allow a
% single column figure to be placed prior to an earlier double column
% figure. The latest version and documentation can be found at:
% http://www.ctan.org/tex-archive/macros/latex/base/



%\usepackage{stfloats}
% stfloats.sty was written by Sigitas Tolusis. This package gives LaTeX2e
% the ability to do double column floats at the bottom of the page as well
% as the top. (e.g., "\begin{figure*}[!b]" is not normally possible in
% LaTeX2e). It also provides a command:
%\fnbelowfloat
% to enable the placement of footnotes below bottom floats (the standard
% LaTeX2e kernel puts them above bottom floats). This is an invasive package
% which rewrites many portions of the LaTeX2e float routines. It may not work
% with other packages that modify the LaTeX2e float routines. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/sttools/
% Documentation is contained in the stfloats.sty comments as well as in the
% presfull.pdf file. Do not use the stfloats baselinefloat ability as IEEE
% does not allow \baselineskip to stretch. Authors submitting work to the
% IEEE should note that IEEE rarely uses double column equations and
% that authors should try to avoid such use. Do not be tempted to use the
% cuted.sty or midfloat.sty packages (also by Sigitas Tolusis) as IEEE does
% not format its papers in such ways.





% *** PDF, URL AND HYPERLINK PACKAGES ***
%
%\usepackage{url}
% url.sty was written by Donald Arseneau. It provides better support for
% handling and breaking URLs. url.sty is already installed on most LaTeX
% systems. The latest version can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/misc/
% Read the url.sty source comments for usage information. Basically,
% \url{my_url_here}.





% *** Do not adjust lengths that control margins, column widths, etc. ***
% *** Do not use packages that alter fonts (such as pslatex).         ***
% There should be no need to do such things with IEEEtran.cls V1.6 and later.
% (Unless specifically asked to do so by the journal or conference you plan
% to submit to, of course. )

% Sebastian
\usepackage[usenames,dvipsnames,svgnames,table]{xcolor} % http://en.wikibooks.org/wiki/LaTeX/Colors
\PassOptionsToPackage{hyphens}{url}
\usepackage{hyperref}
\usepackage{multirow}
\usepackage[export]{adjustbox} % draw boxes around figures
%\usepackage{subcaption} %conflict with IEEEtran, solution see below
\usepackage{tabularx}
\usepackage{booktabs}
% boxes for results
\usepackage[framemethod=tikz]{mdframed}
\newmdenv[
  innerleftmargin=7pt,
  innerrightmargin=7pt,
  tikzsetting={draw=black,dashed,line width=0.5pt,dash pattern = on 4pt off 2pt},
  linecolor=white,
  backgroundcolor=white
]{dashedbox}
\newmdenv[
  innerleftmargin=7pt,
  innerrightmargin=7pt,
  tikzsetting={draw=black, line width=0.5pt},
  linecolor=black,
  backgroundcolor=white
]{normalbox}
% configure enumerate/itemize environment
\let\labelindent\relax
\usepackage{enumitem}

% Subcaption fix (see http://www.michaelshell.org/tex/ieeetran/)
\makeatletter
\let\MYcaption\@makecaption
\makeatother
\usepackage[font=footnotesize]{subcaption}
\makeatletter
\let\@makecaption\MYcaption
\makeatother

% Editorial Commands
\usepackage[export]{adjustbox} % draw boxes around figures
\usepackage{amssymb,amsmath}
\newcommand{\boxedtext}[1]{\fbox{\scriptsize\bfseries\textsf{#1}}}
\newcommand{\nota}[2]{
	\boxedtext{#1}
		{\small$\blacktriangleright$\emph{\textsl{#2}}$\blacktriangleleft$}
}
\newcommand{\todo}[1]{{\color{red}\nota{TODO}{#1}}} % Wichtig: Vor oder nach \todo{} Leerzeichen entfernen
%\renewcommand\todo[1]{} % todos ein-/ausschalten

% Disable single lines at the start of a paragraph (Schusterjungen)
\clubpenalty = 10000
% Disable single lines at the end of a paragraph (Hurenkinder)
\widowpenalty = 10000 \displaywidowpenalty = 10000

% Code
\newcommand{\mycode}[1]{{\small\texttt{#1}}}

% New table column types (http://tex.stackexchange.com/questions/12703/how-to-create-fixed-width-table-columns-with-text-raggedright-centered-raggedlef)
\usepackage{array}
\newcolumntype{L}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{R}[1]{>{\raggedleft\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}

% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}


\begin{document}
%
% paper title
% can use linebreaks \\ within to get better formatting as desired
%\title{Navigate, Understand, Communicate:\\ How Developers Locate Performance Bugs Using Profiling Data}
\title{Navigate, Understand, Communicate:\\ How Developers Locate Performance Bugs}

% author names and affiliations
% use a multiple column layout for up to three different
% affiliations
%\author{\IEEEauthorblockN{Sebastian Baltes, Oliver Moseler, Stephan Diehl}
%\IEEEauthorblockA{Computer Science Deparment\\
%University of Trier\\
%Trier, Germany\\
%Email: s.baltes@uni-trier.de, diehl@uni-trier.de}
%\and
%\IEEEauthorblockN{Fabian Beck}
%\IEEEauthorblockA{VISUS\\
%University of Stuttgart\\
%Stuttgart, Germany\\
%Email: fabian.beck@visus.uni-stuttgart.de}
%}

% conference papers do not typically use \thanks and this command
% is locked out in conference mode. If really needed, such as for
% the acknowledgment of grants, issue a \IEEEoverridecommandlockouts
% after \documentclass

% for over three affiliations, or if they all won't fit within the width
% of the page, use this alternative format:
% 
\author{
\IEEEauthorblockN{
Sebastian Baltes\IEEEauthorrefmark{1},
Oliver Moseler\IEEEauthorrefmark{1},
Fabian Beck\IEEEauthorrefmark{2}, and
Stephan Diehl\IEEEauthorrefmark{1}}
\IEEEauthorblockA{\IEEEauthorrefmark{1}
University of Trier, Germany\\
%Email: s.baltes@uni-trier.de, diehl@uni-trier.de
}
\IEEEauthorblockA{\IEEEauthorrefmark{2}
VISUS, University of Stuttgart, Germany\\
%Email: fabian.beck@visus.uni-stuttgart.de
}}

% use for special paper notices
%\IEEEspecialpapernotice{(Invited Paper)}




% make the title area
\maketitle


\begin{abstract}
%\boldmath
\emph{Background:} Performance bugs can lead to severe issues regarding computation efficiency, power consumption, and user experience. Locating these bugs is a difficult task because developers have to judge for every costly operation whether runtime is consumed necessarily or unnecessarily.
\emph{Objective:} We wanted to investigate how developers, when locating performance bugs, navigate through the code, understand the program, and communicate the detected issues. 
\emph{Method:} We performed a qualitative user study observing twelve developers trying to fix documented performance bugs in two open source projects. The developers worked with a profiling and analysis tool that visually depicts runtime information in a list representation and embedded into the source code view. 
\emph{Results:} We identified typical navigation strategies developers used for pinpointing the bug, for instance, following method calls based on runtime consumption. The integration of visualization and code helped developers to understand the bug. Sketches visualizing data structures and algorithms turned out to be valuable for externalizing and communicating the comprehension process for complex bugs.
\emph{Conclusion:} Fixing a performance bug is a code comprehension and navigation problem. Flexible navigation features based on executed methods and a close integration of source code and performance information support the process. 

\end{abstract}
% IEEEtran.cls defaults to using nonbold math in the Abstract.
% This preserves the distinction between vectors and scalars. However,
% if the conference you are submitting to favors bold math in the abstract,
% then you can use LaTeX's standard command \boldmath at the very start
% of the abstract to achieve this. Many IEEE journals/conferences frown on
% math in the abstract anyway.

% no keywords




% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
\IEEEpeerreviewmaketitle

\section{Introduction}

Performance is a non-functional requirement that every software needs to fulfill, at least to some extent. It might be hard to optimize the performance of already well-implemented algorithms.
But often, bugs in form of unnecessarily complex or slow operations affect the performance of software.
 \citeauthor{Jin12}~\cite{Jin12} define \emph{performance bugs} as ``defects where relatively simple source code changes can significantly speed up software, while preserving functionality.''
In that sense, performance bugs significantly differ from usual bugs, which are deviations of the program behavior from specified functional requirements. Nonetheless, performance bugs are critical as well because they could corrupt user experience, reduce system throughput, increase latency, and waste computational resources~\cite{Jin12, Nistor13}. Although they can be fixed with simple changes, locating and understanding them is a difficult task: complex chains of executed methods need to be traced and, for every statement and branch, the developer needs to clarify whether runtime was consumed not more than appropriate.

To build tools that support developers in locating and fixing performance bugs, it is essential to understand how the debugging process works. Different studies have already been conducted that investigate fixing functional bugs~\cite{Ko06, Johnson13, Lawrance13, Beck15}. We are, however, not aware of any study that focuses on observing developers locating performance bugs. Results from other studies on debugging processes cannot be transferred directly because the steps and tools required to optimize a non-functional requirement like performance are substantially different from those applied for fixing a functional bug.
These differences include: (i) developers cannot analyze whether a program is correct regarding performance because there only exist better or worse solutions; (ii) developers need to investigate not only program state but also runtime consumption; and (iii) collecting runtime information requires to set up realistic benchmarks that differ from usual regression tests. Also, \citeauthor{Jin12}~\cite{Jin12} already pointed at the lack of studies on how performance bugs are fixed by developers.

The user study presented in this paper aims at filling this gap by investigating how developers \emph{navigate} through code, \emph{understand} performance problems, and \emph{communicate} with each other to fix performance bugs.
It is based on a visual performance analysis tool~\cite{Beck13b} that we extended to provide developers versatile means of navigation and support for comprehension (Section~\ref{sec:tool}).
The specific research questions of this paper cover activities that enable developers to fix a performance bug (Section~\ref{sec:research_questions}).
The evaluation we designed to answer these questions is an extensive qualitative user study where twelve developers fixed real-world performance bugs in pairs (Section~\ref{sec:design}).
The results are based on a detailed evaluation of interviews, recordings, and interaction logs.
They provide insights into how developers navigate to locate the bugs and communicate the detected issues to other developers (Section~\ref{sec:results}).
While the controlled setting and the detailed analysis support the credibility of these results, the relatively small number of participants and our focus on collection libraries limits their generalizability (Section~\ref{sec:validity}).
Finally, we discuss our findings in the context of previous studies on related subjects (Section~\ref{sec:related_work}) and name implications for tools and future work (Section~\ref{sec:conclusion}).


\section{Visual Performance Analysis}
\label{sec:tool}

Profiling tools record individual program runs and assign measured performance values to code entities such as methods. While performance considerations could include runtime, memory, or latency, we focus on analyzing runtime consumption as one of the most prominent performance metrics. The standard user interface for inspecting profiling results are lists of code entities that can be sorted by  runtime consumption or organized hierarchically to follow execution sequences. When exploring performance bugs, however, we assume that the source code is a valuable asset as well: developers need to switch back and forth between performance information and code to figure out why a certain amount of runtime was consumed in a specific code entity and how code entities are related to each other. Using different tools or views to explore runtime information and code could result in considerable navigation overhead and unnecessary cognitive load.
Since we want to explore how developers navigate to locate performance bugs, we provide different options for navigation: First, our study is based on IntelliJ IDEA, a popular integrated development environment (IDE) for Java that includes versatile options to navigate through a software project. Second, we provide a sortable list of code entities showing performance information comparable to standard profiling tools (Fig.~\ref{fig:list}). Third, based on an approach introduced by Beck et al.~\cite{Beck13b}, we add word-sized visualizations to augment code entities with performance information in the code editor (Fig.~\ref{fig:insitu}). This section briefly describes the specific tool used in the study, which is an extended version of the original approach by Beck et al.~\cite{Beck13b}. It is capable of profiling Java systems and uses a sampling-based profiling technique.

\begin{figure}[tbp] \centering
\includegraphics[width=0.8\columnwidth, trim=0.0in 0.0in 0.0in 0.0in]{figures/tool/sparkline_new.eps}
\caption{Enlarged and annotated sparkline visualization augmenting the code of a method \texttt{run()}  with runtime information (annotations in blue).}
\label{fig:sparkline}
\end{figure}

\subsection{In-situ Visualization}

Our tool integrates word-sized visualizations, also known as \emph{sparklines}~\cite{Tufte06}, to augment the source code view of the IDE (Fig.~\ref{fig:insitu}). Other so-called \emph{in-situ software visualization} approaches~\cite{Harward10} have already been used to visualize static or dynamic software metrics~\cite{Harward10, Roethlisberger09}, to monitor numeric variables~\cite{Beck13b}, or to support feature location tasks~\cite{Beck15}. The motivation for these kinds of approaches is to avoid \emph{split-attention effects}~\cite{Ayres05,Paas05} that likely affect code navigation and comprehension whenever developers need to switch between code representation and analysis view. Providing such an in-situ approach in the evaluation will give insights whether the suggested advantages are relevant in practical application and how the approach is received by developers.

The in-situ approach enriches every method declaration with a word-sized sparkline visualization that summarizes the most important runtime information regarding the execution of the method in a recorded program execution (\emph{method visualization}, Fig.~\ref{fig:sparkline}).
The central information is the \emph{method time} (i.e., the percentage of total runtime the method has been active) depicted as a value and color-coded in the background of the main rectangle on a scale from light green (low) to dark red (high). As a striped part, the \emph{self time} (i.e., the part of the runtime that was actually consumed by statements of the method) is added to that rectangle. Small arrows at the left indicate how many callers (0, 1, 2, or $>$2) of the method were active; an analogous representation is added for callees on the right. Another box on the right summarizes the threads that executed the method: each little square represents an instance of a thread, the color encodes its type. More details and examples can be found in the description of the original approach~\cite{Beck13b}. 
Within the body of each method, every method call that is covered by recorded execution information is assigned a simplified in-situ visualization (\emph{method call visualization}, Fig.~\ref{fig:insitu}). It expresses the percentage of runtime that the call propagates with respect to the total \emph{method time}.
To support analysis on a higher level of abstraction, the visualizations augmenting every class declaration aggregate the runtime of all methods contained in the class (this feature has been added to the original approach~\cite{Beck13b}). Tooltip dialogs are available for all in-situ visualizations on demand and allow retrieving precise information on runtime, caller, callees, and threads (Fig.~\ref{fig:tooltip}).

\begin{figure}[tbp] \centering
\includegraphics[width=1.0\columnwidth, trim=0.0in 0.0in 0.0in 0.0in, clip=true]{figures/tool/insitu.eps}
\caption{In-situ visualization of performance information within the code view for classes, methods, and method calls.}
\label{fig:insitu}
\end{figure}

\begin{figure}[tbp] \centering
\includegraphics[width=1.0\columnwidth, trim=1.63in 0.03in 0.01in 0.0in, clip=true]{figures/tool/tooltip.eps}
\caption{Tooltip dialog showing details of callers and callees of a in-situ visualization for a class \texttt{TestClass}.}
\label{fig:tooltip}
\end{figure}


\subsection{List Representation}

To also provide a more traditional representation, we added a list view of performance information to the tool (Fig.~\ref{fig:list}). The list presents all executed methods or classes (selectable as tabs) sorted by total runtime consumption in decreasing order (i.e., the `hottest' code entities are listed at the top). For sake of consistency and to provide richer information, we used the same sparkline visualizations as described above to summarize the runtime information of methods and classes. Again, details of the depicted information can be retrieved as tooltip dialogs on demand (Fig.~\ref{fig:tooltip}).
To quickly filter the list, developers might define include and exclude filters by inserting terms in the corresponding text fields at the top of the view; multiple terms are connected with a logical `OR'. 



\subsection{Navigation}

Navigating through the source code of the studied software system is a central task when locating performance bugs.
Therefore, we provide specialized navigation options in addition to the features already available in the IDE (e.g., project overview, class outline, method calls as hyperlinks). While the IDE navigation is limited to static call information, our extensions focus on enabling the developers to follow dynamic instances of method calls, that is, the calls actually performed during the recorded execution:

\begin{itemize}[leftmargin=*, itemsep=1ex]
\item \textbf{In-situ Visualization:} All methods listed in the tooltips of the in-situ visualizations are clickable; the respective class is opened and the editor jumps to the selected method.

\item \textbf{List Representation:} The list items are clickable and open the respective entity in the editor. Again, tooltips provide an option to jump to callers and callees of the  entity.
\end{itemize}

These features do not replace any IDE features, but extend them.
Developers are free to use (or not to use) any of the presented visualizations and navigation features for locating a performance bug.

\begin{figure}[tbp] \centering
\includegraphics[width=1.0\columnwidth, trim=0.0in 4.49in 0.0in 0.0in, clip=true]{figures/tool/list.eps}
\caption{List representation of code entities (methods and classes) sorted by consumed runtime with filter options (image cropped).}
\label{fig:list}
\end{figure}

\section{Research Questions}
\label{sec:research_questions}

From the general goal of better understanding how developers locate and fix performance bugs, we derive research questions that we address in this paper. We first formulate general questions that are too broad to be covered in a single user study. Subquestions acting as instances of the general questions focus the problem to specific issues that we answer based on results of our user study. 

When working on change tasks, like fixing performance bugs, developers do not aim at understanding a system entirely, but want to understand just enough to make the required change~\cite{Singer97}.
To this end, they navigate the source code and try to understand the relevant parts to build up their task context~\cite{Murphy05}. Information sources used for navigation and navigation strategies reflect this understanding process. We also want to learn how the additional features provided by the in-situ visualization approach influence the navigation process.

\noindent \textbf{RQ1} (Navigating and Understanding): \emph{How do developers navigate and what information and representation is supportive for locating a performance bug?}

\begin{itemize}[leftmargin=*, itemsep=1ex]
\item[] \textbf{RQ1.1} \emph{How was information from the profiling tool or other parts of the IDE used to locate the performance bug?}
\item[] \textbf{RQ1.2} \emph{Is the in-situ visualization of the profiling data 
beneficial compared to a traditional list representation?}
\item[] \textbf{RQ1.3} \emph{What navigation strategies do developers pursue to locate a specific performance bug?}
\end{itemize}

Communication can be considered as an externalization of a reasoning process. To gain deeper insights into the understanding process behind locating a performance bug, we also wanted to study how developers explain performance issues. A setting where developers communicate with each other naturally while solving a software development task is pair programming~\cite{Williams00}. As an additional externalization and medium of communication, we investigated the use of sketches for developing an understanding of performance bugs. Like pair programming, sketches are a common and natural method for collaboration in software engineering~\cite{Baltes14, Cherubini07}.

\noindent \textbf{RQ2} (Understanding and Communicating): \emph{How do developers try to understand and explain the causes of performance bugs?}

\begin{itemize}[leftmargin=*, itemsep=1ex]
\item[] \textbf{RQ2.1} \emph{How do developers communicate with each other when locating a performance bug?}
\item[] \textbf{RQ2.2} \emph{Could sketches help understand and communicate a performance bug?}
\end{itemize}

The study described in the following aims at answering these specific research questions. Conducting the study as a realistic software engineering task was a principle that guided its design.

\section{Study Design}
\label{sec:design}

We designed the study as a structured qualitative observation study in a controlled setting.
Twelve software developers participated in teams of two, resulting in six study sessions.
Each session was divided into three phases: after a tutorial, the teams were asked to locate and fix four real-life performance bugs, followed by a questionnaire collecting feedback from the participants.
After summarizing demographic data of the participants, we describe details of the study design and how the sessions were executed.

\subsection{Participants}
\label{sec:participants}

% Colors: 0 = \cellcolor{RedOrange},  1 = \cellcolor{LightSalmon},  2 = \cellcolor{LightGray},  3 = \cellcolor{LightSkyBlue}, 4 = \cellcolor{DeepSkyBlue} 
\begin{table*}[t]
	\caption{Participants (experience in object-oriented programming, Java, collections and data structures, IntelliJ, other IDEs, fixing performance bugs, our tool, and profiling tools in general)}
	\label{tab:participants}
	\centering
	\begin{tabular}{ccccC{0.85cm}C{0.85cm}C{0.85cm}C{0.85cm}C{0.85cm}C{1cm}C{1cm}C{0.85cm}}
		\hline
		Team	& Participant		& Current Occupation		& Work Exp.			& \multicolumn{8}{c}{Experience (no exp. = 0 to 4 = expert)} \\
				&				&						& (years)			& \multicolumn{1}{c}{OOP} & \multicolumn{1}{c}{Java} & \multicolumn{1}{c}{Collec.} & \multicolumn{1}{c}{IntelliJ} & \multicolumn{1}{c}{IDEs} & \multicolumn{1}{c}{Perf.Bugs} & \multicolumn{1}{c}{Our Tool}  & \multicolumn{1}{c}{Profiling} \\ % & \multicolumn{1}{c}{Pair Pr.}\\
		\hline
		\multirow{2}{*}{T1}	& P1 		& Research assistant				& 5		& 4\cellcolor{DeepSkyBlue} & 4\cellcolor{DeepSkyBlue} & 3\cellcolor{LightSkyBlue} & 3\cellcolor{LightSkyBlue} & 3\cellcolor{LightSkyBlue} & 1\cellcolor{LightSalmon} & 1\cellcolor{LightSalmon} & 0\cellcolor{RedOrange} \\ % & 2\cellcolor{LightGray}\\
							& P2		& Research assistant				& 5		& 4\cellcolor{DeepSkyBlue} & 4\cellcolor{DeepSkyBlue} & 4\cellcolor{DeepSkyBlue} & 1\cellcolor{LightSalmon} & 4\cellcolor{DeepSkyBlue} & 2\cellcolor{LightGray} & 1\cellcolor{LightSalmon} & 1\cellcolor{LightSalmon} \\ % & 2\cellcolor{LightGray} \\
		\hline
		\multirow{2}{*}{T2}	& P3		& MSc student, industry exp.		& 1		& 3\cellcolor{LightSkyBlue} & 3\cellcolor{LightSkyBlue} & 2\cellcolor{LightGray} & 0\cellcolor{RedOrange} & 3\cellcolor{LightSkyBlue} & 1\cellcolor{LightSalmon} 	& 0\cellcolor{RedOrange} & 2\cellcolor{LightGray} \\ % & 1\cellcolor{LightSalmon} \\
							& P4		& MSc student, industry exp.	 	& 3		& 3\cellcolor{LightSkyBlue} & 3\cellcolor{LightSkyBlue} & 3\cellcolor{LightSkyBlue} & 1\cellcolor{LightSalmon} & 2\cellcolor{LightGray} & 1\cellcolor{LightSalmon} & 0\cellcolor{RedOrange} & 1\cellcolor{LightSalmon} \\ % & 0\cellcolor{RedOrange}\\
		\hline
		\multirow{2}{*}{T3}	& P5 		& Software developer		& 3		& 4\cellcolor{DeepSkyBlue} & 3\cellcolor{LightSkyBlue} & 4\cellcolor{DeepSkyBlue} & 1\cellcolor{LightSalmon} & 3\cellcolor{LightSkyBlue} & 3\cellcolor{LightSkyBlue} & \cellcolor{LightSalmon}1 & 2\cellcolor{LightGray} \\ % & 3\cellcolor{LightSkyBlue} \\
							& P6		& Diploma student			& 4		& 3\cellcolor{LightSkyBlue} & 3\cellcolor{LightSkyBlue} & 3\cellcolor{LightSkyBlue} & 4\cellcolor{DeepSkyBlue} & 2\cellcolor{LightGray} & 1\cellcolor{LightSalmon} & 1\cellcolor{LightSalmon} & 0\cellcolor{RedOrange} \\ % & 0\cellcolor{RedOrange} \\
		\hline
		\multirow{2}{*}{T4}	& P7		& MSc student			& 0			& 3\cellcolor{LightSkyBlue} & 2\cellcolor{LightGray} & 3\cellcolor{LightSkyBlue} & 1\cellcolor{LightSalmon} & 2\cellcolor{LightGray} & 1\cellcolor{LightSalmon} & 0\cellcolor{RedOrange} & 0\cellcolor{RedOrange} \\ % & 0\cellcolor{RedOrange} \\
							& P8		& MSc student			& 0			& 1\cellcolor{LightSalmon} & 1\cellcolor{LightSalmon} & 0\cellcolor{RedOrange} & 0\cellcolor{RedOrange} & 1\cellcolor{LightSalmon} & 1\cellcolor{LightSalmon} & 0\cellcolor{RedOrange} & 1\cellcolor{LightSalmon} \\ % & 0\cellcolor{RedOrange} \\
		\hline
		\multirow{2}{*}{T5}	& P9		& Research assistant, industry exp.	& 10		& 3\cellcolor{LightSkyBlue} & 2\cellcolor{LightGray} & 3\cellcolor{LightSkyBlue} & 0\cellcolor{RedOrange} & 4\cellcolor{DeepSkyBlue} & 4\cellcolor{DeepSkyBlue} 	& 0\cellcolor{RedOrange} & 3\cellcolor{LightSkyBlue} \\ % & 0\cellcolor{RedOrange} \\
							& P10	& Research assistant, industry exp.	& 6			& 2\cellcolor{LightGray} & 2\cellcolor{LightGray} & 2\cellcolor{LightGray} & 3\cellcolor{LightSkyBlue} & 1\cellcolor{LightSalmon} & 3\cellcolor{LightSkyBlue} & 0\cellcolor{RedOrange}		& 2\cellcolor{LightGray} \\ % & 1\cellcolor{LightSalmon} \\
		\hline
		\multirow{2}{*}{T6}	& P11 	& Software developer	& 15		& 3\cellcolor{LightSkyBlue} & 1\cellcolor{LightSalmon} & 3\cellcolor{LightSkyBlue} & 0\cellcolor{RedOrange} & 3\cellcolor{LightSkyBlue} & 2\cellcolor{LightGray} & 1\cellcolor{LightSalmon} & 2\cellcolor{LightGray} \\ % & 2\cellcolor{LightGray} \\
							& P12	& Software developer	& 1			& 3\cellcolor{LightSkyBlue} & 3\cellcolor{LightSkyBlue} & 2\cellcolor{LightGray} & 2\cellcolor{LightGray} & 2\cellcolor{LightGray} & 1\cellcolor{LightSalmon} & 0\cellcolor{RedOrange} & 1\cellcolor{LightSalmon} \\ % & 0\cellcolor{RedOrange} \\
		\hline
							&		& mean values:	& 4.4		& 3.0\cellcolor{LightSkyBlue} & 2.6\cellcolor{LightSkyBlue} & 2.7\cellcolor{LightSkyBlue} & 1.3\cellcolor{LightSalmon} & 2.5\cellcolor{LightSkyBlue} & 1.8\cellcolor{LightGray} & 0.4 \cellcolor{RedOrange} & 1.3\cellcolor{LightSalmon} \\ % & 0.9\cellcolor{LightSalmon}\\
		\hline
	\end{tabular}

\end{table*}


The twelve software developers who participated in our study were all male and between 22 and 43 years old (mean value: 30.7 years). 
As summarized in Table~\ref{tab:participants}, three of them worked as software developers, four were research assistants at a computer science department (two of them worked full-time in the software industry before), and five were graduate students enrolled in a computer science program (two of them were working half-time as software developers in industry). In total, ten participants had professional work experience in software development (between 1 and 15 years).
We asked the participants to rate their experience in different areas related to the study on Likert items ranging from 0 (no experience) to 4 (expert). %The result of these ratings can be found in Table~\ref{tab:participants}.
Summarizing the ratings, participants had a good level of experience in object-oriented programming, the Java programming language, collections and data structures, and working with integrated development environments.
Four participants had no experience with the IntelliJ IDEA IDE.
However, this did not seem to cause any problems during the study, presumably because modern IDEs have quite similar features and our participants got used to the unfamiliar IDE quickly. %after the introductory phase.
Every participant had at least some experience in fixing performance bugs, but only five participants were experienced (experience $\ge 2$).
Most of them did not know our profiling and visualization tool before. The experience with profiling tools in general was rather low.
Nevertheless, in each team, there was at least one member having some experience with profiling tools.

\subsection{Procedure}

The study procedure was divided into three phases. During the whole study, at least one of the authors was present to help participants in case of questions. All tasks were executed in the same order by all teams.

\noindent\textbf{1) Tutorial:} In the first phase, each pair was introduced to profiling in general, the sampling and analysis approach that our tool uses, and various features of the tool.
To this end, we presented both a video introduction and slides to every team.
At the end of the introductory phase, the participants were asked to analyze the runtime of a binary search tree implementation with two given inputs to familiarize them with the tool.

\begin{figure}[b!] \centering
\includegraphics[width=0.8\columnwidth, frame]{figures/setup/setup_small_cropped.eps}
\caption{Study setup}
\label{fig:setup}
\end{figure} 

\noindent\textbf{2) Locating Bugs:} The teams were prompted to locate and fix four real-life performance bugs having different levels of difficulty. 
After each bug fix, we conducted a structured interview. The laboratory setup is depicted in Figure~\ref{fig:setup}.
The questions for the structured interviews as well as the introductory slides, the video, and the task descriptions for each bug were presented on a laptop next to the computer running the IDE.
These materials are part of the supplementary material~\cite{SupplementaryMaterial}.
Applying pair programming~\cite{Williams00}, one developer was in control of the keyboard and mouse (\emph{driver}), while the other team member acted as a consultant and supervisor (\emph{navigator}).
We switched \textit{driver} and \textit{navigator} after each bug, so that each developer was \emph{driver} for fixing two bugs. 
Participants were asked to verbalize their thoughts, following the \textit{thinking aloud method}~\cite{Boren00}.
To assess how our participants use the profiling tool and navigate through the code, we logged certain user actions, audio-recorded the sessions, and captured the screen.
This enabled us to analyze the understanding process and the communication between the participants.
The pair programming setting in conjunction with the thinking aloud method makes talking more natural compared to a study where the participant is alone.
This setting is based on the \textit{constructive interaction} method proposed by \citeauthor{Miyake86}~\cite{Miyake86}. 

Developers were given sheets of paper and pens that they were allowed to use at any point during the study.
To get insights into the understanding and communication process, they were asked to sketch out the cause of the performance problem and their proposed solution.
This is a realistic scenario because sketches and diagrams play an important role in the daily work of software developers~\cite{Baltes14, Cherubini07}.
We recorded the whole sketching process using an HD video recorder pointed at the area on the table where pen and paper were placed.
This allowed us to analyze developers' understanding of the performance problem using the externalized mental model, the sketch.

\noindent\textbf{3) Questionnaire} In the third and last phase, participants were asked to fill out a questionnaire with questions about demographic data, work experience, experience in different areas (see Section~\ref{sec:participants}), and the usability of the tool.

\subsection{Performance Bugs}

The four performance bugs we used in the second phase were taken from the Apache Commons Collections project (Bugs 1--3) and the \mycode{com.google.common.collect} package of the Guava Libraries project (Bug 4).
These projects were selected because they were mentioned containing interesting performance bugs~\cite{Jin12, Nistor13}, are understandable without specific prior knowledge, and are implemented in Java.
All chosen performance bugs were well-documented and code illustrating the bug was provided. 
Since many performance bugs manifest only for particular inputs~\cite{Nistor13, Jin12}, we prepared test classes to reproduce each bug.
At the beginning of a session, participants were asked to open the test class for the corresponding bug and start the profiling.
They were free to modify or extend the test class if needed.

Based on manual inspection, we classified the bugs as \emph{easy} or \emph{difficult}. In case teams got stuck with one of the bugs, we also prepared advice on how to proceed, which the instructor provided on demand (available as supplementary material~\cite{SupplementaryMaterial}, together with links to the bug descriptions).
 
\noindent\textbf{Bug 1} (easy): 
In class \mycode{ListUtils}, the static method \mycode{retainAll(Collection, Collection)} iterates through 
the first collection and checks whether an element is also contained in the second one; it returns the intersection of the two collections. 
The runtime strongly depends on the dynamic type of the parameters: calling this method with two linear lists results in quadratic runtime complexity, but linear complexity in case the second parameter allows containment checks in constant time (e.g., a hash set). 

\noindent\textbf{Bug 2} (easy):
Class \mycode{SetUniqueList} implements a linear list using a set to maintain uniqueness of elements internally. The performance problem is that the method \mycode{retainAll(...)} calls its superclass implementation as well as \mycode{retainAll(...)} on the internally used set. This leads to two retainment computations, while only one and a few simple maintenance operations would be necessary.

\noindent\textbf{Bug 3} (difficult): 
Within class \mycode{MultiValueMap\$Values}, the omission to override method \mycode{containsAll(...)} causes the superclass method to be called. This method iterates through the collection passed as parameter and checks for every element if it is contained in the current instance of \mycode{Values}. The method \mycode{contains(...)} uses an iterator to loop over all values. This iterator is actually a concatenation of iterators because a \mycode{MultiValueMap} can store multiple values per key. To create such a concatenation of iterators, it takes a complete iteration over all stored value lists. Thus, a call of \mycode{contains(...)} iterates through all values twice and the frequent creation of the iterator is unnecessary.
    
\noindent\textbf{Bug 4} (difficult):  
Class \mycode{RegularImmutableSet} implements a hash table that uses the open addressing approach with linear probing~\cite{Cormen09} to resolve hash collisions. Method \mycode{contains(Object)} consumes much runtime for some special input, which creates long sequences of filled slots so that a lookup operation degrades closer to linear complexity than to constant time (primary clustering issue~\cite{Cormen09}). As a problem of the underlying algorithm, not its implementation, this is not a performance bug according to the definition of \citeauthor{Jin12}~\cite{Jin12}. However, the symptoms the developers observe (i.e., a high runtime consumption) are the same as for a performance bug. We included this example to test how developers react if there is no easy fix available for the performance issue.


\section{Results}
\label{sec:results}

The study sessions lasted between 94 and 164 minutes (mean value: 121 minutes), resulting in over 12 hours of audio and video recordings.
To answer the research questions formulated in Section~\ref{sec:research_questions}, we first analyzed the transcribed answers of the interviews following each bug locating task (Bug 1--4).
To this end, we conducted a \textit{cross-case analysis}~\cite{Seaman99}:
We started with the interview transcripts of the first two teams and wrote down short summaries of participants' statements related to the research questions. 
Then, we compared these statements to determine similarities and differences.
The result of this step was a list with preliminary propositions, based only on the answers of the first two teams.
For each proposition, we wrote down the statements supporting or refuting it.
For all remaining groups, the following process was repeated:
We analyzed the answers of the next team and compiled a list of statements.
Then, we determined if any of those statements supported or refuted the existing propositions.
Supporting statements were added to the corresponding proposition.
In case a statements refuted a proposition, either the proposition was revised or the statement was added to the list of refuting statements for that proposition.
Any additional propositions suggested by the team's statements were added.

The result of this process was a final list of propositions each with a set of supporting and refuting statements.
The transcription of the interviews and a first iteration of the cross-case analysis was conducted by one author.
Then, based on the extracted statements, the resulting propositions were discussed with two other authors.
We decided to drop all propositions with less than four supporting statements. 

For a detailed analysis of code navigation (based on interaction logs) and verbal communication (based on audio and video recordings), we focused on Bug 3.
We assumed that, when starting with the third bug, the participants were familiar with the tool and had some hands-on experience with the Apache Commons Collections library.
Moreover, according to our observations, this bug was the most difficult one to locate and fix. 

\subsection{RQ1 (Navigating and Understanding)}

The first research question (RQ1) covers the navigation process and code comprehension problems related to it. The first and second subquestion (RQ1.1, RQ1.2) were answered based on the results of the cross-case analysis reported in Table~\ref{tab:navigation-cca-1}, while the third one (RQ1.3) was addressed by investigating the recorded interaction logs.

\begin{table}[t]
	\caption{Propositions based on cross-case analysis of interview answers related to RQ1.1 (top) and RQ1.2 (bottom).}
	\label{tab:navigation-cca-1}
	\centering
	\begin{tabularx}{1\columnwidth}{cXc}
		\toprule
		No. & Proposition	& Teams\\		%& \#Sup. Insp.\\
		
		\midrule
		
		1.1 & The dynamic instance of a method call and connected runtime information are important for navigation. & T1, T3, T4, T5 \\ % & 7\\ 
		1.2 & Following high quantities of runtime in the dynamic method call graph is helpful as a navigation strategy. & T1, T2, T3, T6 \\ %& 6 \\
		1.3 & The more complex the performance bug is, the less helpful the provided tool support and information becomes. & T1, T3, T5, T6  \\ %5\\
		
		\midrule 
		2.1 & The integration into the code view provides additional context for the profiling visualization. & T1, T2, T4, T6 \\ % & 14 \\

		2.2 & The overview (list view) was not needed in this setting. & T1, T4, T5  \\ %5 \\

		2.3 & The overview (list view) could be used as a starting point for further analyses. & T1, T2, T4 \\ %4 \\	
		\bottomrule	
					
	\end{tabularx}
\end{table}

\noindent\textbf{RQ1.1} \emph{How was information from the profiling tool or other parts of the IDE used to locate the performance bug?} 

Participants highlighted that locating the bug was particularly supported by using dynamic instances of method calls as links and by the connected runtime information (Prop. 1.1). In this context, the method call visualization that provides runtime information for a specific call was also helpful (T1, T3, T4). Participant P8, for instance, found it useful that ``one could jump to the actual implementation that was executed'' instead of the position where a method has been declared (e.g., in an interface).
This allows developers to follow paths through the code, chasing the largest quantities of runtime consumption as a main navigation strategy (Prop. 1.2).
However, participants also noted that the more complex the bug was, the less helpful the tool support and provided information became (Prop. 1.3). For instance, it was difficult not to get lost in the dynamic call graph (T6). 


\emph{Discussion:} These summarized statements confirm that, besides the actual performance information, the performed calls need to be explored to gain an understanding of the performance bug. Although we already provided sophisticated support for navigating along those dynamic calls, the call structure could still become too complex. How to provide better context to prevent developers from getting lost in the call graph remains an open research question. 


\noindent \textbf{RQ1.2} \emph{Is the in-situ visualization of the profiling data 
beneficial compared to a traditional list representation?}

In general, the integration of profiling information into the source code was received very positively. In contrast to the list representation, the code provides context necessary to understand the performance information (Prop. 2.1).
Three teams (T1, T4, T5) agreed that the list view of our profiling tool was not required for navigation in the setting of the user study (Prop. 2.2). A reason could be that we gave the performance bugs to our participants ``on a silver platter'' (P4).
Participants remarks indicated that in case they were looking for new performance bugs, the list view would be a good starting point to get an overview on the runtime of all captured methods and classes (Prop. 2.3). 

\emph{Discussion:} In general, integrating source code and performance information seems to be a promising direction to follow. Since our focus was not on detecting new performance bugs, but on locating and understanding known ones, our scenario is, however, limited.
Although the list representation was rated as not necessary for our setting, we cannot conclude that it is superfluous. It rather seems likely that both representations complement each other for different usage scenarios.

\noindent\textbf{RQ1.3} \emph{What navigation strategies do developers pursue to locate a specific performance bug?}

\begin{table}[tbp]
	\caption{Frequency of navigation events discerned by types of navigation.}
	\label{tab:navigation-statistics}
	\centering
	\begin{tabularx}{1\columnwidth}{Xrrrr}
		\toprule
		Type of navigation & \multicolumn{4}{c}{Usage} \\
		& \multicolumn{2}{c}{Total} & \multicolumn{2}{c}{Bug 3}\\	
		\midrule
		Method call visualization & 236 & (25\%) & 98 & (24\%) \\
		Method visualization & 39 & (4\%) & 30 & (7\%) \\
		Class visualization & 7 & (1\%) & 7 & (2\%)\\
		Overview list & 20 & (2\%) & 0 & (0\%)\\
		IDE & 657 & (69\%) & 269 & (67\%)\\
		\bottomrule			
	\end{tabularx}
\end{table}

Every navigation event changing the source code view was recorded. We discern IDE events from different events in the profiling tool. As Table~\ref{tab:navigation-statistics} summarizes for all sessions (Bug 1--4), about two thirds of the navigation events were conducted through standard IDE navigation features.
The other third is related to the profiling tool, dominated by the navigation through method call visualizations. Among the other profiling navigation options, only navigation through tooltips of visualizations next to method declarations played a visible role, while class list visualizations and the list view were used only very rarely.
The distribution of events for Bug 3, which we use in closer detail in the following, is very similar to the overall distribution.
Comparing Bug 3 to the other bugs, a different navigation behavior cannot be observed from these aggregated numbers.

We developed a visualization for the event data to analyze navigation strategies and usage patterns (Fig.~\ref{fig:strategies}). The visualization shows a sequence of navigation events on a discrete time axis. On the left, all visited methods and corresponding classes are listed vertically in order of first visit. The color of each event depends on the time the participants spend in the respective method (yellow: $<$6s; orange: 6--26s; red: $>$26s; gray: unknown).
We chose these split points such that events are distributed uniformly. Links between events indicate the type of navigation used: a solid black line represents navigation through a method call visualization, a solid blue line navigation through a method visualization, and a dashed black line all other types of navigation (IDE navigation and class visualizations). Fig.~\ref{fig:strategies} shows two examples, the complete set of visualizations is provided as part of the supplementary material~\cite{SupplementaryMaterial}.

\begin{figure}
\begin{minipage}[b]{1\columnwidth}
\centering
\includegraphics[width=0.88\columnwidth]{figures/nav-vis/group4_perf_bug_3.eps}
\subcaption{Strategy 1 (\emph{Toggle})}
\label{fig:strategy1}
\end{minipage}%
\hfil
\begin{minipage}[b]{1\columnwidth}
\centering
\includegraphics[width=0.88\columnwidth]{figures/nav-vis/group6_perf_bug_3.eps}
\subcaption{Strategy 2 (\emph{Path Following})}
\label{fig:strategy2}
\end{minipage}%
\caption{Visualized first phase of the navigation log for T4 and T6 (Bug 3).}
\label{fig:strategies}
\end{figure}

% Colors: 0 = \cellcolor{RedOrange},  1 = \cellcolor{LightSalmon},  2 = \cellcolor{LightGray},  3 = \cellcolor{LightSkyBlue}, 4 = \cellcolor{DeepSkyBlue} 
\begin{table*}[t]
\renewcommand\arraystretch{0.8}
%Time to fix/resolution, not including concluding remarks and questionnaire
	\caption{Interactions while locating performance bug 3\\(D: During, A: After locating bug, $^{*}$: navigator took over role of driver, Codes: see Table~\ref{tab:codes})} % color coding: $\Delta\ge15$
	\label{tab:communication}
	\centering
	\begin{tabular}{ccccccccccccccc}
		\toprule
		Team & Time & Success & Driver & Navigator & \multicolumn{8}{c}{Codes} & First Strategy & Sketch \\
		& (min.) & & & &Total & DC+HC & DR+HR & QC+QR & PN+PI & CO & RD+RC+RE & Other & & \\
		\midrule
		\multirow{3}{*}{T1}	 & \multirow{3}{*}{30} & \multirow{3}{*}{\checkmark} &  &  & 165 & 46 & 11 & 28 & 5 & 10 & 11 & 54 & \multirow{3}{*}{1} & \multirow{3}{*}{D} \\
		& &	&  P2 & 	& 45\%& 57\% & 55\% &21\% & 0\% & 20\% & 55\% & 54\% & & \\
		& &	& & P1 & 55\%& 43\% & 45\% & 79\% & 100\% & 80\% & 45\% & 46\% & & \\
		\midrule
		\multirow{3}{*}{T2}	 & \multirow{3}{*}{30} & \multirow{3}{*}{\checkmark} &  &  & 112 & 21 & 19 & 24 & 9 & 6 & 9 & 24 & \multirow{3}{*}{1} & \multirow{3}{*}{A}  \\
		& &	&  P4 & 	& 57\% & 67\% &58\% & 54\% & 11\% &33\% & 56\% & 75\% & & \\
		& &	& & P3 & 43\% & 33\% &42\% & 46\% & 89\% &67\% & 44\% & 25\%	& & \\
		\midrule
		\multirow{3}{*}{T3}	 & \multirow{3}{*}{24} & \multirow{3}{*}{\checkmark} &  &  & 78 & 18 & 13 & 10 & 6 & 7 & 3 & 21 & \multirow{3}{*}{2} & \multirow{3}{*}{A} \\
		& &	&  P5 & 	& 63\% & 83\% & 85\% & 90\% & 0\% & 0\% & 100\% & 52\% & & \\
		& &	& & P6 & 37\% & 17\%  & 15\% & 10\% & 100\%  & 100\%  & 0\%	& 48\%	& & \\
		\midrule
		\multirow{3}{*}{T4}	 & \multirow{3}{*}{35} & \multirow{3}{*}{\checkmark} &  &  & 136 & 24 & 22 & 20 & 15 & 7 & 10 & 38 & \multirow{3}{*}{1} & \multirow{3}{*}{D} \\
		& &	&  P7 & 	& 46\% & 58\% & 68\% & 20\% & 0\% & 29\% & 20\% & 68\% & & \\
		& &	& & P8 & 54\% & 42\%  & 32\% & 80\% & 100\%  & 71\%  & 80\% & 32\% & & \\
		\midrule
		\multirow{3}{*}{T5}	 & \multirow{3}{*}{20} & \multirow{3}{*}{$\circ$} &  &  & 48 & 14 & 9 & 10 & 2 & 0 & 2 & 11 & \multirow{3}{*}{-} & \multirow{3}{*}{D} \\
		& &	&  P10$^{*}$ & & 35\% & 21\% & 44\% & 30\% & 0\% & 0\% & 100\% & 45\% & & \\
		& &	& & P9$^{*}$ & 65\% & 79\%  & 56\% & 70\% & 100\%  & 0\%  & 0\% & 55\% & & \\
		\midrule
		\multirow{3}{*}{T6}	 & \multirow{3}{*}{24} & \multirow{3}{*}{$\times$} &  &  & 40 & 15 & 13 & 1 & 2 & 3 & 0 & 6 & \multirow{3}{*}{2} & \multirow{3}{*}{D}  \\
		& &	&  P12 & & 63\% & 73\% & 77\% & 0\% & 0\% & 0\% & 0\% & 67\% & & \\
		& &	& & P11 & 38\% & 27\%  & 23\% & 100\% & 100\%  & 100\%  & 0\% & 33\% & & \\
		\bottomrule
	\end{tabular}
\end{table*}

The first phase of locating the bug is of particular interest because participants try to understand the circumstances of the performance bug and explore the corresponding code.
A general observation is that the method and method call navigations become rare when the team starts solving the bug. Analyzing the navigation phase at the beginning, we identified two general strategies.
Parts of the teams that started with Strategy 1 later followed Strategy 2 and vice versa (Strategy 1 $\rightarrow$ 2: T4, T5; Strategy 2 $\rightarrow$ 1: T6), some alternating several times (T4, T5). 

\begin{itemize}[leftmargin=*, itemsep=1ex]
\item \textbf{Strategy 1 (\emph{Toggle})}: Some teams frequently switched back and forth between the performance test class and important classes regarding the bug (T1, T2, T4, T5; Fig.~\ref{fig:strategy1}). To this end, they use only IDE functionality. In a way, this strategy is comparable to a breadth-first search while exploring the circumstances of the bug. 
\item \textbf{Strategy 2 (\emph{Path Following})}: Another approach was to follow dynamic calls with high runtime consumption through the method call visualizations until reaching a method with high self time (T3, T6; Fig.~\ref{fig:strategy2}).
This is comparable to following a path on the dynamic call graph to its end, similar to a depth-first search. 
\end{itemize}

\emph{Discussion:} Given the heterogeneous characteristics of participants (see Table~\ref{tab:participants}), we were surprised to find clear, repeated strategies. Since most of the interaction subsequences can be assigned unambiguously to either of the strategies, these two strategies seem to be applicable rather universally. None of the strategies is obviously dominating the other, neither in regard of frequency nor order. Further, we were not able to detect other strategies on a comparably generic level.

\begin{normalbox}
\noindent \textbf{RQ1} (Navigating and Understanding): \emph{How do developers navigate and what information and representation is supportive for locating a performance bug?}

\noindent\emph{Summary}: Dynamic instances of method calls and related runtime consumption are central sources of information for locating a performance bug. Visually integrating this information into the code can be beneficial and, in our scenario, replaced a list representation of the performance information. We identified two main navigation strategies that occured in alternating patterns: (i) to toggle between performance test and production code, (ii) to follow paths of dynamic calls.
\end{normalbox}

\begin{table}[tbp]
	\caption{Codes used in Table~\ref{tab:communication}}
	\label{tab:codes}
	\centering
	\begin{tabularx}{\columnwidth}{cX}
		\toprule
		Code	& Description \\
		\midrule
		DC		& Describes source code (e.g., data structure, architecture, algorithm)\\
		HC		& Expresses hypothesis about how the source code works.\\
		DR		& Talks about runtime or refers to profiling data  \\
		HR 		& Expresses hypothesis about runtime \\
		QC		& Question regarding source code (e.g., data structure, architecture, algorithm)\\
		QR		& Questions that explicitly mentions the runtime or profiling data \\
		PN		& Prompt to navigate (e.g., ``go to this method'') \\
		PI		& Prompt to implement (e.g., ``you have to write \texttt{for (int i: ...)}'') \\
		CO		& Disrupting comment (e.g., ``Stop! We have to...'') \\
		RD		& Reads documentation/source code comment aloud \\
		RC		& Read source code aloud \\
		RE 		& Reference to source code (``There is the problem.'') \\
		\bottomrule	
	\end{tabularx}
\end{table}

\subsection{RQ2 (Understanding and Communicating)}
\label{sec:communicating}

The second research question (RQ2) focuses on the understanding process while locating a bug, expressed by the communication that takes place.
To answer the first subquestion (RQ2.1), we transcribed the audio recordings from all sessions for Bug 3.
Then, one of the authors employed an open coding approach~\cite{Corbin08, Charmaz14} and iteratively assigned codes to each statement of the participants:
In a first initial coding phase, the transcripts were coded on a statement level and memos were utilized to structure emerging patterns and similar codes.
In a second, focused, coding phase the codes were revised or merged where applicable, concentrating on how the team members interacted with each other.
During this phase, we iterated over the transcripts several times.
For the second subquestion (RQ2.2), we again utilized the transcribed recordings, but we also integrated results from the cross-case analysis.

\noindent\textbf{RQ2.1} \emph{How do developers communicate with each other when locating a performance bug?}

Table~\ref{tab:communication} shows the results for the twelve codes we base our analysis on; Table~\ref{tab:codes} provides descriptions for the codes.
A complete list with all codes and the data for each session are available as supplementary material~\cite{SupplementaryMaterial}.
The locating phase for Bug 3 lasted between 20 and 35 minutes.
Four teams created a working fix, but all teams needed help by the instructor.
Team T5 created a fix that altered the semantics of the data structure and, for team T6, the instructor had to reveal the complete solution before they were able to implement a fix.
In the following, we will describe the communication behavior of the teams based on the transcripts and codes. 

Most teams expressed their first hypothesis about a possible performance problem in the first half of the session and two teams (T1, T4) in the second half.
The role of the navigator differed across the teams:
In half of the teams (T1, T4, T5), the navigator was very active, asking questions about the source code or profiling data (value in column QC+QR $>$75\%), prompting the driver to navigate to certain methods (PN) or dictating when the driver was writing source code (PI).
In T2, the communication was rather balanced between driver and navigator.
The passive navigators in T3 and T6 mostly observed and infrequently interacted with the driver.  
In all except one team (T5), the navigator expressed interrupting comments (CO), e.g., to indicate his notion of a piece of code.
In most cases, the navigator responded to these comments.
However, two teams stand out: In T3, the driver almost completely ignored the navigator's comments and continued with his source code descriptions and hypotheses (DC+HC 83\%, DR+HR 85\%).
We observed similar behavior for T5, where the navigator took over the role of the driver and finally implemented the solution himself.
One reason for the dominant driver in T3 and the rather dominant navigator in T5 could be a difference in expertise:
Both the driver of T3 and the navigator of T5 rated themselves as more experienced in OOP, collections, and fixing performance bugs compared to their teammates (see Table~\ref{tab:participants}).
When understanding the program, it was quite common to point to source code and read it or the related documentation aloud (RE+RC+RD).
The role of the navigator is often described as thinking strategically instead of focusing on the implementation~\cite{Chong07}.
However, in our study, driver and navigator mostly worked on the same level of abstraction, for instance when reading source code aloud or talking about it (DC+HC).

\emph{Discussion:} Unlike for the navigation strategies, we could not detect as clear communication strategies.
Probably depending on the different level of expertise of the participants and the composition of the teams, the communication patterns diverge more between the teams.
Most teams formulated a hypothesis for the cause of the performance bug relatively early, i.e., in the first half of the session.
They discussed the architecture and algorithms related to the performance bug while working together on their hypothesis.
However, in two teams a dominant team member lead the interaction.

\begin{table}[tbp]
	\caption{Propositions based on cross-case analysis of interview answers related to RQ2.2.}
	\label{tab:navigation-cca-2}
	\centering
	\begin{tabularx}{1\columnwidth}{cXc}
		\toprule
		No. & Proposition	& Teams\\		%& \#Sup. Insp.\\
		
		\midrule

		3.1 & Sketches are a useful tool for explaining a performance bug, but context information is needed to understand them afterwards. & T1, T2, T4, T6 \\ %  & 12 \\
				
		3.2 & Sketches are a suitable documentation means (if ``polished'' enough). & T1, T2, T3, T4, T6 \\ % & 7\\

		3.3 & If and how much sketching occurs depends on the sketching experience of the developers. & T1, T2, T5, T6 \\ % & 6\\

		3.4 & A common sketch vocabulary is needed in the team. & T1, T2, T5 \\ %  & 5 \\

		3.5 & More complex problems or data structures are more likely to be sketched. & T1, T2, T4, T6 \\ %  & 8 \\
						
		3.6 & Sketches can be used to explain dynamic aspects of a program. & T1, T2, T4\\		
		\bottomrule			
	\end{tabularx}
\end{table}

\noindent\textbf{RQ2.2} \emph{Could sketches help understand and communicate a performance bug?}

Four teams created a sketch spontaneously while locating Bug 3, the other two were asked to sketch the problem and their solution afterwards (see Table~\ref{tab:communication}).
Some teams sketched only the static structure of a \mycode{MultiValueMap} (T1, T4, T5), others expressed also dynamic aspects like the execution of method \mycode{contains(...)} (T2, T3, T6; see Fig.~\ref{fig:sketch}).
Expressing dynamic behavior was also mentioned several times during the interviews (see Table~\ref{tab:navigation-cca-2}, Prop. 3.6).
Despite only depicting the static structure, teams T1 and T4 referenced their sketch several times during the session; team T4 used it to explore alternative hypotheses about the data structure (Fig.~\ref{fig:sketch}).
In all teams that created sketches during the location phase, the navigator sketched the data structure and explained aspects of it to the driver.

\begin{figure}[tbp] \centering
	\includegraphics[width=0.55\columnwidth, valign=c]{figures/sketches/sketch_alternatives.eps}
	\hspace{1.5em}
	\includegraphics[width=0.3\columnwidth, valign=c]{figures/sketches/sketch_dynamic.eps}
	\caption{Sketches to explore alternatives (left) and dynamic behavior (right)}
	\label{fig:sketch}
\end{figure}

During the interviews, developers rated sketches as being a useful tool for explaining performance bugs to someone else, especially when trying to understand the data structures in use (e.g., the \mycode{MultiValueMap}).
However, it is not only the artifact that matters but the whole context of creating a sketch (e.g., order of creation, related source code, conversation) (Prop. 3.1).
This makes it difficult to use the sketches afterwards for explaining the problem to someone else, but if they are edited and ``polished'' enough, they may be used for documenting a bug fix (Prop. 3.2).
Generally, the participants noted that the sketching practice depends on the experience of the developers (Prop. 3.3).
One participant even reported a ``training effect'' while sketching the team's solution for Bug 2.   
Furthermore, participants pointed at the need of a common sketch vocabulary for the team (Prop. 3.4).
This vocabulary can be informal and may emerge during a meeting.
P3, for instance, noted that in his team, they agreed on the convention that ``the circle with the dots is always a \mycode{HashSet}''.
Also, more complex problems or data structures are more likely to be sketched (Prop. 3.5).
Several participants reported that there is a point when it becomes too difficult to keep a problem or a data structure in the mind.
This is when a sketch is created and hence the developer's mental model gets externalized.

\emph{Discussion:} Although imposed by the study procedure, sketching was considered mostly positive as an aid for explaining a performance bug. The natural use of sketching performed by the navigator in four teams during the location process suggests that it is an appropriate tool for pair programming scenarios. It becomes less clear, however, whether sketching would also be a useful externalization for a single developer when trying to solve a complex performance bug. 

\begin{normalbox}
\noindent \textbf{RQ2} (Understanding and Communicating): \emph{How do developers try to understand and explain the causes of performance bugs?}

\noindent\emph{Summary}: Most pair programming teams formulated a hypothesis for the cause of the performance bug early and discussed the architecture and algorithms related to the bug.
Within a team, the role of the navigator can range from active (posing questions, commenting) to passive (mostly observing).
Sketches may be a useful tool to explain performance problems to co-workers.
\end{normalbox}


\section{Threats to Validity}
\label{sec:validity}

By inviting participants with diverse backgrounds (research assistants and graduate students with and without industry experience as well as professional software developers) and choosing four real-world performance bugs from widely used open source projects, we tried to limit threats to the external validity of our study.
However, the focus on performance bugs in collection libraries and the limited size of our subject sample limit the generalizability of our findings. 
To counter participants' low experience with profiling tools, we conducted a detailed introduction into our sampling approach and let them experiment with our tool before giving them the first performance bug.
The fact that some participants had low experience with the IntelliJ IDEA IDE may also affect the validity of our study.
We tried to mitigate this threat by concentrating on the third bug where we assumed that the developers had time enough to gather experience with the IDE.
The same applies for the collection libraries in which our participants had to fix the performance bugs.

Since our study sessions took quite long, the internal validity may be affected by fatigue effects, especially for the last bug.
Another threat to internal validity may be the support by one of the authors during the study.
To mitigate this threat, we only gave the participants hints when they got stuck.
Furthermore, the hints had been prepared before conducting the study and were given in the same order to every team.
The fact that most teams did not work together before our study could also affect the results.
Thus, we concentrated on the third bug for our analysis, when the team members had at least some time to get used to each other and the environment.

The reliability of our results may be limited due to the fact that only one author conducted each the initial cross-case analysis and the coding of the interview transcripts.
We tried to mitigate this ``lone researcher bias''~\cite{Burnard08} by later discussing the results among the authors.


\section{Related Studies}
\label{sec:related_work}

As mentioned earlier, we are not aware of any previous study investigating in particular how developers locate performance bugs.
However, studies have been performed that generally address change tasks, program comprehension, feature location, and code navigation.
Beyond that, studies exist that lead to first tools to automatically detect performance bugs in software projects.
We summarize related work and link the results to ours.

\citeauthor{Jin12}~\cite{Jin12} conducted a study of 109 real-world performance bugs to extract certain characteristics.
Based on their findings, they implemented a rule-based performance bug detection and found many previously unknown performance problems in several open source projects.
\citeauthor{Nistor13}~\cite{Nistor13} used the bugs that \citeauthor{Jin12} collected to identify how these bugs depend on loops and implemented an automated oracle for performance bugs.
Using this oracle, they found 42 new performance bugs in 9 open source Java projects. We used three of those bugs in our study.

\citeauthor{Roehm12}~\cite{Roehm12} conducted an observational study with 28 professional software developers.
They found that developers prefer face-to-face communication over documentation, which supports our choice to conduct the study in a pair programming setting.
Furthermore, the bug locating phase of our study models the problem-solution-test work pattern they described: for bug fixing tasks, developers first identify the problem, then search for and apply a solution, and finally test the correctness of the solution.
Similar to \citeauthor{Sillito08}~\cite{Sillito08} and \citeauthor{Ko07}~\cite{Ko07}, \citeauthor{Roehm12} found that while developers comprehend software, they ask and answer questions and test hypotheses about application behavior.
We can partially confirm this for our setting (see Section~\ref{sec:communicating}).
Insufficient investigation of the code prior to a change task may lead to an ``ignorant surgery''~\cite{Parnas94} and thus to less successful solutions~\cite{Robillard04}.
We found that, before implementing a fix, developers extensively navigate and discuss the source code that they think is related to a performance problem.

\citeauthor{Baltes14}~\cite{Baltes14} found in a study involving 394 software practitioners that sketches and diagrams play an important role for understanding and explaining source code.
In accordance to that, \citeauthor{Cherubini07}~\cite{Cherubini07} name understanding and communicating as two of the most important motivations for creating sketches.
In our study, the participants found sketches to be a useful tool for explaining performance bugs.

Most existing studies found no significant influence of developers' personality on pair programming effectiveness~\cite{Salleh09}.
However, developers' expertise can affect pair programming sessions: 
\citeauthor{Chong07} conducted an ethnography study of professional pair programmers from two software development teams.
They found that in teams with different levels of expertise for driver and navigator, the less knowledgeable developer had a tendency to become more passive, letting the expert dominate the interaction.
We also observed this behavior in two of six teams.
Further, they reported that during pair programming, developers usually discuss issues on the same level of abstraction, being in line with \citeauthor{Bryant08}'s findings~\cite{Bryant08}.
We can confirm this with our study.

\citeauthor{Lawrance13}~\cite{Lawrance13} suggest an information foraging perspective on debugging where, instead of testing hypotheses, the developers follow a scent while navigating through the code. In an evaluation with twelve developers, they found that a prediction of developers' code navigation behavior based on this theory was more accurate than predictions based on other models. Also, \citeauthor{Beck15}~\cite{Beck15} describe locating specific code as an iterative foraging process including steps like search, filter, and follow relations. The latter step is similar to exploring the dynamic call graph (Prop. 1.2).
In particular, they investigated how developers locate features related to functional bugs and feature requests in a qualitative user study with 20 developers.
In conformance to our results, for locating a specific feature, they confirm the importance to read the source code and the integration of code with additional information. 

\section{Conclusion}
\label{sec:conclusion}

To better understand how developers locate performance bugs, we conducted a qualitative user study observing twelve developers in a controlled setting. Working in pairs, they tried to solve four different performance issues from two open source Java systems. They used a profiling tool that not only represents profiling results as a list, but also integrates profiling information visually into the source code view of the IDE.
The results of the study show that this integration was well-received by the developers. The in-situ visualization largely replaced the list representation within the investigated scenario, although the list might be helpful as a starting point of further performance analyses.
Hence, adding in-situ visualizations would be a promising opportunity to further improve existing profiling tools.
In general, following dynamic calls is an important means of navigation for locating a performance bug. In particular, we identified two navigation strategies: (i) switching back and forth between performance test and production code, (ii) tracing paths through the dynamic call graph. 
A next step is to investigate to what extent existing tools already support these strategies and derive suggestions for enhancements. 
In our study, we found diverse patterns in the communication between developers working in pairs.
In future work, we plan to further analyze the interaction between the developers while locating and fixing performance bugs.
One interesting aspect could be the knowledge transfer that takes place during these tasks~\cite{Plonka15, Zieris14}.
Sketches could be an appropriate medium to explain performance bugs, in particular, describing complex problems.

% use section* for acknowledgement
\section*{Acknowledgment}

The authors thank all developers participating in the study. Fabian Beck is indebted to the Baden-W\"urttemberg Stiftung for the financial support of this research project within the Postdoctoral Fellowship for Leading Early Career Researchers.


% An example of a floating figure using the graphicx package.
% Note that \label must occur AFTER (or within) \caption.
% For figures, \caption should occur after the \includegraphics.
% Note that IEEEtran v1.7 and later has special internal code that
% is designed to preserve the operation of \label within \caption
% even when the captionsoff option is in effect. However, because
% of issues like this, it may be the safest practice to put all your
% \label just after \caption rather than within \caption{}.
%
% Reminder: the "draftcls" or "draftclsnofoot", not "draft", class
% option should be used if it is desired that the figures are to be
% displayed while in draft mode.
%
%\begin{figure}[!t]
%\centering
%\includegraphics[width=2.5in]{myfigure}
% where an .eps filename suffix will be assumed under latex, 
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{Simulation Results}
%\label{fig_sim}
%\end{figure}

% Note that IEEE typically puts floats only at the top, even when this
% results in a large percentage of a column being occupied by floats.


% An example of a double column floating figure using two subfigures.
% (The subfig.sty package must be loaded for this to work.)
% The subfigure \label commands are set within each subfloat command, the
% \label for the overall figure must come after \caption.
% \hfil must be used as a separator to get equal spacing.
% The subfigure.sty package works much the same way, except \subfigure is
% used instead of \subfloat.
%
%\begin{figure*}[!t]
%\centerline{\subfloat[Case I]\includegraphics[width=2.5in]{subfigcase1}%
%\label{fig_first_case}}
%\hfil
%\subfloat[Case II]{\includegraphics[width=2.5in]{subfigcase2}%
%\label{fig_second_case}}}
%\caption{Simulation results}
%\label{fig_sim}
%\end{figure*}
%
% Note that often IEEE papers with subfigures do not employ subfigure
% captions (using the optional argument to \subfloat), but instead will
% reference/describe all of them (a), (b), etc., within the main caption.


% An example of a floating table. Note that, for IEEE style tables, the 
% \caption command should come BEFORE the table. Table text will default to
% \footnotesize as IEEE normally uses this smaller font for tables.
% The \label must come after \caption as always.
%
%\begin{table}[!t]
%% increase table row spacing, adjust to taste
%\renewcommand{\arraystretch}{1.3}
% if using array.sty, it might be a good idea to tweak the value of
% \extrarowheight as needed to properly center the text within the cells
%\caption{An Example of a Table}
%\label{table_example}
%\centering
%% Some packages, such as MDW tools, offer better commands for making tables
%% than the plain LaTeX2e tabular which is used here.
%\begin{tabular}{|c||c|}
%\hline
%One & Two\\
%\hline
%Three & Four\\
%\hline
%\end{tabular}
%\end{table}


% Note that IEEE does not put floats in the very first column - or typically
% anywhere on the first page for that matter. Also, in-text middle ("here")
% positioning is not used. Most IEEE journals/conferences use top floats
% exclusively. Note that, LaTeX2e, unlike IEEE journals/conferences, places
% footnotes above bottom floats. This can be corrected via the \fnbelowfloat
% command of the stfloats package.

% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
%\IEEEtriggeratref{8}
% The "triggered" command can be changed if desired:
%\IEEEtriggercmd{\enlargethispage{-5in}}

% references section

% can use a bibliography generated by BibTeX as a .bbl file
% BibTeX documentation can be easily obtained at:
% http://www.ctan.org/tex-archive/biblio/bibtex/contrib/doc/
% The IEEEtran BibTeX style support page is at:
% http://www.michaelshell.org/tex/ieeetran/bibtex/
\bibliographystyle{IEEEtranN}
% argument is your BibTeX string definitions and bibliography database(s)
%\bibliography{IEEEabrv,../bib/paper}
%
% <OR> manually copy in the resultant .bbl file
% set second argument of \begin to the number of references
% (used to reserve space for the reference number labels box)
\bibliography{literature,additional}

% that's all folks
\end{document}



esem15-debugging-arXiv.bbl
% Generated by IEEEtranN.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{35}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtranN.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl

\bibitem[Jin et~al.(2012)Jin, Song, Shi, Scherpelz, and Lu]{Jin12}
G.~Jin, L.~Song, X.~Shi, J.~Scherpelz, and S.~Lu, ``Understanding and detecting
  real-world performance bugs,'' in \emph{{PLDI}'12}, 2012, pp. 77--88.

\bibitem[Nistor et~al.(2013)Nistor, Song, Marinov, and Lu]{Nistor13}
A.~Nistor, L.~Song, D.~Marinov, and S.~Lu, ``Toddler: detecting performance
  problems via similar memory-access patterns,'' in \emph{{ICSE}'13}, 2013, pp.
  562--571.

\bibitem[Ko et~al.(2006)Ko, Myers, Coblenz, and Aung]{Ko06}
A.~Ko, B.~A. Myers, M.~J. Coblenz, and H.~H. Aung, ``An exploratory study of
  how developers seek, relate, and collect relevant information during software
  maintenance tasks,'' \emph{{IEEE} TSE}, vol.~32, no.~12, pp. 971--987, 2006.

\bibitem[Johnson et~al.(2013)Johnson, Song, Murphy{-}Hill, and
  Bowdidge]{Johnson13}
B.~Johnson, Y.~Song, E.~R. Murphy{-}Hill, and R.~W. Bowdidge, ``Why don't
  software developers use static analysis tools to find bugs?'' in
  \emph{{ICSE}'13}, 2013.

\bibitem[Lawrance et~al.(2013)Lawrance, Bogart, Burnett, Bellamy, Rector, and
  Fleming]{Lawrance13}
J.~Lawrance, C.~Bogart, M.~M. Burnett, R.~K.~E. Bellamy, K.~Rector, and S.~D.
  Fleming, ``How programmers debug, revisited: An information foraging theory
  perspective,'' \emph{{IEEE} TSE}, vol.~39, no.~2, pp. 197--215, 2013.

\bibitem[Beck et~al.(2015)Beck, Dit, Velasco-Madden, Weiskopf, and
  Poshyvanyk]{Beck15}
F.~Beck, B.~Dit, J.~Velasco-Madden, D.~Weiskopf, and D.~Poshyvanyk,
  ``Rethinking user interfaces for feature location,'' in \emph{{ICPC}'15},
  2015.

\bibitem[Beck et~al.(2013)Beck, Hollerich, Diehl, and Weiskopf]{Beck13b}
F.~Beck, F.~Hollerich, S.~Diehl, and D.~Weiskopf, ``Visual monitoring of
  numeric variables embedded in source code,'' in \emph{{VISSOFT}'13}, 2013,
  pp. 1--4.

\bibitem[Tufte(2006)]{Tufte06}
E.~R. Tufte, \emph{Beautiful evidence}.\hskip 1em plus 0.5em minus 0.4em\relax
  Graphics Press, 2006.

\bibitem[Harward et~al.(2010)Harward, Irwin, and Churcher]{Harward10}
M.~Harward, W.~Irwin, and N.~Churcher, ``In situ software visualisation,'' in
  \emph{{ASWEC}'10}, 2010, pp. 171--180.

\bibitem[R{\"{o}}thlisberger et~al.(2009)R{\"{o}}thlisberger, Harry,
  Villaz{\'{o}}n, Ansaloni, Binder, Nierstrasz, and Moret]{Roethlisberger09}
D.~R{\"{o}}thlisberger, M.~Harry, A.~Villaz{\'{o}}n, D.~Ansaloni, W.~Binder,
  O.~Nierstrasz, and P.~Moret, ``Augmenting static source views in {IDEs} with
  dynamic metrics,'' in \emph{{ICSM}'09}, 2009, pp. 253--262.

\bibitem[Ayres and Sweller(2014)]{Ayres05}
P.~Ayres and J.~Sweller, ``The split-attention principle in multimedia
  learning,'' in \emph{The Cambridge Handbook of Multimedia Learning}.\hskip
  1em plus 0.5em minus 0.4em\relax Cambridge Univ. Press, 2014.

\bibitem[Paas and Sweller(2014)]{Paas05}
F.~Paas and J.~Sweller, ``Implications of cognitive load theory for multimedia
  learning,'' in \emph{The Cambridge Handb. of Multim. Learning}.\hskip 1em
  plus 0.5em minus 0.4em\relax Cambridge Univ. Pr., 2014.

\bibitem[Singer et~al.(1997)Singer, Lethbridge, Vinson, and Anquetil]{Singer97}
J.~Singer, T.~C. Lethbridge, N.~G. Vinson, and N.~Anquetil, ``An examination of
  software engineering work practices,'' in \emph{{CASCON}'97}, 1997, p.~21.

\bibitem[Murphy et~al.(2005)Murphy, Kersten, Robillard, and Cubranic]{Murphy05}
G.~C. Murphy, M.~Kersten, M.~P. Robillard, and D.~Cubranic, ``The emergent
  structure of development tasks,'' in \emph{{ECOOP}'05}, 2005, pp. 33--48.

\bibitem[Williams et~al.(2000)Williams, Kessler, Cunningham, and
  Jeffries]{Williams00}
L.~A. Williams, R.~R. Kessler, W.~Cunningham, and R.~Jeffries, ``Strengthening
  the case for pair programming,'' \emph{{IEEE} Software}, vol.~17, no.~4, pp.
  19--25, 2000.

\bibitem[Baltes and Diehl(2014)]{Baltes14}
S.~Baltes and S.~Diehl, ``Sketches and diagrams in practice,'' in
  \emph{{FSE}'14}, 2014, pp. 530--541.

\bibitem[Cherubini et~al.(2007)Cherubini, Venolia, DeLine, and Ko]{Cherubini07}
M.~Cherubini, G.~Venolia, R.~DeLine, and A.~J. Ko, ``Let's go to the
  whiteboard: how and why software developers use drawings,'' in
  \emph{{CHI}'10}, 2007, pp. 557--566.

\bibitem[Baltes et~al.()Baltes, Moseler, Beck, and
  Diehl]{SupplementaryMaterial}
\BIBentryALTinterwordspacing
S.~Baltes, O.~Moseler, F.~Beck, and S.~Diehl. Navigate, understand,
  communicate: How developers locate performance bugs -- supplementary
  material. [Online]. Available: \url{http://st.uni-trier.de/study-debugging}
\BIBentrySTDinterwordspacing

\bibitem[Boren and Ramey(2000)]{Boren00}
T.~Boren and J.~Ramey, ``Thinking aloud: Reconciling theory and practice,''
  \emph{IEEE TPC}, vol.~43, no.~3, 2000.

\bibitem[Miyake(1986)]{Miyake86}
N.~Miyake, ``Constructive interaction and the iterative process of
  understanding,'' \emph{Cog. sci.}, vol.~10, no.~2, 1986.

\bibitem[Cormen(2009)]{Cormen09}
T.~H. Cormen, \emph{Introduction to Algorithms}.\hskip 1em plus 0.5em minus
  0.4em\relax MIT, 2009.

\bibitem[Seaman(1999)]{Seaman99}
C.~B. Seaman, ``Qualitative methods in empirical studies of software
  engineering,'' \emph{{IEEE} TSE}, vol.~25, no.~4, 1999.

\bibitem[Corbin and Strauss(2008)]{Corbin08}
J.~Corbin and A.~Strauss, \emph{Basics of qualitative research}, 3rd~ed.\hskip
  1em plus 0.5em minus 0.4em\relax Sage Publications, 2008.

\bibitem[Charmaz(2014)]{Charmaz14}
K.~Charmaz, \emph{Constructing grounded theory}.\hskip 1em plus 0.5em minus
  0.4em\relax Sage, 2014.

\bibitem[Chong and Hurlbutt(2007)]{Chong07}
J.~Chong and T.~Hurlbutt, ``The social dynamics of pair programming,'' in
  \emph{{ICSE}'07}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2007, pp.
  354--363.

\bibitem[Burnard et~al.(2008)Burnard, Gill, Stewart, Treasure, and
  Chadwick]{Burnard08}
P.~Burnard, P.~Gill, K.~Stewart, E.~Treasure, and B.~Chadwick, ``Analysing and
  presenting qualitative data,'' \emph{British Dental Journal}, vol. 204,
  no.~8, pp. 429--432, 2008.

\bibitem[Roehm et~al.(2012)Roehm, Tiarks, Koschke, and Maalej]{Roehm12}
T.~Roehm, R.~Tiarks, R.~Koschke, and W.~Maalej, ``How do professional
  developers comprehend software?'' in \emph{{ICSE}'12}, 2012, pp. 255--265.

\bibitem[Sillito et~al.(2008)Sillito, Murphy, and Volder]{Sillito08}
J.~Sillito, G.~C. Murphy, and K.~D. Volder, ``Asking and answering questions
  during a programming change task,'' \emph{{IEEE} TSE}, vol.~34, no.~4, pp.
  434--451, 2008.

\bibitem[Ko et~al.(2007)Ko, DeLine, and Venolia]{Ko07}
A.~Ko, R.~DeLine, and G.~Venolia, ``Information needs in colloc. software
  development teams,'' in \emph{{ICSE}'07}, 2007.

\bibitem[Parnas(1994)]{Parnas94}
D.~L. Parnas, ``Software aging,'' in \emph{{ICSE}'94}, 1994.

\bibitem[Robillard et~al.(2004)Robillard, Coelho, and Murphy]{Robillard04}
M.~P. Robillard, W.~Coelho, and G.~C. Murphy, ``How effective developers
  investigate source code: An exploratory study,'' \emph{{IEEE} TSE}, vol.~30,
  no.~12, 2004.

\bibitem[Salleh et~al.(2009)Salleh, Mendes, Grundy, and Burch]{Salleh09}
N.~Salleh, E.~Mendes, J.~Grundy, and G.~Burch, ``An empir. study of the effects
  of personality in pair programming using the five-factor model,'' in
  \emph{{ESEM}'09}, 2009.

\bibitem[Bryant et~al.(2008)Bryant, Romero, and du~Boulay]{Bryant08}
S.~Bryant, P.~Romero, and B.~du~Boulay, ``Pair programming and the mysterious
  role of the navigator,'' \emph{Int. J. Hum. Comput. Studies}, vol.~66, no.~7,
  pp. 519--529, 2008.

\bibitem[Plonka et~al.(2015)Plonka, Sharp, van~der Linden, and
  Dittrich]{Plonka15}
L.~Plonka, H.~Sharp, J.~van~der Linden, and Y.~Dittrich, ``Knowledge transfer
  in pair programming: An in-depth analysis,'' \emph{Int. J. Hum. Comput.
  Studies}, vol.~73, 2015.

\bibitem[Zieris and Prechelt(2014)]{Zieris14}
F.~Zieris and L.~Prechelt, ``On knowledge transfer skill in pair programming,''
  in \emph{{ESEM}'14}, 2014, pp. 11:1--11:10.

\end{thebibliography}
