Design and optimization of DBSCAN Algorithm based on CUDA
\IEEEauthorblockN{Xizhu Wu
\IEEEauthorblockN{Bingchen Wang
Chenglong Zhang
Lei Song
Lianhe Zhao
Yu Dou
and Zihao Yu
%\boldmath DBSCAN is a very classic algorithm for data clustering, which is widely used in many fields. However, with the data scale growing much more bigger than before, the traditional serial algorithm can not meet the performance requirement. Recently, parallel computing based on CUDA has developed very fast and has great advantage on big data. This paper summarizes the algorithms proposed before and improves the performance of the old DBSCAN algorithm by CUDA. The algorithm uses shared memory as much as possible compared with other algorithms and it has very good scalability. A data set is tested on the algorithm of new version. Finally, we analyze the results and give a conclusion that our algorithm is approximately 97 times faster than the serial version.
1506.02226.tex

%% bare_conf.tex
%% V1.3
%% 2007/01/11
%% by Michael Shell
%% See:
%% http://www.michaelshell.org/
%% for current contact information.
%%
%% This is a skeleton file demonstrating the use of IEEEtran.cls
%% (requires IEEEtran.cls version 1.7 or later) with an IEEE conference paper.
%%
%% Support sites:
%% http://www.michaelshell.org/tex/ieeetran/
%% http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/
%% and
%% http://www.ieee.org/

%%*************************************************************************
%% Legal Notice:
%% This code is offered as-is without any warranty either expressed or
%% implied; without even the implied warranty of MERCHANTABILITY or
%% FITNESS FOR A PARTICULAR PURPOSE!
%% User assumes all risk.
%% In no event shall IEEE or any contributor to this code be liable for
%% any damages or losses, including, but not limited to, incidental,
%% consequential, or any other damages, resulting from the use or misuse
%% of any information contained here.
%%
%% All comments are the opinions of their respective authors and are not
%% necessarily endorsed by the IEEE.
%%
%% This work is distributed under the LaTeX Project Public License (LPPL)
%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used,
%% distributed and modified. A copy of the LPPL, version 1.3, is included
%% in the base LaTeX documentation of all distributions of LaTeX released
%% 2003/12/01 or later.
%% Retain all contribution notices and credits.
%% ** Modified files should be clearly indicated as such, including  **
%% ** renaming them and changing author support contact information. **
%%
%% File list of work: IEEEtran.cls, IEEEtran_HOWTO.pdf, bare_adv.tex,
%%                    bare_conf.tex, bare_jrnl.tex, bare_jrnl_compsoc.tex
%%*************************************************************************

% *** Authors should verify (and, if needed, correct) their LaTeX system  ***
% *** with the testflow diagnostic prior to trusting their LaTeX platform ***
% *** with production work. IEEE's font choices can trigger bugs that do  ***
% *** not appear when using other class files.                            ***
% The testflow support page is at:
% http://www.michaelshell.org/tex/testflow/



% Note that the a4paper option is mainly intended so that authors in
% countries using A4 can easily print to A4 and see how their papers will
% look in print - the typesetting of the document will not typically be
% affected with changes in paper size (but the bottom and side margins will).
% Use the testflow package mentioned above to verify correct handling of
% both paper sizes by the user's LaTeX system.
%
% Also note that the "draftcls" or "draftclsnofoot", not "draft", option
% should be used if it is desired that the figures are to be displayed in
% draft mode.
%
\documentclass[conference]{IEEEtran}
% Add the compsoc option for Computer Society conferences.
%
% If IEEEtran.cls has not been installed into the LaTeX system files,
% manually specify the path to it like:
% \documentclass[conference]{../sty/IEEEtran}





% Some very useful LaTeX packages include:
% (uncomment the ones you want to load)


% *** MISC UTILITY PACKAGES ***
%
%\usepackage{ifpdf}
% Heiko Oberdiek's ifpdf.sty is very useful if you need conditional
% compilation based on whether the output is pdf or dvi.
% usage:
% \ifpdf
%   % pdf code
% \else
%   % dvi code
% \fi
% The latest version of ifpdf.sty can be obtained from:
% http://www.ctan.org/tex-archive/macros/latex/contrib/oberdiek/
% Also, note that IEEEtran.cls V1.7 and later provides a builtin
% \ifCLASSINFOpdf conditional that works the same way.
% When switching from latex to pdflatex and vice-versa, the compiler may
% have to be run twice to clear warning/error messages.






% *** CITATION PACKAGES ***
%
%\usepackage{cite}
% cite.sty was written by Donald Arseneau
% V1.6 and later of IEEEtran pre-defines the format of the cite.sty package
% \cite{} output to follow that of IEEE. Loading the cite package will
% result in citation numbers being automatically sorted and properly
% "compressed/ranged". e.g., [1], [9], [2], [7], [5], [6] without using
% cite.sty will become [1], [2], [5]--[7], [9] using cite.sty. cite.sty's
% \cite will automatically add leading space, if needed. Use cite.sty's
% noadjust option (cite.sty V3.8 and later) if you want to turn this off.
% cite.sty is already installed on most LaTeX systems. Be sure and use
% version 4.0 (2003-05-27) and later if using hyperref.sty. cite.sty does
% not currently provide for hyperlinked citations.
% The latest version can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/cite/
% The documentation is contained in the cite.sty file itself.






% *** GRAPHICS RELATED PACKAGES ***
%
\ifCLASSINFOpdf
  % \usepackage[pdftex]{graphicx}
  % declare the path(s) where your graphic files are
  % \graphicspath{{../pdf/}{../jpeg/}}
  % and their extensions so you won't have to specify these with
  % every instance of \includegraphics
  % \DeclareGraphicsExtensions{.pdf,.jpeg,.png}
\else
  % or other class option (dvipsone, dvipdf, if not using dvips). graphicx
  % will default to the driver specified in the system graphics.cfg if no
  % driver is specified.
  % \usepackage[dvips]{graphicx}
  % declare the path(s) where your graphic files are
  % \graphicspath{{../eps/}}
  % and their extensions so you won't have to specify these with
  % every instance of \includegraphics
  % \DeclareGraphicsExtensions{.eps}
\fi
% graphicx was written by David Carlisle and Sebastian Rahtz. It is
% required if you want graphics, photos, etc. graphicx.sty is already
% installed on most LaTeX systems. The latest version and documentation can
% be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/required/graphics/
% Another good source of documentation is "Using Imported Graphics in
% LaTeX2e" by Keith Reckdahl which can be found as epslatex.ps or
% epslatex.pdf at: http://www.ctan.org/tex-archive/info/
%
% latex, and pdflatex in dvi mode, support graphics in encapsulated
% postscript (.eps) format. pdflatex in pdf mode supports graphics
% in .pdf, .jpeg, .png and .mps (metapost) formats. Users should ensure
% that all non-photo figures use a vector format (.eps, .pdf, .mps) and
% not a bitmapped formats (.jpeg, .png). IEEE frowns on bitmapped formats
% which can result in "jaggedy"/blurry rendering of lines and letters as
% well as large increases in file sizes.
%
% You can find documentation about the pdfTeX application at:
% http://www.tug.org/applications/pdftex





% *** MATH PACKAGES ***
%
%\usepackage[cmex10]{amsmath}
% A popular package from the American Mathematical Society that provides
% many useful and powerful commands for dealing with mathematics. If using
% it, be sure to load this package with the cmex10 option to ensure that
% only type 1 fonts will utilized at all point sizes. Without this option,
% it is possible that some math symbols, particularly those within
% footnotes, will be rendered in bitmap form which will result in a
% document that can not be IEEE Xplore compliant!
%
% Also, note that the amsmath package sets \interdisplaylinepenalty to 10000
% thus preventing page breaks from occurring within multiline equations. Use:
%\interdisplaylinepenalty=2500
% after loading amsmath to restore such page breaks as IEEEtran.cls normally
% does. amsmath.sty is already installed on most LaTeX systems. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/required/amslatex/math/





% *** SPECIALIZED LIST PACKAGES ***
%
%\usepackage{algorithmic}
% algorithmic.sty was written by Peter Williams and Rogerio Brito.
% This package provides an algorithmic environment fo describing algorithms.
% You can use the algorithmic environment in-text or within a figure
% environment to provide for a floating algorithm. Do NOT use the algorithm
% floating environment provided by algorithm.sty (by the same authors) or
% algorithm2e.sty (by Christophe Fiorio) as IEEE does not use dedicated
% algorithm float types and packages that provide these will not provide
% correct IEEE style captions. The latest version and documentation of
% algorithmic.sty can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/algorithms/
% There is also a support site at:
% http://algorithms.berlios.de/index.html
% Also of interest may be the (relatively newer and more customizable)
% algorithmicx.sty package by Szasz Janos:
% http://www.ctan.org/tex-archive/macros/latex/contrib/algorithmicx/




% *** ALIGNMENT PACKAGES ***
%
%\usepackage{array}
% Frank Mittelbach's and David Carlisle's array.sty patches and improves
% the standard LaTeX2e array and tabular environments to provide better
% appearance and additional user controls. As the default LaTeX2e table
% generation code is lacking to the point of almost being broken with
% respect to the quality of the end results, all users are strongly
% advised to use an enhanced (at the very least that provided by array.sty)
% set of table tools. array.sty is already installed on most systems. The
% latest version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/required/tools/


%\usepackage{mdwmath}
%\usepackage{mdwtab}
% Also highly recommended is Mark Wooding's extremely powerful MDW tools,
% especially mdwmath.sty and mdwtab.sty which are used to format equations
% and tables, respectively. The MDWtools set is already installed on most
% LaTeX systems. The lastest version and documentation is available at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/mdwtools/


% IEEEtran contains the IEEEeqnarray family of commands that can be used to
% generate multiline equations as well as matrices, tables, etc., of high
% quality.


%\usepackage{eqparbox}
% Also of notable interest is Scott Pakin's eqparbox package for creating
% (automatically sized) equal width boxes - aka "natural width parboxes".
% Available at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/eqparbox/





% *** SUBFIGURE PACKAGES ***
%\usepackage[tight,footnotesize]{subfigure}
% subfigure.sty was written by Steven Douglas Cochran. This package makes it
% easy to put subfigures in your figures. e.g., "Figure 1a and 1b". For IEEE
% work, it is a good idea to load it with the tight package option to reduce
% the amount of white space around the subfigures. subfigure.sty is already
% installed on most LaTeX systems. The latest version and documentation can
% be obtained at:
% http://www.ctan.org/tex-archive/obsolete/macros/latex/contrib/subfigure/
% subfigure.sty has been superceeded by subfig.sty.



%\usepackage[caption=false]{caption}
%\usepackage[font=footnotesize]{subfig}
% subfig.sty, also written by Steven Douglas Cochran, is the modern
% replacement for subfigure.sty. However, subfig.sty requires and
% automatically loads Axel Sommerfeldt's caption.sty which will override
% IEEEtran.cls handling of captions and this will result in nonIEEE style
% figure/table captions. To prevent this problem, be sure and preload
% caption.sty with its "caption=false" package option. This is will preserve
% IEEEtran.cls handing of captions. Version 1.3 (2005/06/28) and later
% (recommended due to many improvements over 1.2) of subfig.sty supports
% the caption=false option directly:
%\usepackage[caption=false,font=footnotesize]{subfig}
%
% The latest version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/subfig/
% The latest version and documentation of caption.sty can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/caption/




% *** FLOAT PACKAGES ***
%
%\usepackage{fixltx2e}
% fixltx2e, the successor to the earlier fix2col.sty, was written by
% Frank Mittelbach and David Carlisle. This package corrects a few problems
% in the LaTeX2e kernel, the most notable of which is that in current
% LaTeX2e releases, the ordering of single and double column floats is not
% guaranteed to be preserved. Thus, an unpatched LaTeX2e can allow a
% single column figure to be placed prior to an earlier double column
% figure. The latest version and documentation can be found at:
% http://www.ctan.org/tex-archive/macros/latex/base/



%\usepackage{stfloats}
% stfloats.sty was written by Sigitas Tolusis. This package gives LaTeX2e
% the ability to do double column floats at the bottom of the page as well
% as the top. (e.g., "\begin{figure*}[!b]" is not normally possible in
% LaTeX2e). It also provides a command:
%\fnbelowfloat
% to enable the placement of footnotes below bottom floats (the standard
% LaTeX2e kernel puts them above bottom floats). This is an invasive package
% which rewrites many portions of the LaTeX2e float routines. It may not work
% with other packages that modify the LaTeX2e float routines. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/sttools/
% Documentation is contained in the stfloats.sty comments as well as in the
% presfull.pdf file. Do not use the stfloats baselinefloat ability as IEEE
% does not allow \baselineskip to stretch. Authors submitting work to the
% IEEE should note that IEEE rarely uses double column equations and
% that authors should try to avoid such use. Do not be tempted to use the
% cuted.sty or midfloat.sty packages (also by Sigitas Tolusis) as IEEE does
% not format its papers in such ways.





% *** PDF, URL AND HYPERLINK PACKAGES ***
%
%\usepackage{url}
% url.sty was written by Donald Arseneau. It provides better support for
% handling and breaking URLs. url.sty is already installed on most LaTeX
% systems. The latest version can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/misc/
% Read the url.sty source comments for usage information. Basically,
% \url{my_url_here}.





% *** Do not adjust lengths that control margins, column widths, etc. ***
% *** Do not use packages that alter fonts (such as pslatex).         ***
% There should be no need to do such things with IEEEtran.cls V1.6 and later.
% (Unless specifically asked to do so by the journal or conference you plan
% to submit to, of course. )


% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}

\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{amssymb}
\usepackage{amsmath}
\graphicspath{{picture/}}

\newcommand{\fun}[2]{#1{\footnotesize #2}}

\begin{document}
%
% paper title
% can use linebreaks \\ within to get better formatting as desired
\title{Design and optimization of DBSCAN Algorithm based on CUDA}


% author names and affiliations
% use a multiple column layout for up to three different
% affiliations
%\author{\IEEEauthorblockN{Xizhu Wu}
%\IEEEauthorblockA{Department of Computer Science and Technonlogy\\
%Nanjing University\\
%Nanjing, China\\
%Email: yuriwxz@gmail.com}
%\and
%\IEEEauthorblockN{Chengxiang Jiang}
%\IEEEauthorblockA{Department of Computer Science and Technonlogy\\
%Nanjing University\\
%Nanjing, China\\
%Email: jcx.nju@gmail.com}
%\and
%\IEEEauthorblockN{Mengtian Li}
%\IEEEauthorblockA{Department of Computer Science and Technonlogy\\
%Nanjing University\\
%Nanjing, China\\
%Email: mtkjaguar@gmail.com\\
%Telephone: 136--7516--2063}
%\and
%\IEEEauthorblockN{Zihao Yu}
%\IEEEauthorblockA{Department of Computer Science and Technonlogy\\
%Nanjing University\\
%Nanjing, China\\
%Email: njucs2010yzh@hotmail.com}}

% conference papers do not typically use \thanks and this command
% is locked out in conference mode. If really needed, such as for
% the acknowledgment of grants, issue a \IEEEoverridecommandlockouts
% after \documentclass

% for over three affiliations, or if they all won't fit within the width
% of the page, use this alternative format:
%
\author{\IEEEauthorblockN{Bingchen Wang,
Chenglong Zhang,
Lei Song,
Lianhe Zhao,
Yu Dou,
and Zihao Yu
}
\IEEEauthorblockA{ Institute of Computing Technology \\
	Chinese Academy of Sciences \\
	Beijing, China 100080
}}




% use for special paper notices
%\IEEEspecialpapernotice{(Invited Paper)}




% make the title area
\maketitle


\begin{abstract}
%\boldmath
DBSCAN is a very classic algorithm for data clustering, which is widely used in many fields. However, with the data scale growing much more bigger than before, the traditional serial algorithm can not meet the performance requirement. Recently, parallel computing based on CUDA has developed very fast and has great advantage on big data. This paper summarizes the algorithms proposed before and improves the performance of the old DBSCAN algorithm by CUDA. The algorithm uses shared memory as much as possible compared with other algorithms and it has very good scalability. A data set is tested on the algorithm of new version. Finally, we analyze the results and give a conclusion that our algorithm is approximately 97 times faster than the serial version.
\end{abstract}
% IEEEtran.cls defaults to using nonbold math in the Abstract.
% This preserves the distinction between vectors and scalars. However,
% if the conference you are submitting to favors bold math in the abstract,
% then you can use LaTeX's standard command \boldmath at the very start
% of the abstract to achieve this. Many IEEE journals/conferences frown on
% math in the abstract anyway.

% no keywords




% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
\IEEEpeerreviewmaketitle



\section{Introduction}
% no \IEEEPARstart

According to the data similarity, a group of data can be divided into some subsets. This progress is called clustering. Generally, there are four types of main cluster method, including partition methods, hierarchical methods, density-based methods and grid-based methods. DBSCAN \cite{1} has been widely used in many aspects as one of the density-based methods. Many methods has been proposed to improve the performance of DBSCAN by using MPI, MAP-REDUCE and other parallel computing methods \cite{2}\cite{3}.

In recent years, researches about General Purpose-Graphics Processing Unit(GPGPU) has been hot, especially the CUDA frame released by NVIDA has provided the support for C which gives much more convenience for the programming and improvement of GPU algorithms \cite{4}\cite{5}.GPU has many advantages in high performance computing such as high degree of parallelism and compute-intensive. Both of them are very suitable for data mining algorithms. Currently, some DBSCAN algorithms using parallel computing based on GPU has been proposed, including G-DBSCAN \cite{6},CUDA-DCLust \cite{7} and CUDASCAN \cite{8}. These algorithms are for more complicated with too many control flow branches. As a result, they have a low degree of parallelism and can not fully use the structure of GPU to improve the performance.

This paper divides the DBSCAN algorithm based on CUDA into three steps. 
\begin{enumerate}
	\item compute the distance between each pair of data-nodes and generate the distance matrix.
	\item get core-points by the distance matrix and generate primitive clusters
	\item merge the primitive clusters.
\end{enumerate}
Each step of the algorithm can be fully parallelized and easy to be improved according to the structure of GPU.  

The remaining of the paper is organized as five parts. The first part gives a brief introduction of the concepts about DBSCAN. The second part describes the DBSCAN algorithm based on GPU. The third part has given the details about how to improve the algorithm based on CUDA. The fourth part shows the experimental results and analysis. Finally, the conclusion of the research has been given.

% You must have at least 2 lines in the paragraph with the drop letter
% (should never be an issue)

\section{Background}

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a data clustering algorithm based on density which can find out the high density connected area separated by those low density area. DBSCAN is not sensitive to the noises and is able to handle any kind of data in various shapes. \cite{9}

Here are some concepts about DBSCAN.

\begin{enumerate}
	\item $\epsilon$-Neighborhood: Given the radius $\epsilon$ of a point, all points within distance $\epsilon$ of it are called $\epsilon$-Neighborhoods.
	\item A point $p$ is a core point if at least $MinPts$ points are within distance $\epsilon$ of it, and those points are said to be directly reachable from $p$. No points are reachable from a non-core point.
	\item A point $q$ is reachable from $p$ if there is a path $p_1, \dots, p_n$ with $p_1 = p$ and $p_n = q$, where each $p_{i+1}$ is directly reachable from $p_i$ (so all the points on the path must be core points, with the possible exception of $q$).
\end{enumerate}

The definition of a cluster mainly contains two things:
\begin{enumerate}
	\item Given the cluster $C$, if $p\in C$ and $q$ is reachable from $p$ within distance $\epsilon$, then $q \in C$.
	\item $\forall p, q \in C$, $p$ is reachable from $q$. 
\end{enumerate}

Serial algorithm of DBSCAN contains three steps.
\begin{enumerate}
	\item Calculate the distance of each pair of data nodes and store them in a two-dimensional matrix��
	\item Build primitive clusters: find all core points and the neighborhood of each core point. The neighborhoods are primitive clusters.
	\item Merge primitive clusters: Find out all reachable core points by iteration and merge their primitive clusters.
\end{enumerate}

\section{Related Work}
DBSCAN algorithm based on GPU mainly includes G-DBSCAN, CUDA-DClust and CUDASCAN.

G-DBSCAN \cite{6} includes two steps: Firstly, generate the matrix of each pair of points according to eps. Secondly, begin to proceed BFS from any point and mark the points on the same tree as a cluster.

CUDA-DClust \cite{7} uses parallel computing of GPU blocks and proceed DFS at multiple points at the same time. Sub-clusters are called chains. By determining if there are overlapping core points between chains, the collision matrix is constructed and then chains are merged according to the collision matrix. 

CUDAScan algorithm \cite{8} divides a data set into domains stored in shared memory to compute. Each domain is calculated by a block to fully take the advantage of the speed of GPU visiting shared memory. Firstly, proceed DBSCAN in each domain to get local clusters separately and concurrently. Secondly, by determining if there are conflicts between local clusters to generate the collision matrix. Finally, merge all local clusters together.

The algorithms above mostly build local clusters with the help of tree structures. This method has a low degree of parallelism and difficult to deal with the cooperation of different blocks. So many global memory accesses consuming much time are in need. CUDAScan reduces visits for global memory through dividing the data. However, the size of blocks is limited and it is not suitable for high density data.

\section{Algorithm Design}

We use the serial algorithm running a test set with 23040 points and each point has a 3D coordinate. Then use the performance analysis tool \emph{gprof} to analyze the results.

\begin{table}[!htbp]
	\centering
	\caption{Execution times for each step of the serial DBSCAN algorithm with data set of size 23040}
	\label{1}
\begin{tabular}{|l|l|l|}
	\hline
	Step & Execution time/s & Ratio \\
	\hline
	Distance calculation & 1.63 & $66.27\%$ \\
	\hline
	primitive clusters construction & 0.80 & $32.62\%$ \\
	\hline
	Clusters merging & 0.03 & $1.22\%$ \\
	\hline
\end{tabular}
\end{table}

Table~\ref{1} shows that distance calculation consumes $66.27\%$ of total time which is the bottleneck of serial algorithm.

\subsection{Use CUDA to accelerate distance calculation}

The distance calculation of each pair of points only depends on the coordinate of both points which has nothing to do with other information. When all the coordinates of each points have been given, the distance of any two points can be calculated concurrently. So it is very easy to use CUDA to accelerate. Because the reason for distance calculation is to see if the distance between two points is shorter than $\epsilon$, distance calculation is faster if using $\epsilon^2$ to compare with instead of $\epsilon$.

\subsubsection{Baseline version}

Each CUDA thread is responsible for the distance calculation between one point and other $N$ points. Each thread fills in one line in the \texttt{distance} matrix. Threads go over the coordinate information of all points by loops which can calculate the distance between them and the goal point in each thread.

\subsubsection{Use memory coalescing}

In the baseline version, threads need to read the coordinate information from the point matrix in the global memory and write back the distance to \texttt{distance} matrix in the global memory. However both visits to global memory do not have the properties of memory coalescing. Transposing for both matrixes is done to improve the performance brought by memory coalescing. But we do not need to actually transpose the matrix. As for point matrix, we only need to change the definition of matrix(changed to \texttt{point[3][N]}) and the code about how to read the coordinate information by CPU. And \texttt{distance} matrix is symmetrical so that each thread writing one row in the matrix will get the same result. After changing, adjacent threads will visit the continuous positions in the point matrix at the same time. Meanwhile, distance can be written on continuous positions in the \texttt{distance} matrix to meet the need of memory coalescing. 

\subsubsection{Use shared memory}

In the last version, each thread will read coordinate information from global memory. In fact, every thread will use coordinate information of each points while calculating distance. Shared memory can be used for the sharing feature which is that each thread reads coordinate info of each points into shared memory so that each point only needs to be read once from the shared memory. It can be used by \texttt{TPB} threads in one thread block. Because the limitation of shared memory, we can not read info of all points into shared memory when \texttt{N} is too large. So we read them in different parts. Read coordinate information of \texttt{TPB} threads into shared memory each time and then calculate the distance between the goal point of each thread and \texttt{TPB} points. Finally, write results into \texttt{distance} matrix. Repeat the process until all points are visited. At the same time, a thread needs the coordinate information of thread goal point to calculate the distance, which is private for each thread. So we can store the information in the thread register before the distance calculation. As a result, the information about coordinate of thread goal points is only read once from the global memory which can be used in the next \texttt{N} times of distance calculation.

\subsubsection{Loop unrolling}

In the version of using shared memory, the inner loop only calculate one distance. We proceed the most inner loop unrolling, one iteration will calculate 32 distance information so that it will reduce control cost related to iterations.

\subsection{Use CUDA to construct primitive clusters}

We use \texttt{cluster} matrix to identify the relationship between the clusters and the points. Each column of the matrix means one cluster and each row means one point. For example, \texttt{cluster[i][j]=1} if and only if \texttt{point[j]} is in the \texttt{i}th cluster. Besides use the variable valid to identify if the cluster is valid. For example, \texttt{valid[i]=1} if and only if the cluster in ith column of \texttt{cluster} matrix is valid. The essence of the construction of primitive clusters is to construct \texttt{cluster} matrix. The specific process is as follows: for each point as in \texttt{point[i]}, scanning all the distance information with other points. If $\texttt{distance[i][j]}\le \epsilon^2$, then \texttt{cluster[i][j]} is set to $1$ which means the \texttt{point[j]} is located in the $\epsilon$-neighborhood of \texttt{point[i]}; During the scan, calculate the points which are in the $\epsilon$-neighborhood of \texttt{point[i]}.If the number is greater than $MinPts$, then \texttt{valid[i]} is set to $1$.

Constructing \texttt{cluster} matrix only depends on the distance information between all points. When the information is given, all columns of \texttt{cluster} matrix are independently constructed which can be built concurrently. So it is easy to use CUDA to accelerate. Each CUDA thread is responsible for writing one column of \texttt{cluster} matrix. The thread use the iteration to visit the corresponding column of \texttt{distance} matrix and to compare distance with $\epsilon^2$ which is recorded. Each CUDA thread is not for one element of the matrix but for a column. The reason for that is we need to count the number of points which are in the $\epsilon$-neighborhood to build the \texttt{valid} vector. The method will lead to extra operations while counting.

\subsubsection{Calculate distance and construct primitive clusters at the same time}

In fact, the \texttt{distance} matrix is only for constructing primitive clusters. The distance calculation and comparing the result with $\epsilon^2$ can be proceeded at the same time. Meanwhile construct the \texttt{cluster} matrix so that the distance do need to be stored in the global memory.

\subsubsection{Put the iteration code outsides}

In the last version, the most inner iteration of the thread is to calculate the square of Euclidean distance between thread goal point and other points. Assuming that the coordinate of the thread goal point is $(t_x,t_y,t_z)$ and the $n$th point is $(p[n]_x, p[n]_y, p[n]_z)$. Then the thread will need the following calculation.

$$dist(t, p[n]) = (t_x-p[n]_x)^2+(t_y-p[n]_y)^2+(t_z-p[n]_z)^2$$

Noticed that for a thread, the information of thread goal point do not change with the iteration. So we can move the calculation only related to thread goal point before the iteration. To do that, we need to change the way of distance calculation. 

\begin{align*}
 & dist(t, p[n]) \\
 = &t_x^2-2t_xp[n]_x+p[n]_x^2 + t_y^2-2t_yp[n]_y+p[n]_y^2 +\\
 &t_z^2-2t_zp[n]_z+p[n]_z^2 \\
 = &(t_x^2+t_y^2+t_z^2) -(2t_xp[n]_x+2t_yp[n]_y+2t_zp[n]_z)+ \\
 & (p[n]_x^2+p[n]_y^2+p[n]_z^2)
\end{align*}

Given $T, X, Y, Z, P[n]$ as follows:
\begin{align*}
	T &= t_x^2+t_y^2+t_z^2 \\
	X &= 2t_x \\
	Y &= 2t_y \\
	Z &= 2t_z \\
	P[n] &= p[n]_x^2+p[n]_y^2+p[n]_z^2
\end{align*}

We can see that $T, X, Y, Z$ are all available to calculate before the iteration and $P[n]$ is not relevant with the thread goal point. We can calculate after reading the information about \texttt{TPB} points and putting it into shared memory which is shared by all threads. So wee only need to calculate once about $p[n]$ and do not need to calculate at each thread.

After putting the iteration code outside, each thread only needs to calculate in the iteration as follows:

$$dist(t, p[n]) = T + P[n] - (Xp[n]_x+Yp[n]_y+Zp[n]_z)$$

\subsection{Use CUDA to merge clusters}

Cluster merging is a process which has strong dependence on order. Every merge changes the status of the cluster. When the ith cluster is merged with the jth cluster, points in the jth cluster will be added and the points in the ith cluster will be deleted. We use a \texttt{cluster} matrix to record a cluster which can helps to compute concurrently. When a point is added in one cluster, the corresponding element in the \texttt{cluster} matrix is set from 0 to 1. There are no conditions for elements changing from 1 to 0. This means that all clusters can be merged to one certain cluster at the same time without the need for synchronize or atomic operations. To delete a cluster, we only need to set corresponding component in the \texttt{valid} vector into 0.

Based on the analysis, we can use CUDA to accelerate the cluster merging easily. The kernel function will accept a target as one parameter to proceed target which means that all clusters are tried to merge with the target cluster. Each thread block is responsible for trying to merge one certain cluster with the target cluster and determining whether they can be merged. If the merge succeeds, set the corresponding component in the \texttt{valid} vector into 0.


\section{Performance Evaluation}

Configurations of GPU used for evalution are listed in Table~\ref{2}: 

\begin{table}[!htbp]
	\centering
	\caption{Configurations of GPU used for evalution}
	\label{2}
\begin{tabular}{|l|l|}
	\hline
	device name & Tesla K10.G2.8GB \\
	\hline
	totalGlobalMem & 3757637632 \\
	\hline
	sharedMemPerBlock & 49152 \\
	\hline
	regsPerBlock & 65536 \\
	\hline
	warpSize & 32 \\
	\hline
	memPitch & 2147483647 \\
	\hline
	maxThreadsPerBlock & 1024 \\
	\hline
	maxThreadsDim[3] & $1024 \times 1024 \times 64$ \\
	\hline
	maxGridSize[3] & $2147483647 \times 65535 \times 65535$ \\
	\hline
	totalConstMem & 65536 \\
	\hline
	device version & major 3, minor 0 \\
	\hline
	clockRate & 745000 \\
	\hline
	textureAlignment & 512 \\
	\hline
	deviceOverlap & 1 \\
	\hline
	multiProcessorCount & 8 \\
	\hline
\end{tabular}
\end{table}

\subsection{Evalution for distance calculation}

\begin{table}[!htbp]
	\centering
	\caption{Evalution for distance calculation with data size of 23040}
	\label{3}
	\begin{tabular}{|l|p{1cm}|p{1cm}|p{1.2cm}|}
	\hline
	kernel & Execution time/ms & Step Speedup & Cumulative Speedup \\
	\hline
	Baseline & 1482.24 & - & - \\
	\hline
	Use memory coalescing & 48.128 & 30.798 & 30.798 \\
	\hline
	Use shared memory & 23.867 & 2.0166 & 62.104 \\
	\hline
	Use loop unrolling & 5.3037 & 4.5000 & 279.47 \\
	\hline
\end{tabular}
\end{table}

From Table~\ref{3}, we can see that the performance improvement brought by memory coalescing is huge: Before we use memory coalescing, each thread of one warp has very far and different address while visiting the global memory so that we has to divide into 32 memory transaction to proceed. After we use memory coalescing, one wrap only needs one memory transaction. In theory, the performance has been improved 32 times. The test result shows that the performance is improved by 30.798 times which is very close to the theory number. Although loop unrolling does not reduce the actual calculating work, the performance improvement brought by it is very good. The complier can carry on static scheduling on the code and reduce the cost of loop control so that it will reduce the effect brought by pipeline blocking.

\subsection{Evaultion for building and merging primitive clusters}

\begin{table}[!htbp]
	\centering
	\caption{Evalution for building and merging primitive clusters with data size of 23040}
	\label{4}
	\begin{tabular}{|p{6cm}|p{1cm}|}
	\hline
	Kernel & Execution time/ms \\
	\hline
	Distance calculation (use unrolling loop) & 5.3037 \\
	\hline
	primitive clusters construction (use shared memory) & 44.848 \\
	\hline
	Distance calculation and primitive clusters construction at the same time & 25.336 \\
	\hline
	Put the iteration code outside & 12.502 \\
	\hline
	Merge clusters & 22.48 \\
	\hline
\end{tabular}
\end{table}

From Table~\ref{4}, we can see that if we separate the process of distance calculation and building primitive clusters, the time will be 50.151ms. But if we proceed the distance calculation and primitive clusters building at the same time, the time is only 25.336ms. The results of both methods are completely identical. It shows that half of the 50.151ms is for visiting global memory.

Besides, before loop code is put outside, it will need 8 floating point calculation to get one element of each \texttt{cluster} matrix. But after loop code is put outsides, the time is reduced to 6 floating point calculation. It is related to compiling results. Before loop code is put outsides, \emph{nvcc} compile about 442 instructions and 342 instructions after that. So we can get that the way of changing distance calculation has make \emph{nvcc} compile some more compact instructions.

Cluster merging is not particular ideal. The time consuming is equal to the serial algorithm. Most of the time is for data transmission.

\subsection{Overall Evalution}

\begin{table*}[t]
	\centering
	\caption{Overall evalution with different data size}
	\label{5}
\begin{tabular}{|p{1.2cm}|p{2.3cm}|p{2.3cm}|p{2.3cm}|p{2.2cm}|p{2.5cm}|p{2cm}|}
	\hline
	Data size & CPU time (distance calculation + primitive clusters construction)/ms & GPU time (distance calculation + primitive clusters construction)/ms & Speedup (distance calculation + primitive clusters construction) & CPU total time/ms & GPU total time (With data transmission)/ms & Overall Speedup \\
	\hline
	5061 & 130 & 0.8142 & 159.67 & 170 & 45.12 & 3.7677 \\
	\hline
	23040 & 1790 & 9.9077 & 180.67 & 1810 & 32.37 & 55.916 \\
	\hline
	60032 & 12180 & 69.323 & 175.70 & 12210 & 124.73 & 97.891 \\
	\hline
\end{tabular}
\end{table*}

From Table~\ref{5}, we can see that distance calculation and building primitive clusters are both bottleneck steps in the serial algorithm. CUDA acceleration has given a great performance improvement. However, CUDA acceleration brings difficulty of cluster merging. 

\section{Discussion}

At first, our plan was as following. First, calculate the distance of each pair of points. Second, get the primitive clusters and generate the primitive Boolean conflict matrix. Thirdly, calculate the transitive closure of Boolean conflict matrix and get global cluster numbers which can be merged by the same cluster markers. Fourth, re-write the result according to the global cluster number. The key step is to calculate the transitive closure which can be realized by Warshall algorithm. So every step has a high degree of parallelism. The plan transforms the original problem into a pure mathematic problem and the degree of parallelism of Warshall algorithm is very high. This seems a pretty good plan. However, the experimental results are not so good. Because even if Warshall algorithm has a high degree of parallelism, each calculation needs a launch of kernel which costs 3ms. And Warshall algorithm needs start Kernel for $N$ (the number of points) times.

The relationship of cluster and point is stored by matrix. At the worst case, the store method will need $O(N^2)$ space. And the scale of 60000 has reached the space limitation. Further study is still needed before we find out algorithm with good performance and scalability.

\section{Conclusion}

In fact, a few key codes in the algorithm cost most of the time in the whole program. So the key of improving the performance of the algorithm lies in how to deal with these hotspot. The limits of DBSCAN is how to calculate the distance of each pair of points. We separate the calculate process into many small and un-related processes by analyzing the relevance of the process. Meanwhile, we modify the serial algorithm, get primitive clusters and then merge clusters. The relevance of merge process has been reduced and the parallelization of the algorithm is also improved.

Besides the parallelization brought by multiple cores, algorithm improvement based on the structure of GPU can dramatically speed up single thread computing. Much time has been reduced on memory coalescing, bank conflict and branch divergency by analyzing data structure, diving the calculate process and controlling the memory accessing operations. Some skills used in the serial algorithm are also suitable for the improvement of CUDA. We modify the comparison between the distance and the threshold, as well as unrolling the loops. These skills brings very good results.

According to computing results, the execution speed of our algorithm is 97.891 times higher than the serial algorithm of DBSCAN on the condition of a large data scale.

% An example of a floating figure using the graphicx package.
% Note that \label must occur AFTER (or within) \caption.
% For figures, \caption should occur after the \includegraphics.
% Note that IEEEtran v1.7 and later has special internal code that
% is designed to preserve the operation of \label within \caption
% even when the captionsoff option is in effect. However, because
% of issues like this, it may be the safest practice to put all your
% \label just after \caption rather than within \caption{}.
%
% Reminder: the "draftcls" or "draftclsnofoot", not "draft", class
% option should be used if it is desired that the figures are to be
% displayed while in draft mode.
%
%\begin{figure}[!t]
%\centering
%\includegraphics[width=2.5in]{myfigure}
% where an .eps filename suffix will be assumed under latex,
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{Simulation Results}
%\label{fig_sim}
%\end{figure}

% Note that IEEE typically puts floats only at the top, even when this
% results in a large percentage of a column being occupied by floats.


% An example of a double column floating figure using two subfigures.
% (The subfig.sty package must be loaded for this to work.)
% The subfigure \label commands are set within each subfloat command, the
% \label for the overall figure must come after \caption.
% \hfil must be used as a separator to get equal spacing.
% The subfigure.sty package works much the same way, except \subfigure is
% used instead of \subfloat.
%
%\begin{figure*}[!t]
%\centerline{\subfloat[Case I]\includegraphics[width=2.5in]{subfigcase1}%
%\label{fig_first_case}}
%\hfil
%\subfloat[Case II]{\includegraphics[width=2.5in]{subfigcase2}%
%\label{fig_second_case}}}
%\caption{Simulation results}
%\label{fig_sim}
%\end{figure*}
%
% Note that often IEEE papers with subfigures do not employ subfigure
% captions (using the optional argument to \subfloat), but instead will
% reference/describe all of them (a), (b), etc., within the main caption.


% An example of a floating table. Note that, for IEEE style tables, the
% \caption command should come BEFORE the table. Table text will default to
% \footnotesize as IEEE normally uses this smaller font for tables.
% The \label must come after \caption as always.
%
%\begin{table}[!t]
%% increase table row spacing, adjust to taste
%\renewcommand{\arraystretch}{1.3}
% if using array.sty, it might be a good idea to tweak the value of
% \extrarowheight as needed to properly center the text within the cells
%\caption{An Example of a Table}
%\label{table_example}
%\centering
%% Some packages, such as MDW tools, offer better commands for making tables
%% than the plain LaTeX2e tabular which is used here.
%\begin{tabular}{|c||c|}
%\hline
%One & Two\\
%\hline
%Three & Four\\
%\hline
%\end{tabular}
%\end{table}


% Note that IEEE does not put floats in the very first column - or typically
% anywhere on the first page for that matter. Also, in-text middle ("here")
% positioning is not used. Most IEEE journals/conferences use top floats
% exclusively. Note that, LaTeX2e, unlike IEEE journals/conferences, places
% footnotes above bottom floats. This can be corrected via the \fnbelowfloat
% command of the stfloats package.


% conference papers do not normally have an appendix


% use section* for acknowledgement



% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
%\IEEEtriggeratref{8}
% The "triggered" command can be changed if desired:
%\IEEEtriggercmd{\enlargethispage{-5in}}

% references section

% can use a bibliography generated by BibTeX as a .bbl file
% BibTeX documentation can be easily obtained at:
% http://www.ctan.org/tex-archive/biblio/bibtex/contrib/doc/
% The IEEEtran BibTeX style support page is at:
% http://www.michaelshell.org/tex/ieeetran/bibtex/
%\bibliographystyle{IEEEtran}
% argument is your BibTeX string definitions and bibliography database(s)
%\bibliography{IEEEabrv,../bib/paper}
%
% <OR> manually copy in the resultant .bbl file
% set second argument of \begin to the number of references
% (used to reserve space for the reference number labels box)
\begin{thebibliography}{1}

\bibitem{1}
Ester M, Kriegel H P, Sander J, et al., \emph{A density-based algorithm for discovering clusters in large spatial databases with noise}.\hskip 1em plus 0.5em minus 0.4em\relax Kdd. 1996, 96(34): 226-231.


\bibitem{2}
Patwary M M A, Palsetia D, Agrawal A, et al., \emph{A new scalable parallel DBSCAN algorithm using the disjoint-set data structure}.\hskip 1em plus 0.5em minus 0.4em\relax High Performance Computing, Networking, Storage and Analysis (SC), 2012 International Conference for. IEEE, 2012: 1-11.

\bibitem{3}
He Y, Tan H, Luo W, et al., \emph{MR-DBSCAN: an efficient parallel density-based clustering algorithm using mapreduce}.\hskip 1em plus 0.5em minus 0.4em\relax Parallel and Distributed Systems (ICPADS), 2011 IEEE 17th International Conference on. IEEE, 2011: 473-480.

\bibitem{4}
Nickolls J, Buck I, Garland M, et al., \emph{Scalable parallel programming with CUDA}.\hskip 1em plus 0.5em minus 0.4em\relax Queue, 2008, 6(2): 40-53.

\bibitem{5}
Lindholm E, Nickolls J, Oberman S, et al., \emph{NVIDIA Tesla: A unified graphics and computing architecture}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE Micro, 2008, 28(2): 39-55.

\bibitem{6}
Andrade G, Ramos G, Madeira D, et al., \emph{G-dbscan: A gpu accelerated algorithm for density-based clustering}.\hskip 1em plus 0.5em minus 0.4em\relax Procedia Computer Science, 2013, 18: 369-378.

\bibitem{7}
B\"ohm C, Noll R, Plant C, et al., \emph{Density-based clustering using graphics processors}.\hskip 1em plus 0.5em minus 0.4em\relax Proceedings of the 18th ACM conference on Information and knowledge management. ACM, 2009: 661-670.

\bibitem{8}
Loh W K, Yu H, \emph{Fast density-based clustering through dataset partition using graphics processing units}.\hskip 1em plus 0.5em minus 0.4em\relax Information Sciences, 2014.

\bibitem{9}
M. Ester, H.-P. Kriegel, J. Sander, et al., \emph{A density-based algorithm for discovering clusters in large spatial databases with noise}.\hskip 1em plus 0.5em minus 0.4em\relax Knowledge Discovery and Data Mining (KDD), Portland, Oregon, USA, 1996, pp. 226�C231.

\end{thebibliography}





% that's all folks
\end{document}


1506.02226.extracted_bib
{1 Ester M, Kriegel H P, Sander J, et al., A density-based algorithm for discovering clusters in large spatial databases with noise.\hskip 1em plus 0.5em minus 0.4em\relax Kdd. 1996, 96(34): 226-231.
1506.02226.extracted_bib
{2 Patwary M M A, Palsetia D, Agrawal A, et al., A new scalable parallel DBSCAN algorithm using the disjoint-set data structure.\hskip 1em plus 0.5em minus 0.4em\relax High Performance Computing, Networking, Storage and Analysis (SC), 2012 International Conference for. IEEE, 2012: 1-11.
1506.02226.extracted_bib
{3 He Y, Tan H, Luo W, et al., MR-DBSCAN: an efficient parallel density-based clustering algorithm using mapreduce.\hskip 1em plus 0.5em minus 0.4em\relax Parallel and Distributed Systems (ICPADS), 2011 IEEE 17th International Conference on. IEEE, 2011: 473-480.
1506.02226.extracted_bib
{4 Nickolls J, Buck I, Garland M, et al., Scalable parallel programming with CUDA.\hskip 1em plus 0.5em minus 0.4em\relax Queue, 2008, 6(2): 40-53.
1506.02226.extracted_bib
{5 Lindholm E, Nickolls J, Oberman S, et al., NVIDIA Tesla: A unified graphics and computing architecture.\hskip 1em plus 0.5em minus 0.4em\relax IEEE Micro, 2008, 28(2): 39-55.
1506.02226.extracted_bib
{6 Andrade G, Ramos G, Madeira D, et al., G-dbscan: A gpu accelerated algorithm for density-based clustering.\hskip 1em plus 0.5em minus 0.4em\relax Procedia Computer Science, 2013, 18: 369-378.
1506.02226.extracted_bib
{7 B\"ohm C, Noll R, Plant C, et al., Density-based clustering using graphics processors.\hskip 1em plus 0.5em minus 0.4em\relax Proceedings of the 18th ACM conference on Information and knowledge management. ACM, 2009: 661-670.
1506.02226.extracted_bib
{8 Loh W K, Yu H, Fast density-based clustering through dataset partition using graphics processing units.\hskip 1em plus 0.5em minus 0.4em\relax Information Sciences, 2014.
1506.02226.extracted_bib
{9 M. Ester, H.-P. Kriegel, J. Sander, et al., A density-based algorithm for discovering clusters in large spatial databases with noise.\hskip 1em plus 0.5em minus 0.4em\relax Knowledge Discovery and Data Mining (KDD), Portland, Oregon, USA, 1996, pp. 226�C231.