Reducing the Effects of Detrimental Instances
\IEEEauthorblockN{Michael R. Smith
\IEEEauthorblockN{Michael Shell\IEEEauthorrefmark{1
%\boldmath % It is widely known in the machine learning community that class noise can be (and often is) detrimental to inducing a model of the data. Not all instances in a data set are equally beneficial for inducing a model of the data. Some instances (such as outliers or noise) can be detrimental. However, at least initially, the instances in a data set are generally considered equally in machine learning algorithms. Many current approaches for handling noisy and detrimental instances make a binary decision about whether an instance is detrimental or not. In this paper, we 1) extend this paradigm by weighting the instances on a continuous scale and 2) present a methodology for measuring how detrimental an instance may be for inducing a model of the data. We call our method of identifying and weighting detrimental instances reduced detrimental instance learning (RDIL). We examine RDIL on a set of 54 data sets and 5 learning algorithms and compare RDIL with other weighting and filtering approaches. RDIL is especially useful for learning algorithms where every instance can affect the classification boundary and the training instances are considered individually, such as multilayer perceptrons trained with backpropagation (MLPs). Our results also suggest that a more accurate estimate of which instances are detrimental can have a significant positive impact for handling them.
classification2.tex
% GNUPLOT: LaTeX picture with Postscript
\begingroup
  \makeatletter
  \providecommand\color[2][]{%
    \GenericError{(gnuplot) \space\space\space\@spaces}{%
      Package color not loaded in conjunction with
      terminal option `colourtext'%
    }{See the gnuplot documentation for explanation.%
    }{Either use 'blacktext' in gnuplot or load the package
      color.sty in LaTeX.}%
    \renewcommand\color[2][]{}%
  }%
  \providecommand\includegraphics[2][]{%
    \GenericError{(gnuplot) \space\space\space\@spaces}{%
      Package graphicx or graphics not loaded%
    }{See the gnuplot documentation for explanation.%
    }{The gnuplot epslatex terminal needs graphicx.sty or graphics.sty.}%
    \renewcommand\includegraphics[2][]{}%
  }%
  \providecommand\rotatebox[2]{#2}%
  \@ifundefined{ifGPcolor}{%
    \newif\ifGPcolor
    \GPcolorfalse
  }{}%
  \@ifundefined{ifGPblacktext}{%
    \newif\ifGPblacktext
    \GPblacktexttrue
  }{}%
  % define a \g@addto@macro without @ in the name:
  \let\gplgaddtomacro\g@addto@macro
  % define empty templates for all commands taking text:
  \gdef\gplbacktext{}%
  \gdef\gplfronttext{}%
  \makeatother
  \ifGPblacktext
    % no textcolor at all
    \def\colorrgb#1{}%
    \def\colorgray#1{}%
  \else
    % gray or color?
    \ifGPcolor
      \def\colorrgb#1{\color[rgb]{#1}}%
      \def\colorgray#1{\color[gray]{#1}}%
      \expandafter\def\csname LTw\endcsname{\color{white}}%
      \expandafter\def\csname LTb\endcsname{\color{black}}%
      \expandafter\def\csname LTa\endcsname{\color{black}}%
      \expandafter\def\csname LT0\endcsname{\color[rgb]{1,0,0}}%
      \expandafter\def\csname LT1\endcsname{\color[rgb]{0,1,0}}%
      \expandafter\def\csname LT2\endcsname{\color[rgb]{0,0,1}}%
      \expandafter\def\csname LT3\endcsname{\color[rgb]{1,0,1}}%
      \expandafter\def\csname LT4\endcsname{\color[rgb]{0,1,1}}%
      \expandafter\def\csname LT5\endcsname{\color[rgb]{1,1,0}}%
      \expandafter\def\csname LT6\endcsname{\color[rgb]{0,0,0}}%
      \expandafter\def\csname LT7\endcsname{\color[rgb]{1,0.3,0}}%
      \expandafter\def\csname LT8\endcsname{\color[rgb]{0.5,0.5,0.5}}%
    \else
      % gray
      \def\colorrgb#1{\color{black}}%
      \def\colorgray#1{\color[gray]{#1}}%
      \expandafter\def\csname LTw\endcsname{\color{white}}%
      \expandafter\def\csname LTb\endcsname{\color{black}}%
      \expandafter\def\csname LTa\endcsname{\color{black}}%
      \expandafter\def\csname LT0\endcsname{\color{black}}%
      \expandafter\def\csname LT1\endcsname{\color{black}}%
      \expandafter\def\csname LT2\endcsname{\color{black}}%
      \expandafter\def\csname LT3\endcsname{\color{black}}%
      \expandafter\def\csname LT4\endcsname{\color{black}}%
      \expandafter\def\csname LT5\endcsname{\color{black}}%
      \expandafter\def\csname LT6\endcsname{\color{black}}%
      \expandafter\def\csname LT7\endcsname{\color{black}}%
      \expandafter\def\csname LT8\endcsname{\color{black}}%
    \fi
  \fi
  \setlength{\unitlength}{0.0500bp}%
%  \begin{picture}(3300.00,2500.00)%
  \begin{picture}(2000.00,1507.00)%
    \gplgaddtomacro\gplbacktext{%
      \csname LTb\endcsname%
      \put(230,1250){\makebox(0,0)[r]{\strut{} \textbf A}}%
      \put(1980,200){\makebox(0,0)[r]{\strut{} \textbf B}}%
%      \put(300,2225){\makebox(0,0)[r]{\strut{} \textbf{\large A}}}%
%      \put(2980,300){\makebox(0,0)[r]{\strut{} \textbf{\large B}}}%
%      \put(4690,2193){\makebox(0,0)[r]{\strut{} \textbf{\large C}}}%
%      \put(5700,267){\makebox(0,0)[r]{\strut{} \textbf{\large D}}}%
%      \put(1210,3042){\makebox(0,0)[r]{\strut{} 80}}%
%      \put(1210,3516){\makebox(0,0)[r]{\strut{} 100}}%
%      \put(1767,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0$}}}%
%      \put(2192,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$> 0$}}}%
%      \put(2618,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.1$}}}%
%      \put(3043,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.2$}}}%
%      \put(3468,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.3$}}}%
%      \put(3893,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.4$}}}%
%      \put(4319,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.5$}}}%
%      \put(4744,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.6$}}}%
%      \put(5169,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.7$}}}%
%      \put(5594,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.8$}}}%
%      \put(6020,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.9$}}}%
%      \put(6445,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$= 1$}}}%
%      \put(440,2330){\rotatebox{90}{\makebox(0,0){\strut{}\% of Instances}}}%
%      \put(4106,154){\makebox(0,0){\strut{}Instance Hardness}}%
    }%
%    \gplgaddtomacro\gplfronttext{%
%      \csname LTb\endcsname%
%      \put(5883,3343){\makebox(0,0)[r]{\strut{}UCI}}%
%      \csname LTb\endcsname%
%      \put(5883,3123){\makebox(0,0)[r]{\strut{}non-UCI}}%
%    }%
    \gplbacktext
    \put(0,0){\includegraphics[scale=0.125]{classification2}}%
    \gplfronttext
  \end{picture}%
\endgroup

classification.tex
% GNUPLOT: LaTeX picture with Postscript
\begingroup
  \makeatletter
  \providecommand\color[2][]{%
    \GenericError{(gnuplot) \space\space\space\@spaces}{%
      Package color not loaded in conjunction with
      terminal option `colourtext'%
    }{See the gnuplot documentation for explanation.%
    }{Either use 'blacktext' in gnuplot or load the package
      color.sty in LaTeX.}%
    \renewcommand\color[2][]{}%
  }%
  \providecommand\includegraphics[2][]{%
    \GenericError{(gnuplot) \space\space\space\@spaces}{%
      Package graphicx or graphics not loaded%
    }{See the gnuplot documentation for explanation.%
    }{The gnuplot epslatex terminal needs graphicx.sty or graphics.sty.}%
    \renewcommand\includegraphics[2][]{}%
  }%
  \providecommand\rotatebox[2]{#2}%
  \@ifundefined{ifGPcolor}{%
    \newif\ifGPcolor
    \GPcolorfalse
  }{}%
  \@ifundefined{ifGPblacktext}{%
    \newif\ifGPblacktext
    \GPblacktexttrue
  }{}%
  % define a \g@addto@macro without @ in the name:
  \let\gplgaddtomacro\g@addto@macro
  % define empty templates for all commands taking text:
  \gdef\gplbacktext{}%
  \gdef\gplfronttext{}%
  \makeatother
  \ifGPblacktext
    % no textcolor at all
    \def\colorrgb#1{}%
    \def\colorgray#1{}%
  \else
    % gray or color?
    \ifGPcolor
      \def\colorrgb#1{\color[rgb]{#1}}%
      \def\colorgray#1{\color[gray]{#1}}%
      \expandafter\def\csname LTw\endcsname{\color{white}}%
      \expandafter\def\csname LTb\endcsname{\color{black}}%
      \expandafter\def\csname LTa\endcsname{\color{black}}%
      \expandafter\def\csname LT0\endcsname{\color[rgb]{1,0,0}}%
      \expandafter\def\csname LT1\endcsname{\color[rgb]{0,1,0}}%
      \expandafter\def\csname LT2\endcsname{\color[rgb]{0,0,1}}%
      \expandafter\def\csname LT3\endcsname{\color[rgb]{1,0,1}}%
      \expandafter\def\csname LT4\endcsname{\color[rgb]{0,1,1}}%
      \expandafter\def\csname LT5\endcsname{\color[rgb]{1,1,0}}%
      \expandafter\def\csname LT6\endcsname{\color[rgb]{0,0,0}}%
      \expandafter\def\csname LT7\endcsname{\color[rgb]{1,0.3,0}}%
      \expandafter\def\csname LT8\endcsname{\color[rgb]{0.5,0.5,0.5}}%
    \else
      % gray
      \def\colorrgb#1{\color{black}}%
      \def\colorgray#1{\color[gray]{#1}}%
      \expandafter\def\csname LTw\endcsname{\color{white}}%
      \expandafter\def\csname LTb\endcsname{\color{black}}%
      \expandafter\def\csname LTa\endcsname{\color{black}}%
      \expandafter\def\csname LT0\endcsname{\color{black}}%
      \expandafter\def\csname LT1\endcsname{\color{black}}%
      \expandafter\def\csname LT2\endcsname{\color{black}}%
      \expandafter\def\csname LT3\endcsname{\color{black}}%
      \expandafter\def\csname LT4\endcsname{\color{black}}%
      \expandafter\def\csname LT5\endcsname{\color{black}}%
      \expandafter\def\csname LT6\endcsname{\color{black}}%
      \expandafter\def\csname LT7\endcsname{\color{black}}%
      \expandafter\def\csname LT8\endcsname{\color{black}}%
    \fi
  \fi
  \setlength{\unitlength}{0.0500bp}%
%  \begin{picture}(3300.00,2500.00)%
  \begin{picture}(2000.00,1507.00)%
    \gplgaddtomacro\gplbacktext{%
      \csname LTb\endcsname%
      \put(250,1300){\makebox(0,0)[r]{\strut{} \textbf A}}%
      \put(1980,250){\makebox(0,0)[r]{\strut{} \textbf B}}%
%      \put(300,2300){\makebox(0,0)[r]{\strut{} \textbf{\large A}}}%
%      \put(2980,400){\makebox(0,0)[r]{\strut{} \textbf{\large B}}}%
%      \put(4690,2193){\makebox(0,0)[r]{\strut{} \textbf{\large C}}}%
%      \put(5700,267){\makebox(0,0)[r]{\strut{} \textbf{\large D}}}%
%      \put(1210,3042){\makebox(0,0)[r]{\strut{} 80}}%
%      \put(1210,3516){\makebox(0,0)[r]{\strut{} 100}}%
%      \put(1767,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0$}}}%
%      \put(2192,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$> 0$}}}%
%      \put(2618,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.1$}}}%
%      \put(3043,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.2$}}}%
%      \put(3468,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.3$}}}%
%      \put(3893,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.4$}}}%
%      \put(4319,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.5$}}}%
%      \put(4744,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.6$}}}%
%      \put(5169,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.7$}}}%
%      \put(5594,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.8$}}}%
%      \put(6020,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$\geq 0.9$}}}%
%      \put(6445,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}$= 1$}}}%
%      \put(440,2330){\rotatebox{90}{\makebox(0,0){\strut{}\% of Instances}}}%
%      \put(4106,154){\makebox(0,0){\strut{}Instance Hardness}}%
    }%
%    \gplgaddtomacro\gplfronttext{%
%      \csname LTb\endcsname%
%      \put(5883,3343){\makebox(0,0)[r]{\strut{}UCI}}%
%      \csname LTb\endcsname%
%      \put(5883,3123){\makebox(0,0)[r]{\strut{}non-UCI}}%
%    }%
    \gplbacktext
    \put(0,0){\includegraphics[scale=0.125]{classification}}%
    \gplfronttext
  \end{picture}%
\endgroup

COD.tex
% GNUPLOT: LaTeX picture with Postscript
%auto-ignore
\begingroup
  \makeatletter
  \providecommand\color[2][]{%
    \GenericError{(gnuplot) \space\space\space\@spaces}{%
      Package color not loaded in conjunction with
      terminal option `colourtext'%
    }{See the gnuplot documentation for explanation.%
    }{Either use 'blacktext' in gnuplot or load the package
      color.sty in LaTeX.}%
    \renewcommand\color[2][]{}%
  }%
  \providecommand\includegraphics[2][]{%
    \GenericError{(gnuplot) \space\space\space\@spaces}{%
      Package graphicx or graphics not loaded%
    }{See the gnuplot documentation for explanation.%
    }{The gnuplot epslatex terminal needs graphicx.sty or graphics.sty.}%
    \renewcommand\includegraphics[2][]{}%
  }%
  \providecommand\rotatebox[2]{#2}%
  \@ifundefined{ifGPcolor}{%
    \newif\ifGPcolor
    \GPcolorfalse
  }{}%
  \@ifundefined{ifGPblacktext}{%
    \newif\ifGPblacktext
    \GPblacktexttrue
  }{}%
  % define a \g@addto@macro without @ in the name:
  \let\gplgaddtomacro\g@addto@macro
  % define empty templates for all commands taking text:
  \gdef\gplbacktext{}%
  \gdef\gplfronttext{}%
  \makeatother
  \ifGPblacktext
    % no textcolor at all
    \def\colorrgb#1{}%
    \def\colorgray#1{}%
  \else
    % gray or color?
    \ifGPcolor
      \def\colorrgb#1{\color[rgb]{#1}}%
      \def\colorgray#1{\color[gray]{#1}}%
      \expandafter\def\csname LTw\endcsname{\color{white}}%
      \expandafter\def\csname LTb\endcsname{\color{black}}%
      \expandafter\def\csname LTa\endcsname{\color{black}}%
      \expandafter\def\csname LT0\endcsname{\color[rgb]{1,0,0}}%
      \expandafter\def\csname LT1\endcsname{\color[rgb]{0,1,0}}%
      \expandafter\def\csname LT2\endcsname{\color[rgb]{0,0,1}}%
      \expandafter\def\csname LT3\endcsname{\color[rgb]{1,0,1}}%
      \expandafter\def\csname LT4\endcsname{\color[rgb]{0,1,1}}%
      \expandafter\def\csname LT5\endcsname{\color[rgb]{1,1,0}}%
      \expandafter\def\csname LT6\endcsname{\color[rgb]{0,0,0}}%
      \expandafter\def\csname LT7\endcsname{\color[rgb]{1,0.3,0}}%
      \expandafter\def\csname LT8\endcsname{\color[rgb]{0.5,0.5,0.5}}%
    \else
      % gray
      \def\colorrgb#1{\color{black}}%
      \def\colorgray#1{\color[gray]{#1}}%
      \expandafter\def\csname LTw\endcsname{\color{white}}%
      \expandafter\def\csname LTb\endcsname{\color{black}}%
      \expandafter\def\csname LTa\endcsname{\color{black}}%
      \expandafter\def\csname LT0\endcsname{\color{black}}%
      \expandafter\def\csname LT1\endcsname{\color{black}}%
      \expandafter\def\csname LT2\endcsname{\color{black}}%
      \expandafter\def\csname LT3\endcsname{\color{black}}%
      \expandafter\def\csname LT4\endcsname{\color{black}}%
      \expandafter\def\csname LT5\endcsname{\color{black}}%
      \expandafter\def\csname LT6\endcsname{\color{black}}%
      \expandafter\def\csname LT7\endcsname{\color{black}}%
      \expandafter\def\csname LT8\endcsname{\color{black}}%
    \fi
  \fi
  \setlength{\unitlength}{0.0500bp}%
%  \begin{picture}(7200.00,3080.00)%
   \begin{picture}(6000.00,3200.00)%
    \gplgaddtomacro\gplfronttext{%
      \csname LTb\endcsname%
      \put(560,1094){\rotatebox{90}{\makebox(0,0)[r]{\strut{} 0.10}}}%
      \put(560,1550){\rotatebox{90}{\makebox(0,0)[r]{\strut{} 0.15}}}%
      \put(560,2002){\rotatebox{90}{\makebox(0,0)[r]{\strut{} 0.20}}}%
      \put(560,2450){\rotatebox{90}{\makebox(0,0)[r]{\strut{} 0.25}}}%
      \put(560,2890){\rotatebox{90}{\makebox(0,0)[r]{\strut{} 0.30}}}%
      \put(850,862){\rotatebox{90}{\makebox(0,0)[r]{\strut{}BayesNet}}}%
      \put(1070,862){\rotatebox{90}{\makebox(0,0)[r]{\strut{}DecTable}}}%
      \put(1300,1112){\rotatebox{90}{\makebox(0,0)[r]{\strut{}RIPPER}}}%
      \put(1480,1112){\rotatebox{90}{\makebox(0,0)[r]{\strut{}SimpleCart}}}%
      \put(1710,1600){\rotatebox{90}{\makebox(0,0)[r]{\strut{}LWL}}}%
      \put(1880,1232){\rotatebox{90}{\makebox(0,0)[r]{\strut{}Functional Tree}}}%
      \put(2100,1000){\rotatebox{90}{\makebox(0,0)[r]{\strut{}Logistic}}}%
      \put(2340,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}SMV}}}%
      \put(2525,1075){\rotatebox{90}{\makebox(0,0)[r]{\strut{}MLP}}}%
      \put(2750,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}NB}}}%
      \put(2950,1000){\rotatebox{90}{\makebox(0,0)[r]{\strut{}RBF Network}}}%
      \put(3170,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}1-NN}}}%
      \put(3375,1012){\rotatebox{90}{\makebox(0,0)[r]{\strut{}5-NN}}}%
      \put(3550,1490){\rotatebox{90}{\makebox(0,0)[r]{\strut{}NNge}}}%
      \put(3765,1062){\rotatebox{90}{\makebox(0,0)[r]{\strut{}C4.5}}}%
      \put(4000,1062){\rotatebox{90}{\makebox(0,0)[r]{\strut{}PART}}}%
      \put(4180,1262){\rotatebox{90}{\makebox(0,0)[r]{\strut{}LADTree}}}%
      \put(4380,1142){\rotatebox{90}{\makebox(0,0)[r]{\strut{}NBTree}}}%
      \put(4595,1142){\rotatebox{90}{\makebox(0,0)[r]{\strut{}RandForest}}}%
      \put(4800,1522){\rotatebox{90}{\makebox(0,0)[r]{\strut{}Ridor}}}%
      \put(190,1780){\rotatebox{90}{\makebox(0,0){\strut{}Classifier Output Difference}}}%
    }%
    \gplbacktext
    \put(0,150){\includegraphics[scale=0.65]{COD}}%
    \gplfronttext
  \end{picture}%
\endgroup

iCMLA2014_rev1.tex

%% bare_conf.tex
%% V1.3
%% 2007/01/11
%% by Michael Shell
%% See:
%% http://www.michaelshell.org/
%% for current contact information.
%%
%% This is a skeleton file demonstrating the use of IEEEtran.cls
%% (requires IEEEtran.cls version 1.7 or later) with an IEEE conference paper.
%%
%% Support sites:
%% http://www.michaelshell.org/tex/ieeetran/
%% http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/
%% and
%% http://www.ieee.org/

%%*************************************************************************
%% Legal Notice:
%% This code is offered as-is without any warranty either expressed or
%% implied; without even the implied warranty of MERCHANTABILITY or
%% FITNESS FOR A PARTICULAR PURPOSE! 
%% User assumes all risk.
%% In no event shall IEEE or any contributor to this code be liable for
%% any damages or losses, including, but not limited to, incidental,
%% consequential, or any other damages, resulting from the use or misuse
%% of any information contained here.
%%
%% All comments are the opinions of their respective authors and are not
%% necessarily endorsed by the IEEE.
%%
%% This work is distributed under the LaTeX Project Public License (LPPL)
%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used,
%% distributed and modified. A copy of the LPPL, version 1.3, is included
%% in the base LaTeX documentation of all distributions of LaTeX released
%% 2003/12/01 or later.
%% Retain all contribution notices and credits.
%% ** Modified files should be clearly indicated as such, including  **
%% ** renaming them and changing author support contact information. **
%%
%% File list of work: IEEEtran.cls, IEEEtran_HOWTO.pdf, bare_adv.tex,
%%                    bare_conf.tex, bare_jrnl.tex, bare_jrnl_compsoc.tex
%%*************************************************************************

% *** Authors should verify (and, if needed, correct) their LaTeX system  ***
% *** with the testflow diagnostic prior to trusting their LaTeX platform ***
% *** with production work. IEEE's font choices can trigger bugs that do  ***
% *** not appear when using other class files.                            ***
% The testflow support page is at:
% http://www.michaelshell.org/tex/testflow/



% Note that the a4paper option is mainly intended so that authors in
% countries using A4 can easily print to A4 and see how their papers will
% look in print - the typesetting of the document will not typically be
% affected with changes in paper size (but the bottom and side margins will).
% Use the testflow package mentioned above to verify correct handling of
% both paper sizes by the user's LaTeX system.
%
% Also note that the "draftcls" or "draftclsnofoot", not "draft", option
% should be used if it is desired that the figures are to be displayed in
% draft mode.
%
\documentclass[10pt, conference, compsocconf]{IEEEtran}
% Add the compsoc option for Computer Society conferences.
%
% If IEEEtran.cls has not been installed into the LaTeX system files,
% manually specify the path to it like:
% \documentclass[conference]{../sty/IEEEtran}





% Some very useful LaTeX packages include:
% (uncomment the ones you want to load)


% *** MISC UTILITY PACKAGES ***
%
%\usepackage{ifpdf}
% Heiko Oberdiek's ifpdf.sty is very useful if you need conditional
% compilation based on whether the output is pdf or dvi.
% usage:
% \ifpdf
%   % pdf code
% \else
%   % dvi code
% \fi
% The latest version of ifpdf.sty can be obtained from:
% http://www.ctan.org/tex-archive/macros/latex/contrib/oberdiek/
% Also, note that IEEEtran.cls V1.7 and later provides a builtin
% \ifCLASSINFOpdf conditional that works the same way.
% When switching from latex to pdflatex and vice-versa, the compiler may
% have to be run twice to clear warning/error messages.






% *** CITATION PACKAGES ***
%
%\usepackage{cite}
% cite.sty was written by Donald Arseneau
% V1.6 and later of IEEEtran pre-defines the format of the cite.sty package
% \cite{} output to follow that of IEEE. Loading the cite package will
% result in citation numbers being automatically sorted and properly
% "compressed/ranged". e.g., [1], [9], [2], [7], [5], [6] without using
% cite.sty will become [1], [2], [5]--[7], [9] using cite.sty. cite.sty's
% \cite will automatically add leading space, if needed. Use cite.sty's
% noadjust option (cite.sty V3.8 and later) if you want to turn this off.
% cite.sty is already installed on most LaTeX systems. Be sure and use
% version 4.0 (2003-05-27) and later if using hyperref.sty. cite.sty does
% not currently provide for hyperlinked citations.
% The latest version can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/cite/
% The documentation is contained in the cite.sty file itself.






% *** GRAPHICS RELATED PACKAGES ***
%
\ifCLASSINFOpdf
  % \usepackage[pdftex]{graphicx}
  % declare the path(s) where your graphic files are
  % \graphicspath{{../pdf/}{../jpeg/}}
  % and their extensions so you won't have to specify these with
  % every instance of \includegraphics
  % \DeclareGraphicsExtensions{.pdf,.jpeg,.png}
\else
  % or other class option (dvipsone, dvipdf, if not using dvips). graphicx
  % will default to the driver specified in the system graphics.cfg if no
  % driver is specified.
  % \usepackage[dvips]{graphicx}
  % declare the path(s) where your graphic files are
  % \graphicspath{{../eps/}}
  % and their extensions so you won't have to specify these with
  % every instance of \includegraphics
  % \DeclareGraphicsExtensions{.eps}
\fi
% graphicx was written by David Carlisle and Sebastian Rahtz. It is
% required if you want graphics, photos, etc. graphicx.sty is already
% installed on most LaTeX systems. The latest version and documentation can
% be obtained at: 
% http://www.ctan.org/tex-archive/macros/latex/required/graphics/
% Another good source of documentation is "Using Imported Graphics in
% LaTeX2e" by Keith Reckdahl which can be found as epslatex.ps or
% epslatex.pdf at: http://www.ctan.org/tex-archive/info/
%
% latex, and pdflatex in dvi mode, support graphics in encapsulated
% postscript (.eps) format. pdflatex in pdf mode supports graphics
% in .pdf, .jpeg, .png and .mps (metapost) formats. Users should ensure
% that all non-photo figures use a vector format (.eps, .pdf, .mps) and
% not a bitmapped formats (.jpeg, .png). IEEE frowns on bitmapped formats
% which can result in "jaggedy"/blurry rendering of lines and letters as
% well as large increases in file sizes.
%
% You can find documentation about the pdfTeX application at:
% http://www.tug.org/applications/pdftex





% *** MATH PACKAGES ***
%
\usepackage[cmex10]{amsmath}
% A popular package from the American Mathematical Society that provides
% many useful and powerful commands for dealing with mathematics. If using
% it, be sure to load this package with the cmex10 option to ensure that
% only type 1 fonts will utilized at all point sizes. Without this option,
% it is possible that some math symbols, particularly those within
% footnotes, will be rendered in bitmap form which will result in a
% document that can not be IEEE Xplore compliant!
%
% Also, note that the amsmath package sets \interdisplaylinepenalty to 10000
% thus preventing page breaks from occurring within multiline equations. Use:
%\interdisplaylinepenalty=2500
% after loading amsmath to restore such page breaks as IEEEtran.cls normally
% does. amsmath.sty is already installed on most LaTeX systems. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/required/amslatex/math/





% *** SPECIALIZED LIST PACKAGES ***
%
%\usepackage{algorithmic}
% algorithmic.sty was written by Peter Williams and Rogerio Brito.
% This package provides an algorithmic environment fo describing algorithms.
% You can use the algorithmic environment in-text or within a figure
% environment to provide for a floating algorithm. Do NOT use the algorithm
% floating environment provided by algorithm.sty (by the same authors) or
% algorithm2e.sty (by Christophe Fiorio) as IEEE does not use dedicated
% algorithm float types and packages that provide these will not provide
% correct IEEE style captions. The latest version and documentation of
% algorithmic.sty can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/algorithms/
% There is also a support site at:
% http://algorithms.berlios.de/index.html
% Also of interest may be the (relatively newer and more customizable)
% algorithmicx.sty package by Szasz Janos:
% http://www.ctan.org/tex-archive/macros/latex/contrib/algorithmicx/




% *** ALIGNMENT PACKAGES ***
%
%\usepackage{array}
% Frank Mittelbach's and David Carlisle's array.sty patches and improves
% the standard LaTeX2e array and tabular environments to provide better
% appearance and additional user controls. As the default LaTeX2e table
% generation code is lacking to the point of almost being broken with
% respect to the quality of the end results, all users are strongly
% advised to use an enhanced (at the very least that provided by array.sty)
% set of table tools. array.sty is already installed on most systems. The
% latest version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/required/tools/


%\usepackage{mdwmath}
%\usepackage{mdwtab}
% Also highly recommended is Mark Wooding's extremely powerful MDW tools,
% especially mdwmath.sty and mdwtab.sty which are used to format equations
% and tables, respectively. The MDWtools set is already installed on most
% LaTeX systems. The lastest version and documentation is available at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/mdwtools/


% IEEEtran contains the IEEEeqnarray family of commands that can be used to
% generate multiline equations as well as matrices, tables, etc., of high
% quality.


%\usepackage{eqparbox}
% Also of notable interest is Scott Pakin's eqparbox package for creating
% (automatically sized) equal width boxes - aka "natural width parboxes".
% Available at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/eqparbox/





% *** SUBFIGURE PACKAGES ***
%\usepackage[tight,footnotesize]{subfigure}
% subfigure.sty was written by Steven Douglas Cochran. This package makes it
% easy to put subfigures in your figures. e.g., "Figure 1a and 1b". For IEEE
% work, it is a good idea to load it with the tight package option to reduce
% the amount of white space around the subfigures. subfigure.sty is already
% installed on most LaTeX systems. The latest version and documentation can
% be obtained at:
% http://www.ctan.org/tex-archive/obsolete/macros/latex/contrib/subfigure/
% subfigure.sty has been superceeded by subfig.sty.



%\usepackage[caption=false]{caption}
%\usepackage[font=footnotesize]{subfig}
% subfig.sty, also written by Steven Douglas Cochran, is the modern
% replacement for subfigure.sty. However, subfig.sty requires and
% automatically loads Axel Sommerfeldt's caption.sty which will override
% IEEEtran.cls handling of captions and this will result in nonIEEE style
% figure/table captions. To prevent this problem, be sure and preload
% caption.sty with its "caption=false" package option. This is will preserve
% IEEEtran.cls handing of captions. Version 1.3 (2005/06/28) and later 
% (recommended due to many improvements over 1.2) of subfig.sty supports
% the caption=false option directly:
%\usepackage[caption=false,font=footnotesize]{subfig}
%
% The latest version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/subfig/
% The latest version and documentation of caption.sty can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/caption/




% *** FLOAT PACKAGES ***
%
%\usepackage{fixltx2e}
% fixltx2e, the successor to the earlier fix2col.sty, was written by
% Frank Mittelbach and David Carlisle. This package corrects a few problems
% in the LaTeX2e kernel, the most notable of which is that in current
% LaTeX2e releases, the ordering of single and double column floats is not
% guaranteed to be preserved. Thus, an unpatched LaTeX2e can allow a
% single column figure to be placed prior to an earlier double column
% figure. The latest version and documentation can be found at:
% http://www.ctan.org/tex-archive/macros/latex/base/



%\usepackage{stfloats}
% stfloats.sty was written by Sigitas Tolusis. This package gives LaTeX2e
% the ability to do double column floats at the bottom of the page as well
% as the top. (e.g., "\begin{figure*}[!b]" is not normally possible in
% LaTeX2e). It also provides a command:
%\fnbelowfloat
% to enable the placement of footnotes below bottom floats (the standard
% LaTeX2e kernel puts them above bottom floats). This is an invasive package
% which rewrites many portions of the LaTeX2e float routines. It may not work
% with other packages that modify the LaTeX2e float routines. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/sttools/
% Documentation is contained in the stfloats.sty comments as well as in the
% presfull.pdf file. Do not use the stfloats baselinefloat ability as IEEE
% does not allow \baselineskip to stretch. Authors submitting work to the
% IEEE should note that IEEE rarely uses double column equations and
% that authors should try to avoid such use. Do not be tempted to use the
% cuted.sty or midfloat.sty packages (also by Sigitas Tolusis) as IEEE does
% not format its papers in such ways.





% *** PDF, URL AND HYPERLINK PACKAGES ***
%
%\usepackage{url}
% url.sty was written by Donald Arseneau. It provides better support for
% handling and breaking URLs. url.sty is already installed on most LaTeX
% systems. The latest version can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/misc/
% Read the url.sty source comments for usage information. Basically,
% \url{my_url_here}.





% *** Do not adjust lengths that control margins, column widths, etc. ***
% *** Do not use packages that alter fonts (such as pslatex).         ***
% There should be no need to do such things with IEEEtran.cls V1.6 and later.
% (Unless specifically asked to do so by the journal or conference you plan
% to submit to, of course. )


% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}
\newcommand{\argmax}{\operatornamewithlimits{argmax}}
\newcommand{\argmin}{\operatornamewithlimits{argmin}}  
\usepackage{color,graphicx}
% \usepackage{amsthm}
\usepackage{bbm}
\usepackage{multirow}
\usepackage{rotating}
\usepackage[table]{xcolor}
\usepackage{hhline,longtable,booktabs}
\usepackage{pifont}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%

\begin{document}
%
% paper title
% can use linebreaks \\ within to get better formatting as desired
\title{Reducing the Effects of Detrimental Instances}


% author names and affiliations
% use a multiple column layout for up to three different
% affiliations
\author{\IEEEauthorblockN{Michael R. Smith}
\IEEEauthorblockA{Department of Computer Science\\
Brigham Young University\\
Provo, UT 84602\\
Email: msmith@axon.cs.byu.edu}
\and
\IEEEauthorblockN{Tony Martinez}
\IEEEauthorblockA{Department of Computer Science\\
Brigham Young University\\
Provo, UT 84602\\
Email: martinez@cs.byu.edu}}

% conference papers do not typically use \thanks and this command
% is locked out in conference mode. If really needed, such as for
% the acknowledgment of grants, issue a \IEEEoverridecommandlockouts
% after \documentclass

% for over three affiliations, or if they all won't fit within the width
% of the page, use this alternative format:
% 
%\author{\IEEEauthorblockN{Michael Shell\IEEEauthorrefmark{1},
%Homer Simpson\IEEEauthorrefmark{2},
%James Kirk\IEEEauthorrefmark{3}, 
%Montgomery Scott\IEEEauthorrefmark{3} and
%Eldon Tyrell\IEEEauthorrefmark{4}}
%\IEEEauthorblockA{\IEEEauthorrefmark{1}School of Electrical and Computer Engineering\\
%Georgia Institute of Technology,
%Atlanta, Georgia 30332--0250\\ Email: see http://www.michaelshell.org/contact.html}
%\IEEEauthorblockA{\IEEEauthorrefmark{2}Twentieth Century Fox, Springfield, USA\\
%Email: homer@thesimpsons.com}
%\IEEEauthorblockA{\IEEEauthorrefmark{3}Starfleet Academy, San Francisco, California 96678-2391\\
%Telephone: (800) 555--1212, Fax: (888) 555--1212}
%\IEEEauthorblockA{\IEEEauthorrefmark{4}Tyrell Inc., 123 Replicant Street, Los Angeles, California 90210--4321}}




% use for special paper notices
%\IEEEspecialpapernotice{(Invited Paper)}




% make the title area
\maketitle


\begin{abstract}
%\boldmath
% It is widely known in the machine learning community that class noise can be (and often is) detrimental to inducing a model of the data.
Not all instances in a data set are equally beneficial for inducing a model of the data.
Some instances (such as outliers or noise) can be detrimental.
However, at least initially, the instances in a data set are generally considered equally in machine learning algorithms.
Many current approaches for handling noisy and detrimental instances make a binary decision about whether an instance is detrimental or not.
In this paper, we 1) extend this paradigm by weighting the instances on a continuous scale and 2) present a methodology for measuring how detrimental an instance may be for inducing a model of the data.
We call our method of identifying and weighting detrimental instances \textit{reduced detrimental instance learning} (\textit{RDIL}).
We examine RDIL on a set of 54 data sets and 5 learning algorithms and compare RDIL with other weighting and filtering approaches.
RDIL is especially useful for learning algorithms where every instance can affect the classification boundary and the training instances are considered individually, such as multilayer perceptrons trained with backpropagation (MLPs).
Our results also suggest that a more accurate estimate of which instances are detrimental can have a significant positive impact for handling them.
\end{abstract}
% IEEEtran.cls defaults to using nonbold math in the Abstract.
% This preserves the distinction between vectors and scalars. However,
% if the conference you are submitting to favors bold math in the abstract,
% then you can use LaTeX's standard command \boldmath at the very start
% of the abstract to achieve this. Many IEEE journals/conferences frown on
% math in the abstract anyway.

% no keywords




% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
\IEEEpeerreviewmaketitle



\section{Introduction}
The goal of supervised machine learning is to induce an accurate generalizing function from a set of labeled training instances.
Given that in most cases, all that is known about a task is contained in the set of training instances, at least initially, the instances in a data set are generally considered equally.
However, some instances are more beneficial than others for inducing a model of the data. % that is not taken into account during learning.
For example, outliers or mislabeled instances are not as beneficial as border instances and can even be detrimental in many cases.
In addition, other instances can be detrimental for inducing a model of the data even if they are labeled correctly and are not outliers.

% However, most real-world data sets are noisy.
% % Generally, two types of noise are considered: attribute noise and label noise.
% % Previous work has found that label noise is more harmful than attribute noise \cite{Zhu2004,Nettleton2010}.
% Knowing which instances are noisy is non-trivial as, in most cases, all that is known about a task is contained in the set of training instances.
% Thus, at least initially, the instances in a data set (including the noisy instances) are generally considered equally.
% Some instances can be more beneficial than other instances for inducing a model of the data.
% For example, outliers and mislabeled instances are not as beneficial as border instances and can even be detrimental in many cases.

% One of the difficulties in dealing with noise in a data set is that there is no agreed-upon definition of what constitutes a noisy instance.
% For example, Hickey defines noise as an instance instance that ``obscures the relationship between description and class'' \cite{Hickey1996} while Quinlan defines noise as ``non-systematic errors'' \cite{Quinlan1986}.
% There is also the notion of distinguishing between noise and exceptions (which is a valid instance but it may appear to be noisy).

A possible effect of considering all instances equally, including the detrimental instances, when inducing a model of the data is shown in the hypothetical two-dimensional data set in Figure \ref{figure:example}a.
% Instances A and B adversely affect the induced classification boundary.
The solid line represents the ``actual'' classification boundary and the dashed line represents a potential induced classification boundary.
Instances A and B are detrimental instances that ``pull'' the decision boundary away from the true boundary and cause the instances in the space between the true boundary and induced boundary to be misclassified.
A learning algorithm can more precisely model the data by considering instances differently during training to suppress the effects of detrimental training instances.

This is especially true for learning algorithms such as backpropagation for training multilayer perceptrons (MLP).
Detrimental instances (e.g. instance A) have the greatest effect on the classification boundary since they can have the largest error value.
As shown by Elman \cite{Elman1993}, this is particularly important during the early stages of training a MLP when the initial gradient is calculated.
Elman proposes a method to initially train the MLP with simpler instances then gradually increase to more complex instances.
This procedure has developed into curriculum learning and has had success in deep learning \cite{Bengio2009}.
However, it is not as successful for shallow MLPs \cite{Smith_CL}.
The impact of detrimental instances is lessened in other other learning algorithms since the influence of a particular instance is localized.
For example, \textit{k}-NN only considers the \textit{k} nearest neighbors of an instance.

Assuming that all that is known about a task is contained in the training set, how detrimental an instance is for inducing a model of the data can be estimated based on its relationship with the other instances in the data set.
For example, instance A from Figure \ref{figure:example}a represents a detrimental instance as an outlier--being in a region with instances of a differing class.
In contrast, instance A in the data set shown in Figure \ref{figure:example}b is not as detrimental given additional instances of the same class in the same region.
% Using the information in the instance relationships, we weight each training instance by the likelihood that it will be misclassified.
% % Thus, in this paper, we estimate how noisy an instance is based on how likely it is to be misclassified by a learning algorithm..
% 
As determining if an instance is detrimental exhibits a degree of uncertainty, we examine weighting the instances in a data set by their likelihood of being misclassified.
Instance A from Figure \ref{figure:example}a, for example, has a high likelihood of being misclassified while instance B may have a lower likelihood of being misclassified.
Weighting the instances limits the influence of an instance proportionate to its detrimentality measure.
We present a theoretically-motivated methodology for estimating the likelihood that an instance will be misclassified that lessens the dependence on any one model.
We call this approach of weighting the instances by their probability of being misclassified \textit{reduced detrimental instance learning} (\textit{RDIL}).
% Using the likelihood that an instance will be misclassified to determine how detrimental an instance is allows a learning algorithm to focus more on the instances that are representative of their respective classes without discarding instances.
Filtering or removing detrimental instances prior to training can be viewed as a special case of instance weighting.
We show that both filtering and weighting are viable solutions and examine when each is most beneficial.

We examine RDIL on a set of 5 learning algorithms and 54 data sets.
We compare multiple versions of RDIL with another weighting scheme, pair-wise expectation maximization (PWEM) \cite{Rebbapragada2007}, as well as several filtering algorithms: misclassification filters and an ensemble filter \cite{Brodley1999}, and repeated-edited nearest-neighbor (RENN) \cite{Tomek1976}.
% We also compare RDIL with a voting ensemble.
We find that some learning algorithms benefit more from filtering and others from instance weighting.
Specifically, filtering is more beneficial for decision trees, rule-based learners, and nearest neighbor algorithms.
Weighting the instances has a more significant impact on multilayer perceptrons.
In cases with high amounts of noise, weighting the instances is generally preferable to filtering as it generally achieves higher accuracy and does not require a threshold to be set if filtering is based on a continuous value.
% Our findings can be summarized as follows:
% \begin{itemize}
%  \item \textcolor{blue}{obtaining an accurate estimate of the likelihood that an instance will be misclassified has the most significant effect on the efficacy of filtering and instance weighting.
%  \item some learning algorithms benefit more from filtering and others from instance weighting.
%  Specifically, filtering is more beneficial for decision trees, rule-based learners, and nearest neighbor algorithms.
%  Weighting the instances has a more significant impact on multilayer perceptrons.}
% %  \item when examined over a large number of data sets, noise handling rarely improves the classification accuracy unless there is a substantial amount of noise. In some cases, noise handling significantly decreases the classification accuracy.
% %  \item RDIL significantly outperforms PWEM in most cases.}
%  \item weighting the instances is generally preferable to filtering as it generally achieves higher accuracy and does not require a threshold to be set \textcolor{blue}{if filtering is based on a continuous value.}
% % %  \item RDIL typically outperforms PWEM leading to the observation that a better estimate of the likelihood of an instance being misclassified improves weighting and filtering.
% %  \item a voting ensemble achieves significantly higher average classification accuracy than directly handling noise.
% %  \item using RDIL, PWEM, or filtering with a voting ensemble significantly increases the average classification accuracy only when there is a substantial amount of noise.
% \end{itemize}

% We find that RDIL significantly increases the classification accuracy over not using RDIL for every learning algorithm.
% The most significant improvement in classification accuracy is achieved for learning algorithms where every instance can affect the classification boundary and that consider instances individually rather than collectively such as multilayer perceptrons trained with backpropagation (MLPs) and random forests.
% MLPs increase the average accuracy from 80.7\% to 84.9\% and random forests increase the average accuracy from 81.6\% to 85.1\%.
% Also, RDIL achieves a significantly higher accuracy than a voting ensemble and boosting.
% We also examine RDIL in and with voting ensembles and in boosting.
% In a voting ensemble, RDIL does not significantly affect the classification accuracy.
% RDIL increases the classification accuracy over a voting ensemble in multilayer perceptrons and random forests.
% A voting ensemble achieves a classification accuracy of 83.3\%.
% The highest classification accuracy is achieved when combined with boosting.
% Boosting a multilayer perceptron and random forests with RDIL increases classification accuracy to 84.9\% and 85.1\% respectively.

The remainder of the paper is organized as follows.
Section \ref{section:relatedWorks} reviews related work in handling noise. 
Section \ref{section:math} motivates weighting the instances.
The detrimentality measure is presented in Section \ref{section:estimatingP}.
Our experimental methodology is presented in Section \ref{section:methodology}.
The results of RDIL are provided in Section \ref{section:results}.
% Section \ref{section:method} presents the experimental methodology.
% The results are presented in Section \ref{section:results}.
Section \ref{section:conclusions} concludes the paper.

\begin{figure}
\centering
\begin{tabular}{|c|c|}
\hline
\input{classification}  & 
% \hline
% \multicolumn{1}{c}{a}\\
% \multicolumn{1}{c}{} \\
\input{classification2}\\
\hline
\multicolumn{1}{c}{a}&\multicolumn{1}{c}{b}\\
 %\multicolumn{1}{c}{b} \\
\end{tabular}
\caption{Hypothetical, 2-dimensional data set with two detrimental instances (instances A and B) that shows a) that treating all instances equally in a data set with detrimental instances can adversely affect the classification boundary and b) that how detrimental an instance may be is dependent on the other instances in a data set.}
\label{figure:example}
\end{figure}


% An example of a floating figure using the graphicx package.
% Note that \label must occur AFTER (or within) \caption.
% For figures, \caption should occur after the \includegraphics.
% Note that IEEEtran v1.7 and later has special internal code that
% is designed to preserve the operation of \label within \caption
% even when the captionsoff option is in effect. However, because
% of issues like this, it may be the safest practice to put all your
% \label just after \caption rather than within \caption{}.
%
% Reminder: the "draftcls" or "draftclsnofoot", not "draft", class
% option should be used if it is desired that the figures are to be
% displayed while in draft mode.
%
%\begin{figure}[!t]
%\centering
%\includegraphics[width=2.5in]{myfigure}
% where an .eps filename suffix will be assumed under latex, 
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{Simulation Results}
%\label{fig_sim}
%\end{figure}

% Note that IEEE typically puts floats only at the top, even when this
% results in a large percentage of a column being occupied by floats.


% An example of a double column floating figure using two subfigures.
% (The subfig.sty package must be loaded for this to work.)
% The subfigure \label commands are set within each subfloat command, the
% \label for the overall figure must come after \caption.
% \hfil must be used as a separator to get equal spacing.
% The subfigure.sty package works much the same way, except \subfigure is
% used instead of \subfloat.
%
%\begin{figure*}[!t]
%\centerline{\subfloat[Case I]\includegraphics[width=2.5in]{subfigcase1}%
%\label{fig_first_case}}
%\hfil
%\subfloat[Case II]{\includegraphics[width=2.5in]{subfigcase2}%
%\label{fig_second_case}}}
%\caption{Simulation results}
%\label{fig_sim}
%\end{figure*}
%
% Note that often IEEE papers with subfigures do not employ subfigure
% captions (using the optional argument to \subfloat), but instead will
% reference/describe all of them (a), (b), etc., within the main caption.


% An example of a floating table. Note that, for IEEE style tables, the 
% \caption command should come BEFORE the table. Table text will default to
% \footnotesize as IEEE normally uses this smaller font for tables.
% The \label must come after \caption as always.
%
%\begin{table}[!t]
%% increase table row spacing, adjust to taste
%\renewcommand{\arraystretch}{1.3}
% if using array.sty, it might be a good idea to tweak the value of
% \extrarowheight as needed to properly center the text within the cells
%\caption{An Example of a Table}
%\label{table_example}
%\centering
%% Some packages, such as MDW tools, offer better commands for making tables
%% than the plain LaTeX2e tabular which is used here.
%\begin{tabular}{|c||c|}
%\hline
%One & Two\\
%\hline
%Three & Four\\
%\hline
%\end{tabular}
%\end{table}


% Note that IEEE does not put floats in the very first column - or typically
% anywhere on the first page for that matter. Also, in-text middle ("here")
% positioning is not used. Most IEEE journals/conferences use top floats
% exclusively. Note that, LaTeX2e, unlike IEEE journals/conferences, places
% footnotes above bottom floats. This can be corrected via the \fnbelowfloat
% command of the stfloats package.

\section{Related Work}
\label{section:relatedWorks}
Many real-wold data sets contain detrimental instances that arise from noise (typos, measurement errors, etc.) or from the stochastic nature of the task.
Noise is a subset of detrimental instances.
Previous work has examined how class noise and attribute noise affects the performance of various learning algorithms~\cite{Nettleton2010}. % \cite{Zhu2004,Nettleton2010}.
They found that class noise is generally more harmful than attribute noise.
The consequences of class noise, as summarized by Fr\'{e}nay and Verleysen \cite{Frenay2014}, include 1) a deterioration of classification performance, 2) increased learning requirements and model complexity, and 3) a distortion of observed frequencies.
% These works found that class noise is generally more harmful than attribute noise and that noise in the training set is more harmful that noise in the test set.
% For a thorough overview of the current state of handling label noise in classification problems, consult the survey by Fr\'{e}nay and Verleysen \cite{Frenay2014}.

Most learning algorithms are designed to tolerate a certain degree of detrimental instances by making a trade-off between the complexity of the induced model and minimizing error on the training data to prevent overfit.
For example, to avoid overfit many algorithms use a validation set for early stopping and/or regularization by adding a complexity penalty to the loss function \cite{bishop2006pattern}.
% Despite these precautions, most learning algorithms are not completely robust to noise in the sense that the probability of an instance being misclassified changes with the presence of noise \cite{Frenay2014,Manwani2013}.
Some learning algorithms have been adapted specifically to better handle noise.
Boosting algorithms \cite{Schapire1990}, for example, assign more weight to misclassified instances--which often include mislabeled and noisy instances.
To address this, Servedio \cite{Servedio2003} presented a boosting algorithm that does not place too much weight on any training instance. 
% For support vector machines, Collobert et al. \cite{Collobert2006} use the ramp-loss function to place a bound on the maximum penalty for an instance that lies on the wrong side of the margin.
% Lawrence and Sch\"{o}lkopf \cite{Lawrence2001} explicitly model the possibility that an instance is mislabeled using a generative model and then use expectation maximization to update the probability that an instance is mislabeled.

Preprocessing the data set explicitly handles detrimental instances by removing, weighting, or correcting them.
% All three approaches first attempt to identify which instances are noisy by various criteria.
Filtering detrimental instances has received much attention and has generally been shown to result in an increase in classification accuracy, especially when there are large amounts of noise \cite{Gamberger2000,Smith2011}.
% \textcolor{blue}{Although, recently the broad application of noise handling techniques has been shown to not always increase the classification accuracy, especially when no artificial noise has been added to the data set \cite{Smith_Eval}.
% Saez et al.~\cite{Saez2013} also showed that filtering can be detrimental and further examined predicting the efficacy of filtering in nearest neighbor classification.
% They showed that there is some correlation between data set properties and the efficacy of filtering.}
One frequently used filtering technique removes any instance that is misclassified by a learning algorithm \cite{John95} or set of learning algorithms \cite{Brodley1999}.
% Verbaeten and Van Assche \cite{Verbaeten2003} further pursued the idea of using an ensemble for filtering using ideas from boosting and bagging.
Other approaches use information theoretic or machine learning heuristics to remove noisy instances.
Segata et al. \cite{Segata2009}, for example, remove instances that are too close or on the wrong side of the decision surface generated by a support vector machine.
% Zeng and Martinez \cite{Xinchuan2003} remove instances while training a neural network that have a low probability of being labeled correctly where the probability is calculated using the output from the neural network.
However, filtering has the potential downside of discarding useful instances and/or too many instances.
% However, it is assumed that there are significantly more non-noisy instances and that throwing away a few correct instances with the noisy instances will not have a negative impact on a large data set.

Rather than making a binary decision about the detrimentality of in instance, weighting allows a continuous scale.
Filtering can be considered a special case of weighting where each instance is assigned a weight of 0 or 1.
Rebbapragada and Brodley \cite{Rebbapragada2007} weight the instances using expectation maximization to cluster instances that belong to a pair of the classes.
The probabilities between classes for each instance is compiled and used to weight the influence of each instance.
% In this paper, we also examine weighting--weighting the instances based on their probability of being misclassified.

Data cleaning does not discard any instances, but rather strives to correct the noise in the instances.
As in filtering, the output from a learning algorithm has been used to clean the data.
% Automatic data enhancement \cite{zeng.ida2001} uses the output from a neural network to correct the label for training instances that have a low probability of being correctly labeled.
Polishing \cite{Teng2003} trains a learning algorithm (in this case a decision tree) to predict the value for each attribute (including the class).
The predicted (i.e.~corrected) attribute values for the instances that increase generalization accuracy on a validation set are used instead of the uncleaned attribute values.

% Most previous work has only considered the effects of a proposed noise handling technique on a small number of data sets and learning algorithms--often only using a single learning algorithm.
% Some studies have examined a larger set, but it is more of the exception rather than the rule \cite{Folleco2009,Khoshgoftaar2010}.
% In this paper, we examine the effects of noise on a set of 54 data sets and 5 diverse learning algorithms.

\section{Modeling Detrimentality}
\label{section:math}

Lawrence and Sch\"{o}lkopf \cite{Lawrence2001} model a data set using a generative model that also models the noise.
Let $T$ be a training set composed of instances $\langle x_i, \hat{y}_i\rangle$ drawn i.i.d. from the underlying data distribution $\mathcal{D}$.
Each instance has an associated latent random variable/feature $y_i$.
Thus, $x_i$ is the set of input features, $\hat{y}_i$ is the possibly noisy class label given in the training set, and $y_i$ is the true unknown class label.
Lawrence and Sch\"{o}lkopf assume that the joint distribution $p(x_i,y_i,\hat{y}_i)$ is factorized as $p(\hat{y}_i|y_i)p(x_i|y_i)p(y_i)$ as shown in Figure \ref{figure:generativeModel}a.
Since modeling the prior distribution of the unobserved random variable $y_i$ is not feasible, they estimate the prior distribution of $p(\hat{y}_i)$ with some assumptions about the noise as shown in Figure \ref{figure:generativeModel}b.

\begin{figure*}
\begin{center}
\begin{tabular}{|c|c||c|c|}
\hline
\includegraphics[scale=0.85]{figure1.eps} & \includegraphics[scale=0.85]{figure2.eps} & \includegraphics[scale=0.85]{figure3.eps} & \includegraphics[scale=0.85]{figure4.eps} \\
\hline
\multicolumn{1}{c}{a} & \multicolumn{1}{c}{b}& \multicolumn{1}{c}{c}& \multicolumn{1}{c}{d}\\
\end{tabular}
\caption{Graphical model of a a) generative probabilistic model, b) the generative model proposed by Lawrence and Sch\"{o}lkopf \cite{Lawrence2001} and a discriminative probabilistic model for c) $p(\hat{y}|x)p(x)$ and d) $p(\hat{y}|x,y)p(y|x)p(x)$.}
\label{figure:generativeModel}
\end{center}
\end{figure*}

Following the premise of Lawrence and Sch\"{o}lkopf, we explicitly model the possibility that an instance is mislabeled (i.e. $y \neq \hat{y}$).
Rather than using a generative model, though, we use a discriminative model since we are focusing on classification tasks and do not require the full joint distribution.
Also, discriminative models have been shown to yield better performance on classification tasks \cite{Ng+Jordan:2001}.
A generative model, which models the full joint distribution of $p(x, y, \hat{y})$, differs from a discriminative model which is mostly concerned with modeling the likelihood of the class: $p(\hat{y}|x)$.
% 
% \begin{figure*}[t]
% \begin{center}
% \begin{tabular}{|c|c|}
% \hline
% \includegraphics{figure3.eps} & \includegraphics{figure4.eps} \\
% \hline
% \multicolumn{1}{c}{a} & \multicolumn{1}{c}{b}\\
% \end{tabular}
% \caption{}
% \label{figure:discriminativeModel}
% \end{center}
% \end{figure*}
% 
% A classification task can be formalized as follows. %is composed of an input vector $x_i$ with a corresponding possibly noisy label vector $\hat{y}_i$.
Given a training set $T$, a discriminative model generally seeks to find the most probable hypothesis $h$ that maps each $x_i \mapsto \hat{y}_i$--ignoring the fact that $\hat{y}_i$ may not equal $y_i$.
This is shown graphically in Figure \ref{figure:generativeModel}c where $p(\hat{y}_i | x_i) p(x_i)$ is estimated using a discriminative approach such as a neural network or a decision tree to induce a hypothesis of the data.
The possibility of label noise is not explicitly modeled in this form (i.e. $p(y_i)$ is ignored).
Label noise is generally handled by avoiding overfit such that more probable, simpler hypotheses are preferred.

Label noise can be more explicitly modeled by considering that $\hat{y}_i$ may be noisy by associating a latent random variable $y_i$ with each instance. % $\langle x_i,\hat{y}_i \rangle$.
In this context, a supervised learning algorithm seeks to maximize $p(\hat{y}_i|x_i,y_i)p(y_i|x_i)p(x_i)$--modeled graphically in Figure \ref{figure:generativeModel}d.
This factorization of the likelihood for the observed class label for an instance suggests that it should be weighted by the probability that $y_i$ is the actual class.
Thus, when considering the possibility that $\hat{y}_i \neq y_i$, it is natural to weight the instances by $p(y_i|x_i)$.
This provides the motivation for \textit{reduced detrimental instance learning} (RDIL).
RDIL takes two passes through the data set.
In the first pass, $p(y_i|x_i)$ is calculated.
Next, a learning algorithm is trained with the training instances weighted by $p(y_i|x_i)$.
However, calculating $p(y_i|x_i)$ is not trivial.
A method to estimate $p(y_i|x_i)$ is described in the following section.

% Using Bayes' rule and decomposing $T$ into its individual constituent instances, the maximum a posteriori (MAP) hypothesis is:
% \begin{align}
% \argmax_{h \in \mathcal{H}} p(h|T)&=\frac{p(T|h)p(h)}{p(T)} \nonumber\\
% &=\prod_i p(x_i,\hat{y}_i, y_i|h)p(h)\label{eq:1}
% % &=\prod_i p(\hat{y}_i|x_i,h) p(x_i|h)p(h). \label{eq:1}
% \end{align}
% In Equation \ref{eq:1}, the MAP hypothesis $h$ is found by finding a global optima where all instances are included in the optimization problem including the detrimental instances.
% % However, noisy instances are often detrimental for finding the global optima since they are not representative of the true (and unknown) underlying data distribution $\mathcal{D}$.
% 
% Thus, an instance is the triplet $\langle x_i, \hat{y}_i, y_i\rangle$ and a supervised learning algorithm seeks to maximize $p(\hat{y}_i|x_i,y_i)p(y_i|x_i)p(x_i)$--modeled graphically in Figure \ref{figure:generativeModel}d.
% Using the model in Figure \ref{figure:generativeModel}d, the MAP hypothesis becomes:
% \begin{align}
% \argmax_{h \in \mathcal{H}} p(h|&T)=\prod_i p(x_i,y_i,\hat{y}_i|h)p(h)\nonumber\\
% &=\prod_i p(\hat{y}_i|x_i,y_i,h)p(y_i|x_i,h) p(x_i|h)p(h).\label{eq:noise}
% \end{align}
% Equation \ref{eq:noise} shows that for an instance, the probability of an observed class label ($p(\hat{y}_i|x_i,y_i,h)$) should be weighted by the probability that $y_i$ is the actual class ($p(y_i|x_i,h)$).
% Thus, when considering the possibility that $\hat{y}_i \neq y_i$, it is natural to weight the instances by $p(y_i|x_i,h)$ and provides the motivation for \textit{reduced detrimental instance learning} (RDIL).
% RDIL weights each instance by $p(y_i|x_i,h)$ to explicitly model class noise in a discriminative model. 

% Using a discriminative model $h$ trained on $T$, the quantity $p(y_i|x_i,h)$ can be approximated as $p(\hat{y}_i|x_i,h)$ assuming that $p(y_i|\hat{y}_i)$ is represented in $h$.
% % This will indicate the likelihood that $\hat{y}_i$ is the correct class.
% % It is assumed that $p(\hat{y}_i|y_i)$ is presented in $h$.
% In other words, the induced discriminative model is able to model if one class label is more likely than another class label given an observed noisy label.
% Otherwise, all class labels are assumed to be equally likely given an observed label.
% Thus, $p(y_i|x_i,h)$ can be approximated by finding the class distributions for a given $x_i$ from an induced discriminative model.
% That is, after training a learning algorithm on $T$, the class distribution for an instance $x_i$ can be calculated based on the output from the learning algorithm.
% As shown in Equation \ref{eq:1}, $p(\hat{y}_i|x_i,h)$ is found naturally through a derivation of Bayes' law.
% The quantity $p(\hat{y}_i|x_i,h)$ is the likelihood of $\hat{y}_i$ given a set of input features $x_i$--which a learning algorithm tries to maximize for each instance.
% 


\section{Estimating $p(y_i|x_i)$}
\label{section:estimatingP}
% \textcolor{blue}{The first step in any noise handling technique is to identify noisy instances.
% As discussed in Section \ref{section:relatedWorks}, a variety of approaches have been taken to remove noisy instances, often using heuristics or biases from current learning algorithms or information theoretic measures.
% The identification of noisy and detrimental instances is the most important step in maximizing the efficacy of handling noise.}

In this paper, $p(y_i|x_i)$ is used as the detrimentality measure for each training instance.
In general, $p(y_i|x_i)$ does not make sense outside the context of an induced hypothesis.
Thus, using an induced hypothesis $h$ from a learning algorithm trained on $T$, the quantity $p(y_i|x_i,h)$ can be approximated as $p(\hat{y}_i|x_i,h)$ assuming that $p(y_i|\hat{y}_i)$ is represented in $h$.
% This will indicate the likelihood that $\hat{y}_i$ is the correct class.
% It is assumed that $p(\hat{y}_i|y_i)$ is presented in $h$.
In other words, the induced discriminative model is able to model if one class label is more likely than another class label given an observed noisy label.
% Otherwise, all class labels are assumed to be equally likely given an observed label.
% Thus, $p(y_i|x_i,h)$ can be approximated by finding the class distributions for a given $x_i$ from an induced discriminative model.
After training a learning algorithm on $T$, the class distribution for an instance $x_i$ can be calculated based on the output from the learning algorithm.


The dependence of $p(y_i|x_i)$ on a particular hypothesis $h$ can be removed by summing over all possible hypotheses $h$ in $\mathcal{H}$ and multiplying each $p(\hat{y}_i|x_i,h)$ by $p(h)$:
\begin{equation}
p(y_i|x_i)\approx p(\hat{y}_i|x_i) = \sum_{h\in\mathcal{H}} p(\hat{y}_i|x_i,h)p(h). \label{eq:sumOverH}
\end{equation}
This formulation is infeasible, though, because 1) it is not practical (or possible) to sum over the set of all hypotheses, 2) calculating $p(h)$ is non-trivial, and 3) not all learning algorithms produce a probability distribution.
These limitations make probabilistic generative models attractive, such as the kernel Fisher discriminant algorithm \cite{Lawrence2001}.
However, for classification tasks, discriminative models generally have a lower asymptotic error than generative models \cite{Ng+Jordan:2001}.
% The methodology used to calculate $p(y_i|x_i)$ is described in the following section.

% The quantity $p(y_i|x_i,h)$ is estimated as $p(\hat{y}_i|x_i,h)$ as discussed in the previous section.
% To lessen the dependence of the detrimentality measure $p(\hat{y}_i|x_i)$ on a specific hypothesis, 
In this paper we approximate $p(\hat{y}_i|x_i)$ using a diverse subset of $\mathcal{H}$.
The diversity of the subset of $\mathcal{H}$ refers to a set of hypotheses that differ in their classification of novel instances.
The diverse subset of $\mathcal{H}$ is created using unsupervised meta-learning (UML) \cite{Lee2011}.
UML first uses classifier output difference (COD) \cite{Peterson2005} to measure the diversity between learning algorithms.
COD measures the distance between two learning algorithms as the probability that the learning algorithms make different predictions.
UML then clusters the learning algorithms based on their COD scores with hierarchical agglomerative clustering.
We considered 20 learning algorithms from Weka with their default parameters \cite{weka2009}.
The resulting dendrogram is shown in Figure \ref{figure:COD}, where the height of the line connecting two clusters corresponds to the distance (COD value) between them.
A cut-point of 0.18 was chosen to create 9 clusters and a representative algorithm from each cluster was chosen to create a diverse set of $\mathcal{H}$.
Other numbers of clusters could have been used.
The learning algorithms that are used to estimate $p(\hat{y}_i|x_i)$ are listed in Table \ref{table:LA}.

\begin{figure}
\begin{center}
\vskip -2mm
\input{COD.tex}

\caption{Dendrogram of the considered learning algorithms clustered using unsupervised metalearning based on their classifier output difference.}
\label{figure:COD}
\end{center}
\end{figure}

\begin{table}
\caption{The diverse set of algorithms (as determined by unsupervised meta-learning) used to estimate $p(y_i|x_i)$.}
\label{table:LA}
\centering
% \setlength{\tabcolsep}{3pt}
\begin{tabular}{ll}
\hline
\multicolumn{2}{c}{Learning Algorithms}\\
\hline
*& Multilayer Perceptron trained with Back Propagation (MLP) \\
*& Decision Tree (C4.5) \\
*& Locally Weighted Learning (LWL) \\
*& 5-Nearest Neighbors (5-NN) \\
*& Nearest Neighbor with generalization (NNge) \\
*& Na\"{i}ve Bayes (NB) \\
*& RIpple DOwn Rule learner (RIDOR) \\
*& Random Forest (Random Forest) \\
*& Repeated Incremental Pruning to Produce Error Reduction (RIPPER) \\
\end{tabular}
\end{table}

$p(\hat{y}_i|x_i)$ is estimated for each training instance using 10-fold cross-validation (the instance $\langle x_, \hat{y}_i \rangle$ is \textit{not} used to induce the hypothesis $h$).
Using a set of diverse hypotheses induced by the learning algorithms in $\mathcal{L}$, we approximate $p(\hat{y}_i|x_i)$ as:
\begin{align}
p(\hat{y}_i|x_i) \approx p(\hat{y}_i|x_i,\mathcal{L}) 
&= \frac{1}{|\mathcal{L}|} \sum_{j=1}^{|\mathcal{L}|} p(\hat{y}_i|x_i, l_j(T)) \label{eq:final}
\end{align}
where $l_j(T)$ is the hypothesis induced by the $j^{th}$ learning algorithm trained on $T$.
From Equation \ref{eq:sumOverH}, $p(h)$ is estimated as $\frac{1}{|\mathcal{L}|}$ for the hypotheses induced by the learning algorithms in $\mathcal{L}$ and as zero for all of the other hypotheses in $\mathcal{H}$.
% Each instance is then weighted by $p(\hat{y}_i|x_i,\mathcal{L})$  as estimated in Equation \ref{eq:final}.
% To calculate $p(\hat{y}_i|x_i,\mathcal{L})$, we examine using 1) all 9 learning algorithms listed in Table \ref{table:LA}; 2) using MLP, random forest, and RIPPER; and 3) only using a random forest.
% These subsets of learning algorithms were chosen based on having a relatively high correlation with $p(\hat{y}_i|x_i,\mathcal{L})$ \cite{Smith2012_IH}.
% except when only a single learning algorithm is used.
% To estimate $p(\hat{y}_i|x_i)$ for a random forest, an induced random forest returns the class counts from the leaf node of each tree in the forest that classifies an instance.
% The counts for each class are then summed together and normalized between 0 and 1.

\section{Methodology}
\label{section:methodology}
% \textcolor{blue}{In this section, we present our methodology for weighting the instances by our estimate of $p(y_i|x_i)$ into each considered learning algorithm as well as our experimental methodology.}
% Once $p(\hat{y}_i|x_i,\mathcal{L})$ is calculated for each training instance, the learning algorithms then take the weighting into account during training.
We investigate the effects of filtering and weighting on the C4.5, 5NN, MLP, Random Forest, and RIPPER learning algorithms (abbreviated in Table \ref{table:LA}). 
Table \ref{table:QW} summarizes how an instance is weighted by $p(y_i|x_i)$ for the examined learning algorithms.
For MLPs trained with backpropagation, the error ($(t-o)f'(net)$) is scaled by $p(y_i|x_i)$ where $(t-o)$ is the difference between the target value and the output of the network, $f'(net)$ is the derivative of the activation function $f$ and $net$ is the sum of the product of each input $i_j$ and its corresponding weight $w_j$: $net=\sum_j w_ji_j$.
For Random Forests, the distribution for selecting instances in the random trees is weighted by $p(y_i|x_i)$ rather than being uniformly weighted. % refers to how bagging samples the instances.
For the other learning algorithms that keep track of counts, each instance is weighted by $p(y_i|x_i)$.
$\sum_{c_i}$ represents summing over instances that meet some criterion $c_i$ and $\sum_T$ sums over all of the instances in the data set.
% \textcolor{blue}{For the SVM, the $C$ parameter is scaled by $p(y_i|x_i)$.
% The SVM makes a greater attempt to correctly classify instances with larger $p(y_i|x_i)$ values.}

\begin{table}[t]
\centering
\caption{How instance weighting is integrated into the considered learning algorithms.}
\setlength{\tabcolsep}{2pt}
\begin{tabular}{l|c|c}
LA & Orig & RDIL\\
\hline
MLP & $(t-o)f'(net)$ & $p(y_i|x_i)(t-o)f'(net)$\\
\hline
Random Forest & Uniform dist & Weighted by $p(y_i|x_i)$\\
\hline
% SVM & $\operatornamewithlimits{min}\limits_{w,\xi,b}\frac{1}{2}\|w\|^2 + C\sum\limits_{i=1}^n \xi_i$ & $\operatornamewithlimits{min}\limits_{w,\xi,b}1/2\|w\|^2 + p(y_i|x_i)C\sum\limits_{i=1}^n \xi_i$\\
% \hline
C4.5,& Count number of instances, i.e.  & Sum $p(y_i|x_i)$ \\
5-NN, & \multirow{2}{*}{$\frac{\sum_{c_i} 1}{\sum_T 1}$} & \multirow{2}{*}{$\frac{\sum_{c_i} p(y_i|x_i)}{\sum_T p(y_i|x_i)}$}\\
RIPPER&  & \\
& &\\
% \hline
% SMV & $\thinmuskip=0mu\argmin\limits_{{\bf w}, \xi, b} \left\{\frac{1}{2}\vert\vert{\bf w}\vert\vert^2 + C \sum^n\limits_{i=1} \xi_i\right\}$& $\thinmuskip=0mu\argmin\limits_{{\bf w}, \xi, b} \left\{\frac{1}{2}\vert\vert{\bf w}\vert\vert^2 + C \sum^n\limits_{i=1} \xi_ip(y_i|x_i)\right\}$\\
\end{tabular}
\label{table:QW}
\end{table}

We consider three weighting schemes: RDIL-$\mathcal{L}$, RDIL-Biased, and PWEM described below.
1) \textbf{RDIL-$\mathcal{L}$} uses $p(\hat{y}_i|x_i, \mathcal{L})$. % estimated using the diverse set of learning algorithms presented in Table \ref{table:LA}.
Since not all learning algorithms in Table \ref{table:LA} produce a probability distribution, the Kronecker delta function $\delta(g(x_i),y_i)$ is used in this paper instead of $p(\hat{y}_i|x_i,h)$ where $h(x_i)$ returns the predicted class from the induced hypothesis $h$ given input features $x_i$.
2) \textbf{RDIL-Biased} approximates $p(y_i|x_i)$ as $p(\hat{y}_i|x_i,h)$ where the hypothesis $h$ is induced by the same learning algorithm that is used to induce a model of the data.
To get a real-value from a single hypothesis, we compute a classifier score for each instance from the learning algorithm.
Below, we present how we calculate the classifier scores for the investigated learning algorithms.

\noindent {\bf MLP:}
For multiple classes, each class from a data set is represented with an output node.
After training a MLP with backpropagation, the classifier score is the largest value of the output nodes normalized between zero and one: % (the \textit{softmax} \cite{Bridle1989}):
%
$
\hat{p}(y | x) = \frac{o_i(x)}{\sum_i^{|Y|} o_i(x)}
$
%
where $y$ is a class from the set of possible classes $Y$ and $o_i$ is the value from the output node corresponding to class $y_i$.

\noindent {\bf C4.5:}
To calculate a classifier score, an instance first follows the induced set of rules until it reaches a leaf node.
The classifier score is the number of training instances that have the same class as the examined instance divided by all of the training instances that also reach the same leaf node.

\noindent {\bf 5-NN:}
The percentage of the nearest-neighbors that agree with the class label of an instance as the classifier score.

\noindent {\bf Random Forest:}
For each tree, an instance follows the induced set of rules until it reaches a leaf node.
The counts from the reached leaf nodes for each class are summed together and then normalized between 0 and 1.

\noindent {\bf RIPPER:}
The percentage of training instances that are covered by a rule and share the same class as the examined instance.

% \noindent \textcolor{blue}{{\bf SVM:}
% Fits a logistic model to the outputs. In the case of more than two output classes, pair-wise coupling is used.}

\noindent Obviously, a classifier score does not produce a true probability.
However, the classifier scores approximate the confidence of $p(y_i|x_i)$.
3) Pair-wise expectation maximization (\textbf{PWEM}) \cite{Rebbapragada2007} weights each instance using the EM algorithm.
% First each data set is binarized.
For each pair of classes, the instances that belong to the two classes are clustered using EM where the number of clusters is determined using the Bayesian Information Criterion \cite{Kass1995}. 
Given the $Y-1$ clusterings ($Y$ is the number of classes in the data set), $p(y|x)$ is calculated as:
\thinmuskip=0mu
\begin{align}
 p(y_i|x_i)\! &= \!\sum_\theta p(\theta)p(y_i|x_i, \theta)\!= \!\sum_\theta p(\theta)\!\sum_{c=1}^k p(y_i|c,\theta)p(c|x_i,\theta) \nonumber
\end{align}
where $\theta$ is a clustering model induced using the EM algorithm, $c$ is a cluster in $\theta$, and $k$ is the number of clusters in $\theta$.

% Also, $p(\hat{y}_i|x_i, l_j(T))$ is estimated using the indicator function since not all learning algorithms produce a probability distribution over the output classes.
% Set up as such, the ensemble filter counts how many times an instance is misclassified by a set of learning algorithms.
% Brodley and Friedl \cite{Brodley1999} examined an ensemble of three learning algorithms on five data sets with artificially generated noise inserted into the data sets.
We also compare weighting with three filtering techniques.
1) \textbf{Filter-$\mathcal{L}$} uses $p(\hat{y}_i|x, \mathcal{L})$ for filtering, similar to the three learning algorithm ensemble filter examined by Brodley and Friedl \cite{Brodley1999}.
Instances that are misclassified by 50\% of the learning algorithms in the ensemble are filtered from the training set. 
% This approach is considered the using the \textit{consensus} and is used with the other filtering approaches that we compare against.
Note that other percentages could also be used.
We found that 50\% generally produces good results compared to values of 70\% and 90\%.
% For the results, we report the accuracy from the percentage that produces the highest accuracy for each data set.
% This method provides an overly optimistic estimate of filtering.
In practice a validation set could be used to determine the percentage that would be used.
2) {\bf Filter-Biased} removes any instance that is misclassified by the same learning algorithm that is being used to induce a model from the training set.
3) Repeated-edited nearest-neighbor (\textbf{RENN}) \cite{Tomek1976} repeatedly removes the instances that are misclassified by a 3-nearest neighbor classifier.

Each noise handling method is evaluated by averaging the results from ten runs of each experiment. 
For each experiment, the data is shuffled and then split into 2/3 for training and 1/3 for testing.
The training and testing sets are stratified.
Random noise is introduced by randomly changing $n\%$ of the training instances to a new label chosen uniformly from the possible class labels (noisy completely at random).
The noise levels are examined at 0\%, 10\%, 20\%, 30\%, and 40\%.
% using 5 by 10-fold cross-validation (running 10-fold cross validation 5 times, each time with a different seed to partition the data).
We examine noise handling using the 5 chosen learning algorithms on a set of 54 data sets from the UCI data repository \cite{UCI_dataRepo}.
% Table \ref{table:dataSets} shows the data sets used in this study organized according to the number of instances, number of attributes, and attribute type.
% %
% \begin{table}
% \centering
% \caption{Datasets used organized by number of instances, number of attributes, and attribute type.}
% \setlength{\tabcolsep}{5pt}
% \begin{tabular}{| c | c | c | c | c |}
% \hline
% \multirow{2}{*}{\# Inst}& \multirow{2}{*}{\# Att} & \multicolumn{3}{| c |}{Attribute Type} \\
% \cline{3-5} 
%  &  & Categorical & Numerical & Mixed \\
% \hline
% \multirow{4}{*}{\begin{sideways}$M <$ 100 \end{sideways}} & \multirow{2}{*}{$k <$ 10} & Balloons & & Post-Operative \\
% & & Contact Lenses & & \\
% \cline{2-5}
% & \multirow{1}{*}{$10 < k < 100$} & Lung Cancer & & Labor \\
% \cline{2-5}
% & \multirow{1}{*}{$k > 100$} & & Colon &  \\
% \hline
% \multirow{16}{*}{\begin{sideways}$100<M<1000$ \end{sideways}} & \multirow{6}{*}{$k < 10$} & Breast-w & Iris & Badges 2 \\
% & & Breast Cancer & Ecoli & Teaching- \\
% & & & Pima Indians & Assistant \\
% & & & Glass & \\
% & & & Bupa &  \\
% & & & Balance Scale &  \\
% \cline{2 - 5}
% & \multirow{9}{*}{$10 < k < 100$} & Audiology & Ionosphere & Annealing \\
% & & Soybean(large) & Wine & Dermatology \\
% & & Lymphography & Sonar & Credit-A \\
% & & Congressional- & Heart-Statlog & Credit-G \\
% & & Voting Records & & Horse Colic \\
% & & Vowel & & Heart-c \\
% & & Primary-Tumor & & Hepatitis \\
% & & Zoo & & Autos \\
% & & & & Heart-h \\
% \cline{2-5}
% & \multirow{1}{*}{$k > 100$} & &  & Arrhythmia \\
% \hline
% \multirow{7}{*}{\begin{sideways}$ 1000 < M < 10000$ \end{sideways}} & \multirow{2}{*}{$k < 10$} & Car Evaluation & Yeast & Abalone \\
% & & Chess & & \\
% & & Titanic & & \\
% \cline{2-5}
% & \multirow{5}{*}{$k < 100$} & Splice & Waveform-5000 & Thyroid- \\
% & &  & Segment & (sick \& \\
% & &  & Spambase & hypothyroid) \\
% & &  & Ozone level- &  \\
% & &  & Detection &  \\
% \hline
% \multirow{5}{*}{\begin{sideways}$ M > 10000$ \end{sideways}} & \multirow{2}{*}{$k < 10$} & Nursery & MAGIC- &  \\
%  & & & Telescope & \\
% \cline{2-5}
%  & \multirow{3}{*}{$k < 100$} & Chess- & & Adult-Census- \\
% & & (King-Rook vs. &  & Income (KDD) \\
% & &  King-Pawn) & & \\
% \hline
% \end{tabular}
% \label{table:dataSets}
% \end{table}
% %
Statistical significance between pairs of algorithms is determined using the Wilcoxon signed-ranks test as suggested by Dem\v{s}ar \cite{Demsar2006}.

% Most previous work in noise handling shows significant improvement when there is a high amount of noise in the data.
As there is no way to determine if an instance is noisy or mislabeled without the use of a domain expert, most previous work adds artificial noise to show the impact of noise and how handling noise improves the accuracy.
Generally, once there are large amounts of noise, a noise handling approach significantly increases the classification accuracy.
In the following experiments, artificial noise is added to the data sets.

\section{Results}
\label{section:results}
In this section, we present the results of our experiments.
For the tables in this section, the algorithm in the first row is the baseline algorithm that the algorithms in the subsequent rows are compared against.
The values in the ``g,e,l'' rows represent the number of times that the accuracy from the baseline algorithm is greater than, equal to, or less than the compared algorithm.
A \xmark~represents cases where the baseline algorithm achieves significantly higher classification accuracy.
A \cmark~represents cases where the accuracy from the compared algorithm is significantly higher than the baseline algorithm.
% A \textit{p}-value of 0.000 represents a \textit{p}-value that is less than 0.001.

% \subsection{The Effects of Noise Handling}
Table \ref{table:noNoise} compares no noise handling (Orig) with the considered noise handling techniques. %: 1) pair-wise expectation-maximization (PWEM), 2) RDIL, 3) an ensemble filter using all 9 of the learning algorithms in Table \ref{table:LA} (filter), and 4) RENN.
% The values in the ``Count'' rows represent the number of times that the accuracy from noise handling was greater than, equal to, or less than not using a noise handling technique.
% The bold \textit{p}-values represent the cases where noise handling achieves significantly high classification accuracy.
The only noise handling technique that significantly increases classification accuracy is a MLP using RDIL-$\mathcal{L}$.
In contrast, no noise handling achieves significantly higher accuracy than using a noise handling technique in several cases.
RENN achieves significantly lower classification accuracy for all of the considered learning algorithms.
This highlights a point that is often overlooked in the noise handling literature--noise handling can be detrimental if used in all cases.
Previous work has generally considered only a few data sets where noise handling is beneficial.
% Deciding when to use noise filtering is examined by S\'{a}ez et al. \cite{Saez2013} for nearest neighbor classification based on data set characteristics.
% They show that there is some correlation between data set properties and the efficacy of filtering and that in some cases filtering can cause the classification accuracy to deteriorate.
The impact of filtering or weighting is also dependent on which learning algorithm is used to induce a model of the data.
As expected, MLPs achieve the most significant increase in accuracy with instance weighting.
On the other hand, C4.5, 5-NN, Random Forests, and RIPPER achieve the most significant increase in accuracy with filtering.

\begin{table}
\centering
\caption{The average accuracy over the 54 data sets for the 5 considered learning algorithms using the investigated noise handling approaches with no artificial noise added to the data sets.
A \cmark~to the left represent cases where the noise handling approach significantly increases the accuracy and \xmark~where the noise handling approach significantly decreases the accuracy.}
\setlength{\tabcolsep}{3.0pt}
\begin{tabular}{l|ccccccccccc}
& C4.5& & 5-NN & & MLP & & Rand For & & RIPPER \\ %& SVM \\
\hline
% Orig & 79.19 & 79.08 & 81.68 & 80.96 & 78.24 \\
% \hline
% RDIL-$\mathcal{L}$ & 78.24 & 78.44 & 82.27 & 80.63 & 77.77 \\
% \textit{p}-val & 0.473 & 0.804 & \textbf{0.027} & 0.620 & 0.705 \\
% Count & 27,1,25 & 23,4,26 & 33,3,17 & 24,2,27 & 26,2,25 \\
% \hline
% RDIL-biased & 79.17 & 78.05 & 81.50 & 80.71 & 77.90 \\
% \textit{p}-val & 0.285 & \cellcolor{gray}\underline{1.000} & 0.251 & 0.698 & 0.809 \\
% Count & 24,7,22 & 15,7,31 & 26,5,22 & 24,4,25 & 21,4,28 \\
% \hline
% PWEM & 76.58 & 77.74 & 82.89 & 81.47 & 74.35 \\
% \textit{p}-val & 0.912 & \cellcolor{gray}\underline{0.995} & 0.203 & 0.838 & \cellcolor{gray}\underline{1.000} \\
% Count & 21,3,29 & 18,3,32 & 28,3,22 & 19,1,33 & 14,1,38 \\
% \hline
% $\mathcal{L}$-filter & 79.43 & 79.12 & 81.81 & 81.46 & 78.89 \\
% \textit{p}-val & 0.072 & 0.117 & 0.410 & 0.105 & 0.087 \\
% Count & 25,11,17 & 23,9,21 & 23,4,26 & 28,2,23 & 27,5,21 \\
% \hline
% Biased-filter & 79.23 & 76.68 & 81.44 & 80.93 & 77.41 \\
% \textit{p}-val & 0.504 & \cellcolor{gray}\underline{1.000} & 0.569 & 0.421 & \cellcolor{gray}\underline{0.962} \\
% Count & 22,7,24 & 15,4,34 & 20,10,23 & 21,11,21 & 17,7,29 \\
% \hline
% RENN & 76.67 & 76.68 & 78.75 & 77.94 & 76.52 \\
% \textit{p}-val & \cellcolor{gray}\underline{0.998} & \cellcolor{gray}\underline{1.000} & \cellcolor{gray}\underline{0.999} & \cellcolor{gray}\underline{1.000} & \cellcolor{gray}\underline{0.995} \\
% Count & 19,3,31 & 15,4,34 & 15,1,37 & 17,2,34 & 18,2,33\\
Orig & 79.31 & & 79.37 && 81.67 && 81.18 && 78.35 \\ %& 79.92 \\
\hline
RDIL-$\mathcal{L}$ & 78.19 & & 78.72 & & \textbf{82.26} & \cmark & 80.82 & & 77.86 & \\ %& 79.05 \\
% Stat sig. & - & - & \cmark & - & - \\
% \textit{p}-val & 0.332 & 0.162 & \cellcolor{gray}\underline{0.974} & 0.323 & 0.240 \\ %& \textbf{0.003} \\
g,e,l & 27,1,26& & 27,4,23 & & 18,3,33 & & 28,2,24 & & 26,2,26 \\ %& 34,3,17 \\
\hline
RDIL-Biased & 79.29 &  & 78.34 & \xmark & 81.49 &  & 80.94 &  & 77.98 &  \\ %& 79.96 \\
% \textit{p}-val & 0.658 & \textbf{$<$0.001} & 0.718 & 0.281 & 0.146 \\ %& 0.860 \\
g,e,l & 23,7,24 & & 32,7,15 & & 23,5,26 & & 26,4,24 & & 29,4,21 \\ %& 11,25,18 \\
\hline
PWEM & 76.41 & & 78.02 & \xmark & 82.79 & & 81.51 & & 74.17 & \\ %& 77.71 \\
% \textit{p}-val & 0.062 & \textbf{0.003} & 0.737 & 0.125 & \textbf{$<$0.001} \\ %& \textbf{$<$0.001} \\
g,e,l & 30,3,21 & & 33,3,18 & & 23,3,28 & & 34,1,19 & & 39,1,14 \\ %& 36,4,14 \\
\hline
Filter-$\mathcal{L}$  & 79.55 & & 79.40 & & 81.80 & & 81.66 & & 78.98 & \\ %& 78.85\\
% \textit{p}-val & 0.913 & 0.854 & 0.584 & 0.873 & 0.893 \\ %& \textbf{0.010} \\
g,e,l & 25,11,18 & & 23,9,22 & & 23,4,27 & & 28,2,24 & & 27,5,22\\ %& 19,3,32 \\

% & 78.14 & 77.71 & 80.73 & 79.50 & 77.56 & 78.85 \\
% \textit{p}-val & 0.259 & \textbf{0.005} & 0.144 & \textbf{0.028} & 0.118 & \textbf{0.010} \\
% Count & 26,1,27 & 17,3,34 & 22,2,30 & 21,1,32 & 21,3,30 & 19,3,32 \\
\hline
Filter-Biased & 79.34 & & 76.99 & \xmark & 81.39 & & 81.16 & & 77.20 & \\ %& 79.08 \\
% \textit{p}-val & 0.422 & \textbf{$<$0.001} & 0.339 & 0.584 & \textbf{0.025} \\ %& \textbf{$<$0.001} \\
g,e,l & 25,7,22 & & 35,4,15 & & 24,10,20 & & 21,12,21 & & 30,7,17 \\ %& 27,11,16 \\
\hline
RENN & 76.83 & \xmark & 76.99 & \xmark & 78.80 & \xmark & 78.20 & \xmark & 76.65 & \xmark \\ %& 77.24 \\
% \textit{p}-val & \textbf{0.002} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{0.004} \\ %& \textbf{$<$0.001} \\
g,e,l & 32,3,19& & 35,4,15 & & 38,1,15 & & 35,2,17 & & 34,2,18 \\ %& 40,2,12 \\
% \hline

\end{tabular}
\label{table:noNoise}
\end{table}

% \begin{figure*}
% \label{figure:artificialNoise}
% \centering
% \setlength{\tabcolsep}{0pt}
% \begin{tabular}{cc}
% \input{test_0} & \input{test_1} \\
% C4.5 & 5NN\\
% \input{test_2} & \input{test_3}\\
% MLP & Random Forest\\
% \input{test_4} & \input{key}\\
% RIPPER\\
% \end{tabular}
% \caption{Graphs for each learning algorithm comparing the average accuracy achieved when using the investigated noise handling techniques.}
% \end{figure*}


% Figure 5 graphs the percent reduction in error for each learning algorithm and noise handling method for the considered artificial noise levels of 0\%, 10\%, 20\%, 30\%, and 40\%.
% The percent reduction in error is the percentage of error that is reduced when a noise handling technique is used compared to the error obtained with no noise handling:
% \begin{equation}
% \%RE = \frac{noise-orig}{100-orig} \nonumber
% \end{equation}
% where $noise$ is the accuracy achieved when using a noise handling technique and $orig$ is the accuracy obtained when not using a noise handling technique.
% The x-axis represents the no change in error from not using a noise handling technique, negative values represent an increase in error when using a noise handling technique and positive values represent a decrease in error compared to not using noise handling.
% In general, the reduction in the percentage of error increases as the amount of noise increases.
% The decrease in error is greatest for RDIL and the ensemble filter for each learning algorithm.
% % Table \ref{table:significance} summarizes when each noise handling method significantly increases the classification accuracy as indicated by a checkmark (\cmark).
% The accuracies and \textit{p}-values for each learning algorithm and noise handling technique are shown in Table \ref{table:LAsig} in \ref{section:sigTables}.

Examining the performance of the considered learning algorithms without noise handling (``orig'' in Tables \ref{table:weightingSchemes} and \ref{table:filtering}), we note that MLPs and random forests generally achieve the highest classification accuracy and may be the most tolerant to the inherent detrimental instances in each data set.
However, MLPs and Random Forests also appear to be the least robust to noise as they obtain the lowest average classification accuracy when more than 10\% of the instances are corrupted with noise.
With no artificial noise, MLPs and Random Forests achieve about 81\% accuracy.
With 20\% artificial noise, the average accuracy decreases to about 72\%.
On the other hand, C4.5, 5-NN, and RIPPER achieve an average accuracy of about 79\% with no artificial noise and an average accuracy of about 74\% with 20\% artificial noise.
With high degrees of noise, the built-in noise handling mechanisms of learning algorithms become more beneficial.

% \begin{table}
% \centering
% \caption{Tables of the learning algorithms and filtering techniques for each noise level. A checkmark (\cmark) indicates that the noise handling technique causes a significant increase in classification accuracy.}
% \begin{tabular}{l|ccccc||ccccc}
% & \multicolumn{5}{c||}{PWEM} & \multicolumn{5}{c}{RDIL} \\
%  & 0 & 0.1 & 0.2 & 0.3 & 0.4 & 0 & 0.1 & 0.2 & 0.3 & 0.4 \\
% \hline
% C4.5 &  &  &  & \cmark & \cmark &  &  & \cmark & \cmark & \cmark \\
% \hline
% 5NN &  &  & \cmark & \cmark & \cmark &  & \cmark & \cmark & \cmark & \cmark \\
% \hline
% MLP &  & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark \\
% \hline
% RF &  &  & \cmark & \cmark & \cmark &  & \cmark & \cmark & \cmark & \cmark \\
% \hline
% RIPPER &  &  &  &  &  &  &  & \cmark & \cmark & \cmark \\
% \multicolumn{11}{c}{} \\
% & \multicolumn{5}{c||}{filter} & \multicolumn{5}{c}{RENN} \\
%  & 0 & 0.1 & 0.2 & 0.3 & 0.4 & 0 & 0.1 & 0.2 & 0.3 & 0.4 \\
% \hline
% C4.5 &  & \cmark & \cmark & \cmark & \cmark &  &  &  & \cmark & \cmark \\
% \hline
% 5NN &  & \cmark & \cmark & \cmark & \cmark &  &  &  & \cmark & \cmark \\
% \hline
% MLP &  & \cmark & \cmark & \cmark & \cmark &  &  & \cmark & \cmark & \cmark \\
% \hline
% RF &  & \cmark & \cmark & \cmark & \cmark &  &  & \cmark & \cmark &  \\
% \hline
% RIPPER &  &  & \cmark & \cmark & \cmark &  &  &  &  & \\
% \end{tabular}
% \label{table:significance}
% \end{table}
% 
% Examining each noise handling technique, RENN causes the less amount of significant increases in accuracy, especially for the RIPPER learning algorithm. 
% MAYBE keep talking about this...OR just get rid of it.
\subsection{Weighting Schemes}

% \textcolor{blue}{In the previous section, we examined filtering (completely removing an instance from the training data) and weighting the training instances.}
As instance weighting is not as well explored as filtering, we now examine various weighting schemes to handle class noise.
Table \ref{table:weightingSchemes} compares RDIL-$\mathcal{L}$ with RDIL-Biased and PWEM.
The accuracies from the algorithms with no weighting are given to better measure the effectiveness of the methods.
% The random forest learning algorithm was chosen because it has the highest correlation with $p(\hat{y}_i|x_i)$ out of the examined learning algorithms \cite{Smith2012_IH}.
% The three learning algorithms in RDIL\_3 were also chosen because they have a relatively high correlation with $p(\hat{y}_i|x_i)$ and because they are diverse (see Section \ref{section:methodology}).
% The values in bold represent the cases where RDIL achieves significantly higher classification accuracy than the compared weighting method.
RDIL-$\mathcal{L}$ significantly outperforms the other weighting schemes in most cases (represented by bold $\textit{p}$-values): 24 out of the 25 cases for PWEM, and 18 out of the 25 cases for RDIL-Biased.
In no case does a competing weighting scheme achieve significantly higher classification accuracy than RDIL-$\mathcal{L}$.
Recall that the nine learning algorithms were chosen to be diverse so as to represent more of the hypothesis space $\mathcal{H}$.
This suggests that a better estimation of $p(\hat{y}_i|x_i)$ produces better results for weighting and filtering.
This is shown empirically as RDIL-$\mathcal{L}$ and Filter-$\mathcal{L}$ have the most significant increase in accuracy for each learning algorithm (Table \ref{table:noNoise}).
However, there is an obvious trade-off since obtaining a more accurate estimate of $p(\hat{y}_i|x_i)$ is more computationally expensive.


\begin{table}
\centering
\caption{A comparison of the average accuracy from investigated instance weighting methods on the considered learning algorithms.
Bold values with a \cmark~represent cases where RDIL-$\mathcal{L}$ (R-$\mathcal{L}$) acheives significantly higher accuracy than RDIL-Biased (R-B) or PWEM (PW).}
\setlength{\tabcolsep}{0pt}
\centering
\begin{tabular}{l|ccccc||ccccc}
& \multicolumn{5}{c||}{C4.5}& \multicolumn{5}{c}{5-NN} \\
  & 0\% & 10\% & 20\% & 30\% & 40\% & 0\% & 10\% & 20\% & 30\% & 40\% \\
\hline          
R-$\mathcal{L}$ & 78.19 & 77.03 & 76.33 & 74.32 & 71.08 & 78.72 & 77.86 & 77.01 & 75.07 & 70.63 \\
\hline                     
R-B & 79.29 & 77.14 & \textbf{75.20}\cmark~ & \textbf{71.06}\cmark~ & \textbf{65.74}\cmark~ & \textbf{78.34}\cmark~& \textbf{77.41}\cmark~ & \textbf{75.39}\cmark~ & \textbf{71.59}\cmark~ & \textbf{64.92}\cmark \\
% \textit{p}-val & 0.738 & 0.141 & \textbf{0.005} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{0.002} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} \\
g,e,l & 25,4,25 & 32,2,19 & 38,0,16 & 44,0,10 & 43,1,10 & 34,7,13 & 41,3,9 & 43,1,10 & 46,1,7 & 44,1,9 \\
\hline
PW & \textbf{76.41}\cmark~ & \textbf{74.50}\cmark~ & \textbf{73.34}\cmark~ & \textbf{70.94}\cmark~ & \textbf{68.28}\cmark~ & \textbf{78.02}\cmark~ & \textbf{77.54}\cmark~ & \textbf{76.12}\cmark~ & \textbf{73.56}\cmark~ & \textbf{67.18}\cmark~ \\
% \textit{p}-val & \textbf{0.003} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{0.002} & \textbf{0.007} & \textbf{0.002} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} \\
g,e,l & 35,4,15 & 39,3,11 & 38,4,12 & 41,0,13 & 37,0,17 & 34,4,16 & 36,2,15 & 37,1,16 & 37,3,14 & 37,3,14 \\\hline
orig & 79.31 & 76.92 & 74.32 & 69.709 & 63.08 & 79.37 & 77.63 & 74.42 & 69.96 & 62.54\\
\multicolumn{11}{c}{}\\
&\multicolumn{5}{c}{MLP}&\multicolumn{5}{c}{Random Forest}\\                    
R-$\mathcal{L}$ & 82.26 & 80.7 & 78.40 & 75.17 & 69.30 & 80.82 & 79.72 & 78.06 & 75.89 & 70.78 \\\hline                     
R-B & \textbf{81.49}\cmark~ & \textbf{78.19}\cmark~ & \textbf{73.84}\cmark~ & \textbf{69.14}\cmark~ & \textbf{62.10}\cmark~ & 80.94 & \textbf{78.24}\cmark~ & \textbf{74.01}\cmark~ & \textbf{68.25}\cmark~ & \textbf{60.93}\cmark~ \\
% \textit{p}-val & \textbf{0.042} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & 0.523 & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} \\
g,e,l & 34,2,18 & 41,1,11 & 46,1,6 & 48,0,6 & 47,1,6 & 25,1,28 & 42,2,9 & 48,0,5 & 48,1,5 & 49,1,4 \\
\hline
PW & \textbf{82.79}\cmark~ & \textbf{79.67}\cmark~ & \textbf{76.42}\cmark~ & \textbf{71.9}\cmark~ & \textbf{65.95}\cmark~ & 81.51 &\textbf{78.69}\cmark~ & \textbf{76.37}\cmark~ & \textbf{72.54}\cmark~ & \textbf{65.87}\cmark~ \\
% \textit{p}-val & \textbf{0.019} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & 0.207 & \textbf{0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} \\
g,e,l & 34,4,16 & 37,2,14 & 38,1,14 & 42,0,12 & 42,0,12 & 33,4,17 & 34,4,15 & 40,4,9 & 47,1,6 & 48,1,5 \\\hline
orig & 81.67 & 77.46 & 72.25 & 67.17 & 60.46 & 81.18 & 77.75 & 72.72 & 66.87 & 59.63 \\
\multicolumn{11}{c}{}\\
&\multicolumn{5}{c}{RIPPER}\\ %&\multicolumn{5}{c}{SVM}\\                    
R-$\mathcal{L}$ & 77.86 & 76.54 & 75.54 & 73.46 & 69.63 \\ %& 79.05 & 78.23 & 77.11 & 75.12 & 71.71 \\
\cline{1-6}
R-B & 77.98 & 76.28 & \textbf{74.50}\cmark~ & \textbf{70.70}\cmark~ & \textbf{65.97}\cmark~ \\ %& 79.96 & 77.72 & 75.12 & 72.04 & 67.74 \\
% \textit{p}-val & 0.402 & 0.081 & \textbf{0.002} & \textbf{$<$0.001} & \textbf{$<$0.001} \\ %& \cellcolor{gray}\underline{0.997} & \textbf{0.014} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} \\
g,e,l & 27,3,24 & 31,2,20 & 36,3,15 & 46,2,6 & 45,0,9 \\ %& 16,4,34 & 33,4,17 & 42,3,9 & 45,1,8 & 40,1,12 \\
\cline{1-6}
PW & \textbf{74.17}\cmark~ & \textbf{71.94}\cmark~ & \textbf{70.68}\cmark~ & \textbf{68.57}\cmark~ & \textbf{64.82}\cmark~ \\ %& 77.71 & 76.36 & 75.25 & 72.88 & 69.89 \\
% \textit{p}-val & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} \\ %& \textbf{0.013} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{$<$0.001} & \textbf{0.013} \\
g,e,l & 36,4,14 & 44,2,7 & 45,4,5 & 40,2,12 & 41,1,12 \\ \cline{1-6} %& 33,5,16 & 40,4,10 & 37,3,14 & 37,2,15 & 32,2,19 \\
orig & 78.35 & 76.32 & 73.45 & 69.87 & 65.10 \\
\end{tabular}
\label{table:weightingSchemes}
\end{table}

\subsection{Weighting VS Filtering}

We now compare weighting against filtering.
Weighting and filtering are {\em both} viable and significantly increase the classification accuracy when noise is added.
The difference between filtering and weighting techniques depends on the estimation of $p(y_i|x_i)$. 
Generally, estimating $p(y_i|x_i)$ with the set of learning algorithms $\mathcal{L}$ achieves greater classification accuracy than using a biased estimate.
% In essence, filtering is a special case of weighting where only weights of 0 and 1 are assigned to an instance.
% Many common filters remove instances that are misclassified by a single learning algorithm or an ensemble of learning algorithms.
% When an ensemble of learning algorithms is used, a threshold of the percentage of learning algorithms that misclassify an instance must be specified to determine when to filter an instance.
Table \ref{table:filtering} compares RDIL-$\mathcal{L}$ with Filter-$\mathcal{L}$. % and RDIL-Biased with Filter-Biased.
% The values in bold represent the cases where RDIL achieves significantly higher classification accuracy than the filtering method.
% The values that are underlined and in the gray cells are those where filtering achieves significantly higher classification accuracy.
% Overall, RDIL achieves significantly higher classification accuracy than RENN for all of the investigated learning algorithms.
% The ensemble filter is more competitive than RENN, again showing that the approximation of $p(\hat{y}_i|x_i)$ is important for determining which instances to remove.
With no noise, RDIL-$\mathcal{L}$ achieves significantly higher accuracy than Filter-$\mathcal{L}$ for the MLP.
Since each instance can affect the classification boundary for MLPs (as shown in Figure \ref{figure:example}), weighting the instances in the training set has a more significant impact in MLPs than the other learning algorithms which partition the input space. %larger impact than simply weighting them--although not significantly.
On the other hand, the Filter-$\mathcal{L}$ achieves a significantly higher accuracy than RDIL-$\mathcal{L}$ for the four other learning algorithms.
Note that MLP with RDIL-$\mathcal{L}$ achieves the highest overall average accuracy for noise levels 0\%-20\% (RDIL-$\mathcal{L}$ achieves the highest accuracy for 30\% and 40\% noise using Random Forest and C4.5 respectively).
Except for MLPs, the significance of the impact of RDIL-$\mathcal{L}$ increases as the noise level increases except for all of the examined learning algorithms.
RDIL-$\mathcal{L}$ significantly increases the classification accuracy for C4.5, 5-NN, and Random Forests when there are high amounts of noise.

Over all noise levels, RDIL-$\mathcal{L}$ compared with Filter-$\mathcal{L}$ achieves significantly higher classification accuracy in 6 of the 25 cases and the filter-$\mathcal{L}$ achieves significantly higher classification accuracy in 7 cases.
(In the other 12 cases, there is no significant difference).
The Filter-$\mathcal{L}$ has a more significant effect than RDIL-$\mathcal{L}$ for RIPPER at noise levels 0, 0.1 and 0.2 and RDIL-$\mathcal{L}$ never achieves significantly higher classification accuracy than the Filter-$\mathcal{L}$.
Therefore, instance weighting is not the best option for every learning algorithm.
However, with the Filter-$\mathcal{L}$ we chose the threshold that produced the highest classification accuracy on the test set, which is not always possible to do.
Instance weighting avoids the overhead of having to determine a threshold for filtering when using an ensemble filter.
Instance weighting is better for learning algorithms that consider each instance individually and each instance can affect the classification boundary (e.g. MLP).
% The type of learning algorithm will determine whether it is better to filter or to weight the instances for lower noise levels.

\begin{table}
\centering
\caption{A comparison of RDIL-$\mathcal{L}$ (R-$\mathcal{L}$) with the $\mathcal{L}$-filter (F-$\mathcal{L}$) on the considered learning algorithms.
Bold values with a \cmark~represent cases where RDIL-$\mathcal{L}$ (R-$\mathcal{L}$) acheives significantly higher accuracy than the $\mathcal{L}$-filter.
The \xmark~represents cases where the $\mathcal{L}$-filter acheives significantly higher accuracy.}
\setlength{\tabcolsep}{0pt}
\begin{tabular}{l|ccccc||ccccc}
& \multicolumn{5}{c||}{C4.5}& \multicolumn{5}{c}{5-NN} \\  & 0\% & 10\% & 20\% & 30\% & 40\% & 0\% & 10\% & 20\% & 30\% & 40\% \\
\hline
R-$\mathcal{L}$ & 78.19 & 77.03 & 76.33 & 74.32 & 71.08 & 78.72 & 77.86 & 77.01 & 75.07 & 70.63 \\
\hline
F-$\mathcal{L}$ & 79.55~\xmark~ & 78.35~\xmark~ & 76.79 & 73.58 & \textbf{69.30}~\cmark~ & 79.40~\xmark~ & 78.35 & 76.6 & \textbf{74.35}~\cmark~ & \textbf{69.64}~\cmark \\
% \textit{p}-val & \cellcolor{gray}\underline{0.976} & \cellcolor{gray}\underline{0.961} & 0.461 & 0.06 & \textbf{0.008} & \cellcolor{gray}\underline{0.997} & 0.483 & 0.058 & \textbf{0.021} & \textbf{0.032} \\
g,e,l & 19,2,33 & 21,1,31 & 30,1,23 & 33,0,21 & 36,0,18 & 19,4,31 & 28,2,23 & 26,4,24 & 31,0,23 & 31,0,23 \\\hline
orig & 79.31 & 76.92 & 74.32 & 69.709 & 63.08 & 79.37 & 77.63 & 74.42 & 69.96 & 62.54\\
% \cmidrule{1-11}
% RDIL-B & 79.29 & 77.14 & 75.2 & 71.06 & 65.74 & 78.34 & 77.41 & 75.39 & 71.59 & 64.92 \\
% \hline
% Filter-B & 79.34 & 77.34 & 75.09 & 71.26 & 65.58 & 76.99 & 75.98 & 74.33 & 71.82 & 66.12 \\
% \textit{p}-val & 0.539 & 0.833 & 0.190 & 0.657 & 0.238 & \textbf{$<$0.001} & \textbf{$<$0.001} & 0.462 & 0.912 & \cellcolor{gray}\underline{0.978} \\
% Count & 24,6,24 & 24,1,28 & 30,4,20 & 26,2,26 & 31,1,22 & 35,6,13 & 33,4,16 & 25,0,29 & 20,0,34 & 15,0,39 \\
% \hline
\multicolumn{11}{c}{}\\
& \multicolumn{5}{c||}{MLP}& \multicolumn{5}{c}{Random Forest} \\
% noise  & 0\% & 10\% & 20\% & 30\% & 40\% & 0\% & 10\% & 20\% & 30\% & 40\% \\
\hline
R-$\mathcal{L}$ & 82.26 & 80.7 & 78.4 & 75.17 & 69.30 & 80.82 & 79.72 & 78.06 & 75.89 & 70.78 \\
\hline
F-$\mathcal{L}$ & 81.80 & 80.66 & 78.24 & 74.85 & 69.46 & 81.66~\xmark~ & 79.91 & 78.06 & \textbf{75.29}~\cmark~ & \textbf{69.94}~\cmark \\
% \textit{p}-val & \textbf{0.009} & 0.143 & 0.182 & 0.174 & 0.615 & \cellcolor{gray}\underline{0.991} & 0.129 & 0.353 & \textbf{0.031} & \textbf{0.016} \\
g,e,l & 37,2,15 & 30,3,20 & 31,1,21 & 33,0,21 & 29,1,24 & 18,4,32 & 31,0,22 & 26,1,26 & 33,0,21 & 34,3,17 \\\hline
orig & 81.67 & 77.46 & 72.25 & 67.17 & 60.46 & 81.18 & 77.75 & 72.72 & 66.87 & 59.63 \\
% \cmidrule{1-11}
% RDIL-B & 81.49 & 78.19 & 73.84 & 69.14 & 62.1 & 80.94 & 78.24 & 74.01 & 68.25 & 60.93 \\
% \hline
% Filter-B & 81.39 & 78.16 & 75.04 & 70.57 & 64.35 & 81.16 & 78.13 & 74.25 & 69.18 & 62.80 \\
% \textit{p}-val & 0.229 & 0.581 & \cellcolor{gray}\underline{0.999} & \cellcolor{gray}\underline{0.999} & \cellcolor{gray}\underline{1.000} & 0.665 & 0.615 & 0.648 & \cellcolor{gray}\underline{0.980} & \cellcolor{gray}\underline{0.955} \\
% Count & 24,10,20 & 26,1,26 & 18,1,34 & 18,2,34 & 15,1,38 & 22,6,26 & 25,1,27 & 28,0,25 & 20,1,33 & 22,0,32 \\
% \hline
\multicolumn{11}{c}{}\\
& \multicolumn{5}{c||}{RIPPER} \\%& \multicolumn{5}{c}{SVM} \\
% noise  & 0\% & 10\% & 20\% & 30\% & 40\% \\%& 0\% & 10\% & 20\% & 30\% & 40\% \\
\cline{1-6}
R-$\mathcal{L}$ & 77.86 & 76.54 & 75.54 & 73.46 & 69.63 \\
\cline{1-6}
F-$\mathcal{L}$ & 78.98~\xmark~ & 77.82~\xmark~ & 76.40~\xmark~ & 73.92 & 69.84 \\
% \textit{p}-val & \cellcolor{gray}\underline{0.996} & \cellcolor{gray}\underline{0.994} & \cellcolor{gray}\underline{0.965} & 0.518 & 0.468 \\
g,e,l & 15,3,36 & 18,3,32 & 18,0,36 & 27,1,26 & 25,1,28 \\\cline{1-6}
orig & 78.35 & 76.32 & 73.45 & 69.87 & 65.10 \\
% \cmidrule{1-6}
% RDIL-B & 77.98 & 76.28 & 74.50 & 70.70 & 65.97 \\%& 79.96 & 77.72 & 75.12 & 72.04 & 67.74 \\
% \cline{1-6}
% Filter-B & 77.20 & 75.70 & 73.48 & 70.39 & 65.78 \\%& 79.08 & 77.26 & 75.09 & 72.45 & 68.10 \\
% \textit{p}-val & 0.114 & 0.346 & \textbf{$<$0.001} & 0.369 & 0.514 \\%& \textbf{$<$0.001} & 0.101 & 0.758 & 0.948 & \cellcolor{gray}\underline{0.970} \\
% Count & 27,5,22 & 29,2,22 & 37,1,16 & 27,0,27 & 26,1,27 \\%& 28,10,16 & 23,11,20 & 20,9,25 & 20,8,26 & 18,6,29 \\












% $\mathcal{L}$-filter & 79.43 & 78.23 & 76.5 & 73.25 & 68.91 & 79.12 & 78.07 & 76.33 & 74.07 & 69.3 \\
% \hline
% RDIL-$\mathcal{L}$ & 78.24 & 77.1 & 76.19 & 74.07 & 70.84 & 78.44 & 77.86 & 76.76 & 74.79 & 70.3 \\
% \textit{p}-val & 0.926 & \cellcolor{gray}\underline{0.961} & 0.353 & 0.051 & \textbf{0.007} & \cellcolor{gray}\underline{0.996} & 0.483 & \textbf{0.044} & \textbf{0.022} & \textbf{0.031} \\
% Count & 20,2,31 & 21,1,31 & 30,1,22 & 33,0,20 & 35,0,18 & 19,4,30 & 28,2,23 & 26,4,23 & 30,0,23 & 30,0,23 \\
% \cmidrule{1-11}
% b-filter & 79.23 & 77.36 & 75.02 & 71.14 & 65.32 & 76.68 & 75.66 & 74 & 71.47 & 65.68 \\
% \hline
% RDIL-b & 79.17 & 77.08 & 75.13 & 70.86 & 65.47 & 78.05 & 77.11 & 75.06 & 71.21 & 64.48 \\
% \textit{p}-val & 0.578 & 0.847 & 0.196 & 0.789 & 0.262 & $\textbf{0.000}$ & $\textbf{0.000}$ & 0.468 & 0.912 & \cellcolor{gray}\underline{0.975} \\
% Count & 23,6,24 & 24,1,28 & 29,4,20 & 24,2,27 & 30,1,22 & 34,6,13 & 33,4,16 & 24,0,29 & 19,0,34 & 15,0,38 \\
% \hline
% \multicolumn{11}{c}{}\\
% & \multicolumn{5}{c||}{MLP}& \multicolumn{5}{c}{Random Forest} \\
% noise  & 0\% & 10\% & 20\% & 30\% & 40\% & 0\% & 10\% & 20\% & 30\% & 40\% \\
% \hline
% $\mathcal{L}$-filter & 81.81 & 80.64 & 78.12 & 74.71 & 69.25 & 81.46 & 79.69 & 77.99 & 75.19 & 69.78 \\
% \hline
% RDIL-$\mathcal{L}$ & 82.27 & 80.7 & 78.4 & 75.06 & 69.12 & 80.63 & 79.72 & 78.06 & 75.87 & 70.7 \\
% \textit{p}-val & \textbf{0.009} & 0.143 & 0.182 & 0.138 & 0.563 & \cellcolor{gray}\underline{0.986} & 0.129 & 0.353 & \textbf{0.017} & \textbf{0.009} \\
% Count & 37,2,14 & 30,3,20 & 31,1,21 & 33,0,20 & 29,1,23 & 18,4,31 & 31,0,22 & 26,1,26 & 33,0,20 & 34,3,16 \\
% \cmidrule{1-11}
% b-filter & 81.44 & 78.16 & 74.99 & 70.44 & 64.11 & 80.93 & 77.88 & 73.95 & 68.96 & 62.38 \\
% \hline
% RDIL-b & 81.5 & 78.14 & 73.71 & 68.94 & 61.77 & 80.71 & 78.02 & 73.8 & 68.05 & 60.74 \\
% \textit{p}-val & 0.304 & 0.654 & \cellcolor{gray}\underline{0.999} & \cellcolor{gray}\underline{0.999} & \cellcolor{gray}\underline{1.000} & 0.639 & 0.538 & 0.563 & \cellcolor{gray}\underline{0.982} & 0.933 \\
% Count & 23,10,20 & 26,1,26 & 18,1,34 & 17,2,34 & 14,1,38 & 22,6,25 & 25,1,27 & 28,0,25 & 20,1,32 & 22,0,31 \\
% \hline
% \multicolumn{11}{c}{}\\
% & \multicolumn{5}{c||}{RIPPER}&  \\
% noise  & 0\% & 10\% & 20\% & 30\% & 40\% & \\
% \cline{1-6}
% $\mathcal{L}$-filter & 78.89 & 77.71 & 76.34 & 73.85 & 69.73 \\
% \cline{1-6}
% RDIL-$\mathcal{L}$ & 77.77 & 76.54 & 75.47 & 73.38 & 69.53 \\
% \textit{p}-val & \cellcolor{gray}\underline{0.995} & \cellcolor{gray}\underline{0.994} & \cellcolor{gray}\underline{0.959} & 0.520 & 0.462 \\
% Count & 15,3,35 & 18,3,32 & 18,0,35 & 27,1,25 & 25,1,27 \\
% \cmidrule{1-6}
% b-filter & 77.41 & 75.88 & 73.64 & 70.5 & 65.83 \\
% \cline{1-6}
% RDIL-b & 77.9 & 76.19 & 74.43 & 70.62 & 65.92 \\
% \textit{p}-val & 0.161 & 0.435 & \textbf{0.001} & 0.458 & 0.602 \\
% Count & 26,5,22 & 29,2,22 & 36,1,16 & 26,0,27 & 25,1,27 \\
% 
% % RDIL & 78.24 & 77.10 & 76.19 & 74.07 & 70.84 &  78.45 & 77.86 & 76.76 & 74.79 & 70.30 \\
% % \hline
% % filter & 79.43 & 78.23 & 76.50 & 73.25 & 68.91 &  79.12 & 78.07 & 76.33 & 74.07 & 69.30 \\
% % % \textit{p}-val & 0.076 & \textbf{0.040} & 0.651 & 0.950 & \cellcolor{gray}0.993 &  \textbf{0.004} & 0.521 & \cellcolor{gray}0.956 & \cellcolor{gray}0.978 & \cellcolor{gray}0.970 \\
% % \textit{p}-val & 0.926 & \cellcolor{gray}\underline{0.961} & 0.353 & 0.051 & \textbf{0.007} & \cellcolor{gray}\underline{0.996} & 0.483 & \textbf{0.044} & \textbf{0.022} & \textbf{0.031} \\
% % Count & \begin{footnotesize}20,2,31\end{footnotesize} & \begin{footnotesize}21,1,31\end{footnotesize} & \begin{footnotesize}30,1,22\end{footnotesize} & \begin{footnotesize}33,0,20\end{footnotesize} & \begin{footnotesize}35,0,18\end{footnotesize} & \begin{footnotesize}19,4,30\end{footnotesize} & \begin{footnotesize}28,2,23\end{footnotesize} & \begin{footnotesize}26,4,23\end{footnotesize} & \begin{footnotesize}30,0,23\end{footnotesize} & \begin{footnotesize}30,0,23\end{footnotesize} \\
% % \hline
% % RENN & 76.67 & 75.95 & 74.54 & 71.76 & 66.01 & 76.68 & 75.66 & 74.00 & 71.47 & 65.68 \\
% % \textit{p}-val & \textbf{0.000} & \textbf{0.010} & \textbf{0.000} & \textbf{0.000} & \textbf{0.000} & \textbf{0.000} & \textbf{0.000} & \textbf{0.000} & \textbf{0.000} & \textbf{0.000} \\
% % Count & \begin{footnotesize}38,3,12\end{footnotesize} & \begin{footnotesize}32,5,16\end{footnotesize} & \begin{footnotesize}32,4,17\end{footnotesize} & \begin{footnotesize}40,1,12\end{footnotesize} & \begin{footnotesize}45,0,8\end{footnotesize} & \begin{footnotesize}40,5,8\end{footnotesize} & \begin{footnotesize}45,2,6\end{footnotesize} & \begin{footnotesize}44,1,8\end{footnotesize} & \begin{footnotesize}39,2,12\end{footnotesize} & \begin{footnotesize}41,0,12\end{footnotesize} \\
% % \hline
% % \multicolumn{11}{c}{}\\
% % & \multicolumn{5}{c||}{MLP}& \multicolumn{5}{c}{Random Forest} \\
% % noise  & 0\% & 10\% & 20\% & 30\% & 40\% & 0\% & 10\% & 20\% & 30\% & 40\% \\
% % \hline
% % RDIL & 82.27 & 80.70 & 78.40 & 75.06 & 69.12 & 80.64 & 79.72 & 78.06 & 75.87 & 70.70 \\
% % \hline
% % filter & 81.81 & 80.64 & 78.12 & 74.71 & 69.25 & 81.46 & 79.69 & 77.99 & 75.19 & 69.78 \\
% % % \textit{p}-val & \cellcolor{gray}0.991 & 0.859 & 0.820 & 0.864 & 0.440 & \textbf{0.014} & 0.873 & 0.651 & \cellcolor{gray}0.983 & \cellcolor{gray}0.991\\
% % \textit{p}-val & \textbf{0.009} & 0.143 & 0.182 & 0.138 & 0.563 & \cellcolor{gray}\underline{0.986} & 0.129 & 0.353 & \textbf{0.017} & \textbf{0.009} \\
% % Count & \begin{footnotesize}37,2,14\end{footnotesize} & \begin{footnotesize}30,3,20\end{footnotesize} & \begin{footnotesize}31,1,21\end{footnotesize} & \begin{footnotesize}33,0,20\end{footnotesize} & \begin{footnotesize}29,1,23\end{footnotesize} & \begin{footnotesize}18,4,31\end{footnotesize} & \begin{footnotesize}31,0,2\end{footnotesize} & \begin{footnotesize}26,1,26\end{footnotesize} & \begin{footnotesize}33,0,20\end{footnotesize} & \begin{footnotesize}34,3,16\end{footnotesize} \\
% % % Count & 14,2,37 & 20,3,30 & 21,1,31 & 20,0,33 & 23,1,29 & 31,4,18 & 22,0,31 & 26,1,26 & 20,0,33 & 16,3,34 \\
% % \hline
% % RENN & 78.75 & 77.73 & 76.11 & 73.16 & 67.20 & 77.94 & 76.98 & 75.34 & 73.02 & 67.14 \\
% % \textit{p}-val & \textbf{0.000} & \textbf{0.000} & \textbf{0.004} & \textbf{0.019} & \textbf{0.033}  & \textbf{0.000} & \textbf{0.000} & \textbf{0.000} & \textbf{0.000} & \textbf{0.000} \\
% % Count & \begin{footnotesize}46,2,5\end{footnotesize} & \begin{footnotesize}40,0,13\end{footnotesize} & \begin{footnotesize}37,0,16\end{footnotesize} & \begin{footnotesize}36,2,15\end{footnotesize} & \begin{footnotesize}35,0,18\end{footnotesize} & \begin{footnotesize}44,4,5\end{footnotesize} & \begin{footnotesize}41,3,9\end{footnotesize} & \begin{footnotesize}40,0,13\end{footnotesize} & \begin{footnotesize}38,0,15\end{footnotesize} & \begin{footnotesize}44,0,9\end{footnotesize} \\
% % \multicolumn{11}{c}{}\\
% % & \multicolumn{5}{c||}{RIPPER}&  \\
% % noise  & 0\% & 10\% & 20\% & 30\% & 40\% & \\
% % \cline{1-6}
% % RDIL & 77.77 & 76.54 & 75.47 & 73.38 & 69.53 \\
% % \cline{1-6}
% % filter & 78.89 & 77.71 & 76.34 & 73.85 & 69.73 \\
% % % \textit{p}-val & \textbf{0.005} & \textbf{0.006} & \textbf{0.041} & 0.484 & 0.542 \\
% % \textit{p}-val & \cellcolor{gray}\underline{0.995} & \cellcolor{gray}\underline{0.994} & \cellcolor{gray}\underline{0.959} & 0.520 & 0.462 \\
% % Count & \begin{footnotesize}15,3,35\end{footnotesize} & \begin{footnotesize}18,3,32\end{footnotesize} & \begin{footnotesize}18,0,35\end{footnotesize} & \begin{footnotesize}27,1,25\end{footnotesize} & \begin{footnotesize}25,1,27\end{footnotesize} \\
% % % Count & 35,3,15 & 32,3,18 & 35,0,18 & 25,1,27 & 27,1,25 \\
% % \cline{1-6}
% % RENN & 76.52 & 75.75 & 74.46 & 71.65 & 66.67 \\
% % \textit{p}-val & \textbf{0.010} & \textbf{0.026} & \textbf{0.031} & \textbf{0.009} & \textbf{0.000} \\
% % Count & \begin{footnotesize}33,4,16\end{footnotesize} & \begin{footnotesize}34,1,18\end{footnotesize} & \begin{footnotesize}34,1,18\end{footnotesize} & \begin{footnotesize}32,1,20\end{footnotesize} & \begin{footnotesize}36,0,17\end{footnotesize} \\
\end{tabular}
\label{table:filtering}
\end{table}




% \subsection{Comparison with a Voting Ensemble}
% As RDIL and the ensemble filter make use of an ensemble, we next \textcolor{blue}{briefly} compare the results of PWEM, RDIL, and the ensemble filter with a voting ensemble composed of the same learning algorithms used for RDIL and the ensemble filter.
% Each learning algorithm has an equally-weighted vote and the predicted class with the most votes is chosen.
% The results are shown in Table \ref{table:voting}.
% For all noise levels, the voting ensemble achieves significantly higher classification accuracy for all of the investigated learning algorithms except for C4.5 with RDIL with 40\% noise which is not significant.
% Thus, a voting ensemble exhibits robustness to noise.
% 
% \begin{table}
% \centering
% \caption{A comparison of a voting ensemble against the investigated noise handling approaches for the 5 considered learning algorithm across various noise levels.}
% \setlength{\tabcolsep}{1.85pt}
% \begin{tabular}{l|ccccc||ccccc}
% noise  & 0\% & 10\% & 20\% & 30\% & 40\% & 0\% & 10\% & 20\% & 30\% & 40\% \\
% \hline
% Vote & 83.15 & 81.83 & 79.62 & 76.81 & 71.71 \\
% \multicolumn{11}{c}{}\\
% & \multicolumn{5}{c||}{C4.5}& \multicolumn{5}{c}{5NN} \\
% % noise & 0 & 0.1 & 0.2 & 0.3 & 0.4 & 0 & 0.1 & 0.2 & 0.3 & 0.4\\
% \hline
% RDIL & 78.24 & 77.1 & 76.19 & 74.07 & 70.84 & 78.44 & 77.86 & 76.76 & 74.79 & 70.30 \\
% \textit{p}-val & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & 0.170 & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & \textbf{0.001} & \textbf{0.007} \\
% Count & \begin{footnotesize}45,1,7\end{footnotesize} & \begin{footnotesize}43,0,10\end{footnotesize} & \begin{footnotesize}38,0,15\end{footnotesize} & \begin{footnotesize}36,0,17\end{footnotesize} & \begin{footnotesize}29,0,24\end{footnotesize} & \begin{footnotesize}43,4,6\end{footnotesize} & \begin{footnotesize}43,0,10\end{footnotesize} & \begin{footnotesize}37,2,14\end{footnotesize} & \begin{footnotesize}36,1,16\end{footnotesize} & \begin{footnotesize}33,3,17\end{footnotesize} \\
% \hline
% PWEM & 76.58 & 74.64 & 73.49 & 71.07 & 68.39 & 77.74 & 77.26 & 75.83 & 73.25 & 66.80 \\
% \textit{p}-val & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & \textbf{0.021} & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ \\
% Count & \begin{footnotesize}42,3,8\end{footnotesize} & \begin{footnotesize}45,0,8\end{footnotesize} & \begin{footnotesize}42,0,11\end{footnotesize} & \begin{footnotesize}37,0,16\end{footnotesize} & \begin{footnotesize}29,0,24\end{footnotesize} & \begin{footnotesize}42,1,10\end{footnotesize} & \begin{footnotesize}45,0,8\end{footnotesize} & \begin{footnotesize}40,0,13\end{footnotesize} & \begin{footnotesize}42,0,11\end{footnotesize} & \begin{footnotesize}39,2,12\end{footnotesize} \\
% \hline
% $\mathcal{L}$-filter & 79.43 & 78.23 & 76.5 & 73.25 & 68.91 & 79.12 & 78.07 & 76.33 & 74.07 & 69.30 \\
% \textit{p}-val & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ \\
% Count & \begin{footnotesize}44,3,6\end{footnotesize} & \begin{footnotesize}43,0,10\end{footnotesize} & \begin{footnotesize}41,0,12\end{footnotesize} & \begin{footnotesize}45,1,7\end{footnotesize} & \begin{footnotesize}36,0,17\end{footnotesize} & \begin{footnotesize}46,2,5\end{footnotesize} & \begin{footnotesize}41,2,10\end{footnotesize} & \begin{footnotesize}41,2,10\end{footnotesize} & \begin{footnotesize}41,1,11\end{footnotesize} & \begin{footnotesize}36,0,17\end{footnotesize} \\
% % \hline
% % RENN & 76.67 & 75.95 & 74.54 & 71.76 & 66.01 & 76.68 & 75.66 & 74 & 71.47 & 65.68 \\
% % \textit{p}-val & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ \\
% % Count & \begin{footnotesize}49,1,3\end{footnotesize} & \begin{footnotesize}49,0,4\end{footnotesize} & \begin{footnotesize}48,0,5\end{footnotesize} & \begin{footnotesize}48,0,5\end{footnotesize} & \begin{footnotesize}46,0,7\end{footnotesize} & \begin{footnotesize}48,2,3\end{footnotesize} & \begin{footnotesize}45,1,7\end{footnotesize} & \begin{footnotesize}43,0,10\end{footnotesize} & \begin{footnotesize}41,0,12\end{footnotesize} & \begin{footnotesize}41,0,12\end{footnotesize} \\
% \multicolumn{11}{c}{}\\
% & \multicolumn{5}{c||}{MLP}& \multicolumn{5}{c}{Random Forest} \\
% % noise & 0 & 0.1 & 0.2 & 0.3 & 0.4 & 0 & 0.1 & 0.2 & 0.3 & 0.4\\
% \hline
% RDIL & 82.27 & 80.7 & 78.4 & 75.06 & 69.12 & 80.63 & 79.72 & 78.06 & 75.87 & 70.70 \\
% \textit{p}-val & \textbf{0.001} & $\textbf{0.000}$ & \textbf{0.002} & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & \textbf{0.002} & \textbf{0.006} \\
% Count & \begin{footnotesize}36,3,14\end{footnotesize} & \begin{footnotesize}41,0,12\end{footnotesize} & \begin{footnotesize}35,2,16\end{footnotesize} & \begin{footnotesize}35,1,17\end{footnotesize} & \begin{footnotesize}36,0,17\end{footnotesize} & \begin{footnotesize}43,0,10\end{footnotesize} & \begin{footnotesize}40,3,10\end{footnotesize} & \begin{footnotesize}40,2,11\end{footnotesize} & \begin{footnotesize}35,1,17\end{footnotesize} & \begin{footnotesize}30,0,23\end{footnotesize} \\
% \hline
% PWEM & 82.89 & 79.71 & 76.4 & 71.8 & 65.73 & 81.47 & 78.64 & 76.23 & 72.36 & 65.79 \\
% \textit{p}-val & \textbf{0.029} & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ \\
% Count & \begin{footnotesize}33,2,18\end{footnotesize} & \begin{footnotesize}42,1,10\end{footnotesize} & \begin{footnotesize}42,0,11\end{footnotesize} & \begin{footnotesize}42,0,11\end{footnotesize} & \begin{footnotesize}43,2,8\end{footnotesize} & \begin{footnotesize}41,0,12\end{footnotesize} & \begin{footnotesize}45,0,8\end{footnotesize} & \begin{footnotesize}46,1,6\end{footnotesize} & \begin{footnotesize}43,1,9\end{footnotesize} & \begin{footnotesize}49,0,4\end{footnotesize} \\
% \hline
% $\mathcal{L}$-filter & 81.81 & 80.64 & 78.12 & 74.71 & 69.25 & 81.46 & 79.69 & 77.99 & 75.19 & 69.78 \\
% \textit{p}-val & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ \\
% Count & \begin{footnotesize}39,5,9\end{footnotesize} & \begin{footnotesize}38,1,14\end{footnotesize} & \begin{footnotesize}40,0,13\end{footnotesize} & \begin{footnotesize}39,2,12\end{footnotesize} & \begin{footnotesize}35,0,18\end{footnotesize} & \begin{footnotesize}40,4,9\end{footnotesize} & \begin{footnotesize}48,0,5\end{footnotesize} & \begin{footnotesize}44,0,9\end{footnotesize} & \begin{footnotesize}41,1,11\end{footnotesize} & \begin{footnotesize}39,0,14\end{footnotesize} \\
% % RENN & 78.75 & 77.73 & 76.11 & 73.16 & 67.20 & 77.94 & 76.98 & 75.34 & 73.02 & 67.14 \\
% % \textit{p}-val & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ \\
% % Count & \begin{footnotesize}45,1,7\end{footnotesize} & \begin{footnotesize}41,1,11\end{footnotesize} & \begin{footnotesize}39,0,14\end{footnotesize} & \begin{footnotesize}40,0,13\end{footnotesize} & \begin{footnotesize}41,0,12\end{footnotesize} & \begin{footnotesize}51,0,2\end{footnotesize} & \begin{footnotesize}46,1,6\end{footnotesize} & \begin{footnotesize}42,3,8\end{footnotesize} & \begin{footnotesize}41,0,12\end{footnotesize} & \begin{footnotesize}40,2,11\end{footnotesize} \\
% \multicolumn{11}{c}{}\\
% & \multicolumn{5}{c||}{RIPPER}&  \\
% % noise & 0 & 0.1 & 0.2 & 0.3 & 0.4 \\
% \cline{1-6}
% RDIL & 77.77 & 76.54 & 75.47 & 73.38 & 69.53 \\
% \textit{p}-val & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & \textbf{0.017} \\
% Count & \begin{footnotesize}46,2,5\end{footnotesize} & \begin{footnotesize}47,0,6\end{footnotesize} & \begin{footnotesize}40,1,12\end{footnotesize} & \begin{footnotesize}39,0,14\end{footnotesize} & \begin{footnotesize}32,1,20\end{footnotesize} \\
% \cline{1-6}
% PWEM & 74.35 & 72.11 & 70.89 & 68.81 & 65.14 \\
% \textit{p}-val & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ \\
% Count & \begin{footnotesize}45,2,6\end{footnotesize} & \begin{footnotesize}48,1,4\end{footnotesize} & \begin{footnotesize}45,1,7\end{footnotesize} & \begin{footnotesize}43,0,10\end{footnotesize} & \begin{footnotesize}36,0,17\end{footnotesize} \\
% \cline{1-6}
% $\mathcal{L}$-filter & 78.89 & 77.71 & 76.34 & 73.85 & 69.73 \\
% \textit{p}-val & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & \textbf{0.002} \\
% Count & \begin{footnotesize}46,2,5\end{footnotesize} & \begin{footnotesize}48,0,5\end{footnotesize} & \begin{footnotesize}43,0,10\end{footnotesize} & \begin{footnotesize}41,1,11\end{footnotesize} & \begin{footnotesize}36,1,16\end{footnotesize} \\
% % RENN & 76.52 & 75.75 & 74.46 & 71.65 & 66.67 \\
% % \textit{p}-val & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ & $\textbf{0.000}$ \\
% % Count & \begin{footnotesize}50,1,2\end{footnotesize} & \begin{footnotesize}48,0,5\end{footnotesize} & \begin{footnotesize}46,0,7\end{footnotesize} & \begin{footnotesize}48,0,5\end{footnotesize} & \begin{footnotesize}44,0,9\end{footnotesize} \\
% 
% \end{tabular}
% \label{table:voting}
% \end{table}
% 
% \textcolor{blue}{The increase in accuracy by the voting ensemble is not too surprising since ensembles have been shown to perform better than the any one of its constituent base classifiers \cite{Dietterich2000}.
% We focus our attention on the MLP and random forest learning algorithms since they obtain the highest classification accuracy of the considered learning algorithms.
% Although the increase in accuracy by a voting ensemble is significant, it is generally within 1\% of the accuracy achieved by a MLP or a random forest with RDIL.
% A voting ensemble is a safe choice to use in general.
% However, if the accuracy from the learning algorithm is chosen for a particular data set that produces the highest classification accuracy then the average accuracy from RDIL, the ensemble filter, and PWEM is higher than that achieved by the voting ensemble.
% % These results once again emphasize the important of estimating $p(y_i|x_i)$ as the voting ensemble is composed of the learning algorithms used to estimate $p(y_i|x_i)$.
% Although this is impractical it does show that noise handling in single learning algorithm can out perform a voting ensemble.
% Investigating the robustness of a voting ensemble to noise is topic of on-going research.}
% 
% % Dietterich \cite{Dietterich2000} puts forth three reasons why ensembles may perform well and help overcome some of the short-comings of common learning algorithms which may help to understand why a voting ensemble exhibits a robustness to noise:
% % 1) statistical,
% % 2) computational, and
% % 3) representational.
% % This discussion refers to Figure \ref{figure:ensemble} which shows (a) a single induced linear hypothesis (the dashed line) and the ``actual'' hypothesis (the solid line) and (b) the possible effects of using an ensemble.
% % Each dashed line in Figure \ref{figure:ensemble}b represents an induced linear hypothesis of a base learner and the solid curve represents the final hypothesis of the ensemble after combining the hypotheses from the base learners.
% % The first reason, statistical, states that an ensemble averages the induced hypotheses to create less biased hypothesis and in turn reduces the risk of choosing the wrong hypothesis.
% % As can be seen in Figure \ref{figure:ensemble}b, none of the induced hypotheses is the ``actual'' hypothesis.
% % The hypothesis used by the ensemble is an average of the hypotheses induced by the base learners.
% % \textcolor{blue}{The second reason, computational, refers to the fact that some learning algorithms achieve lower classification accuracy on certain data sets.}
% % Averaging the results of multiple hypotheses may approximate the true function better than any individual hypothesis.
% % The third reason, representational, explains that averaging the induced hypotheses from the base learners may produce a hypothesis that cannot be represented by any of the current hypotheses in $\mathcal{H}$.
% % % For example, in Figure \ref{figure:ensemble}b, the base learners are only capable of representing lines yet by ensembling the induced hypotheses, the hypothesis of the ensemble is a curve.
% % Because of the aforementioned advantages of ensembles, ensembles are able to overcome the problems caused by class noise given that the ensemble is composed of a diverse set of learning algorithms.
% % If all of the learning algorithms were similar, then the benefits of an ensemble would drastically decrease.
% % In fact, when we used an ensemble composed of only 3 learning algorithms (5NN, MLP, and random forest), there was a significant decrease in classification accuracy compared to using an ensemble composed of all nine of the diverse learning algorithms.
% % A further investigation of the diversity of the base learning algorithms and its impact on noise and classification accuracy will be pursued in future work.
% % 
% % \begin{figure}
% % \centering
% % \begin{tabular}{|c|c|}
% % \hline
% % \input{classification4}  & 
% % % \hline
% % % \multicolumn{1}{c}{a}\\
% % % \multicolumn{1}{c}{} \\
% % \input{classification3}\\
% % \hline
% % \multicolumn{1}{c}{a}&\multicolumn{1}{c}{b}\\
% %  %\multicolumn{1}{c}{b} \\
% % \end{tabular}
% % \caption{Hypothetical, 2-dimensional data set showing a) a single induced hypothesis and b) the possible effect of using an ensemble.}
% % \label{figure:ensemble}
% % \end{figure}
% % 
% % Finally, we examine the effects of constructing a voting ensemble with base classifiers that were induced using noise handling techniques.
% % The percent reduction in error for a voting ensemble constructed with classifiers that were induced using a noise handling technique compared with a voting ensemble composed of base classifiers that were induced without noise handling is shown in Figure \ref{figure:vote}.
% % The accuracies and \textit{p}-values comparing the use of noise handling in a voting ensemble is in Table \ref{table:voteSig} in \ref{section:voteSig}.
% % The accuracy from a voting ensemble without noise handling is significantly higher than using any of the noise handling approaches when no artificial noise is added to the data set.
% % With artificial noise levels of 0.3 and 0.4, RDIL and the ensemble filter significantly increase the classification accuracy.
% % Thus, only when there are large amounts of noise in the data set does a voting ensemble benefit from the use of noise handling techniques.
% % % This also shows that an ensemble is robust to noise.
% % 
% % \begin{figure*}
% % \begin{center}
% % \input{ensemble.tex}
% % \end{center}
% % \caption{The average reduction of error for using noise handling techniques in a voting ensemble.}
% % \label{figure:vote}
% % \end{figure*}


\section{Conclusions}
\label{section:conclusions}

In this paper we examined handling detrimental instances using the hypotheses from multiple learning algorithms. 
We introduced \textit{reduced detrimental instance learning} (RDIL) which weights each instance based on an approximation of $p(\hat{y}_i|x_i)$.
We examined RDIL on a set of 5 learning algorithms and 54 data sets.
We found that a better estimate of $p(\hat{y}_i|x_i)$ leads to better detrimentality handling in both instance weighting and filtering.
% Future work will include a more thorough examination.
Weighting the instances avoids having to spend extra computational time and having to use training instances to select a threshold for filtering when using an ensemble filter.
Instance weighting has the greatest effect on learning algorithms where every instance can affect the classification boundary and the training instances are considered individually, such as multilayer perceptrons trained with backpropagation (MLPs).
On the other hand, instance filtering had a more significant impact on the C4.5, 5-NN, Random Forest, and RIPPER learning algorithms with no artificial noise.
However, instance weighting was shown to be preferable to filtering for the examined learning algorithms when there are high amounts of noise.
An analysis of when to use a particular noise handling technique is a direction for future work.
% For the C4.5, 5NN, and random forest algorithms, instance weighting significantly increased the classification accuracy.
% There was not a significant difference between weighting and filtering for MLPs and RIPPER with high amounts of noise.}

% Despite the increase in accuracy exhibited by weighting the instances, a voting ensemble composed of a diverse set of learning algorithms (as determined using unsupervised meta-learning \cite{Lee2011}) with each learning algorithm having an equally-weighted vote achieved a significantly higher classification accuracy for all of the learning algorithms and noise handling techniques for all of the examined noise levels over a broad set of data sets.
% Thus, a voting ensemble exhibits a robustness to noise that the individual constituent learning algorithms do not possess.
% \textcolor{blue}{However, there are data sets where noise has a more significant impact than others.
% A voting ensemble would produce greater results on those data sets where the noise has a low impact.
% The robustness of a voting ensemble to noise is a topic of future work.}

% conference papers do not normally have an appendix


% use section* for acknowledgement
% \section*{Acknowledgment}
% 
% 
% The authors would like to thank...
% 




% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
%\IEEEtriggeratref{8}
% The "triggered" command can be changed if desired:
%\IEEEtriggercmd{\enlargethispage{-5in}}

% references section

% can use a bibliography generated by BibTeX as a .bbl file
% BibTeX documentation can be easily obtained at:
% http://www.ctan.org/tex-archive/biblio/bibtex/contrib/doc/
% The IEEEtran BibTeX style support page is at:
% http://www.michaelshell.org/tex/ieeetran/bibtex/
\bibliographystyle{IEEEtran}
% argument is your BibTeX string definitions and bibliography database(s)
% \bibliography{IEEEabrv,../../../bibliography}

% Generated by IEEEtran.bst, version: 1.12 (2007/01/11)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl

\bibitem{Elman1993}
J.~L. Elman, ``Learning and development in neural networks: The importance of
  starting small,'' \emph{Cognition}, vol.~48, pp. 71--99, 1993.

\bibitem{Bengio2009}
Y.~Bengio, J.~Louradour, R.~Collobert, and J.~Weston, ``Curriculum learning,''
  in \emph{Proceedings of the 26th International Conference on Machine
  Learning}.\hskip 1em plus 0.5em minus 0.4em\relax ACM, 2009, pp. 41--48.

\bibitem{Smith_CL}
\BIBentryALTinterwordspacing
M.~R. Smith and T.~Martinez, ``A comparative evaluation of curriculum learning
  with filtering and boosting in supervised classification problems,''
  \emph{Computational Intelligence}, p. to appear, 2014. [Online]. Available:
  \url{http://arxiv.org/pdf/1312.4986}
\BIBentrySTDinterwordspacing

\bibitem{Rebbapragada2007}
U.~Rebbapragada and C.~E. Brodley, ``Class noise mitigation through instance
  weighting,'' in \emph{Proceedings of the 18th European Conference on Machine
  Learning}, 2007, pp. 708--715.

\bibitem{Brodley1999}
C.~E. Brodley and M.~A. Friedl, ``Identifying mislabeled training data,''
  \emph{Journal of Artificial Intelligence Research}, vol.~11, pp. 131--167,
  1999.

\bibitem{Tomek1976}
I.~Tomek, ``An experiment with the edited nearest-neighbor rule,'' \emph{IEEE
  Transactions on Systems, Man, and Cybernetics}, vol.~6, pp. 448--452, 1976.

\bibitem{Nettleton2010}
D.~F. Nettleton, A.~Orriols-Puig, and A.~Fornells, ``A study of the effect of
  different types of noise on the precision of supervised learning
  techniques,'' \emph{Artificial Intelligence Review}, vol.~33, no.~4, pp.
  275--306, 2010.

\bibitem{Frenay2014}
B.~Fr\'{e}nay and M.~Verleysen, ``Classification in the presence of label
  noise: a survey,'' \emph{IEEE Transactions on Neural Networks and Learning
  Systems}, vol.~25, no.~5, pp. 845--869, 2014.

\bibitem{bishop2006pattern}
C.~M. Bishop and N.~M. Nasrabadi, \emph{Pattern Recognition and Machine
  Learning}.\hskip 1em plus 0.5em minus 0.4em\relax springer New York, 2006,
  vol.~1.

\bibitem{Schapire1990}
R.~E. Schapire, ``The strength of weak learnability,'' \emph{Machine Learning},
  vol.~5, pp. 197--227, 1990.

\bibitem{Servedio2003}
R.~A. Servedio, ``Smooth boosting and learning with malicious noise,''
  \emph{Journal of Machine Learning Research}, vol.~4, pp. 633--648, 2003.

\bibitem{Gamberger2000}
D.~Gamberger, N.~Lavra\v{c}, and S.~D\v{z}eroski, ``Noise detection and
  elimination in data preprocessing: Experiments in medical domains,''
  \emph{Applied Artificial Intelligence}, vol.~14, no.~2, pp. 205--223, 2000.

\bibitem{Smith2011}
M.~R. Smith and T.~Martinez, ``Improving classification accuracy by identifying
  and removing instances that should be misclassified,'' in \emph{Proceedings
  of the IEEE International Joint Conference on Neural Networks}, 2011, pp.
  2690--2697.

\bibitem{John95}
G.~H. John, ``Robust decision trees: Removing outliers from databases,'' in
  \emph{Knowledge Discovery and Data Mining}, 1995, pp. 174--179.

\bibitem{Segata2009}
N.~Segata, E.~Blanzieri, and P.~Cunningham, ``A scalable noise reduction
  technique for large case-based systems,'' in \emph{Proceedings of the 8th
  International Conference on Case-Based Reasoning: Case-Based Reasoning
  Research and Development}, 2009, pp. 328--342.

\bibitem{Teng2003}
C.~Teng, ``Combining noise correction with feature selection,'' in \emph{Data
  Warehousing and Knowledge Discovery}, ser. Lecture Notes in Computer Science,
  2003, vol. 2737, pp. 340--349.

\bibitem{Lawrence2001}
N.~D. Lawrence and B.~Sch{\"o}lkopf, ``Estimating a kernel fisher discriminant
  in the presence of label noise,'' in \emph{In Proceedings of the 18th
  International Conference on Machine Learning}, 2001, pp. 306--313.

\bibitem{Ng+Jordan:2001}
A.~Y. Ng and M.~I. Jordan, ``On discriminative vs. generative classifiers: A
  comparison of logistic regression and naive bayes,'' in \emph{Advances in
  Neural Information Processing Systems 14}, 2001, pp. 841--848.

\bibitem{Lee2011}
J.~Lee and C.~Giraud-Carrier, ``A metric for unsupervised metalearning,''
  \emph{Intelligent Data Analysis}, vol.~15, no.~6, pp. 827--841, 2011.

\bibitem{Peterson2005}
A.~H. Peterson and T.~R. Martinez, ``Estimating the potential for combining
  learning models,'' in \emph{Proceedings of the ICML Workshop on
  Meta-Learning}, 2005, pp. 68--75.

\bibitem{weka2009}
M.~Hall, E.~Frank, G.~Holmes, B.~Pfahringer, P.~Reutemann, and I.~H. Witten,
  ``The weka data mining software: an update,'' \emph{SIGKDD Explorations
  Newsletter}, vol.~11, no.~1, pp. 10--18, 2009.

\bibitem{Kass1995}
R.~E. Kass and L.~Wassermann, ``\BIBforeignlanguage{English}{{A reference
  Bayesian test for nested hypotheses and its relationship to the Schwarz
  criterion}},'' \emph{\BIBforeignlanguage{English}{Journal of the American
  Statistical Association}}, vol.~90, no. 431, pp. 928--934, 1995.

\bibitem{UCI_dataRepo}
\BIBentryALTinterwordspacing
K.~Bache and M.~Lichman, ``{UCI} machine learning repository,'' 2013. [Online].
  Available: \url{http://archive.ics.uci.edu/ml}
\BIBentrySTDinterwordspacing

\bibitem{Demsar2006}
J.~Dem\v{s}ar, ``Statistical comparisons of classifiers over multiple data
  sets,'' \emph{Journal of Machine Learning Research}, vol.~7, pp. 1--30, 2006.

\end{thebibliography}




% that's all folks
\end{document}



1406.2237.extracted_bib
{Elman1993 J.~L. Elman, ``Learning and development in neural networks: The importance of starting small,'' Cognition, vol.~48, pp. 71--99, 1993.
1406.2237.extracted_bib
{Bengio2009 Y.~Bengio, J.~Louradour, R.~Collobert, and J.~Weston, ``Curriculum learning,'' in Proceedings of the 26th International Conference on Machine Learning.\hskip 1em plus 0.5em minus 0.4em\relax ACM, 2009, pp. 41--48.
1406.2237.extracted_bib
{Smith_CL \BIBentryALTinterwordspacing M.~R. Smith and T.~Martinez, ``A comparative evaluation of curriculum learning with filtering and boosting in supervised classification problems,'' Computational Intelligence, p. to appear, 2014. [Online]. Available: \url{http://arxiv.org/pdf/1312.4986 \BIBentrySTDinterwordspacing
1406.2237.extracted_bib
{Rebbapragada2007 U.~Rebbapragada and C.~E. Brodley, ``Class noise mitigation through instance weighting,'' in Proceedings of the 18th European Conference on Machine Learning, 2007, pp. 708--715.
1406.2237.extracted_bib
{Brodley1999 C.~E. Brodley and M.~A. Friedl, ``Identifying mislabeled training data,'' Journal of Artificial Intelligence Research, vol.~11, pp. 131--167, 1999.
1406.2237.extracted_bib
{Tomek1976 I.~Tomek, ``An experiment with the edited nearest-neighbor rule,'' IEEE Transactions on Systems, Man, and Cybernetics, vol.~6, pp. 448--452, 1976.
1406.2237.extracted_bib
{Nettleton2010 D.~F. Nettleton, A.~Orriols-Puig, and A.~Fornells, ``A study of the effect of different types of noise on the precision of supervised learning techniques,'' Artificial Intelligence Review, vol.~33, no.~4, pp. 275--306, 2010.
1406.2237.extracted_bib
{Frenay2014 B.~Fr\'{enay and M.~Verleysen, ``Classification in the presence of label noise: a survey,'' IEEE Transactions on Neural Networks and Learning Systems, vol.~25, no.~5, pp. 845--869, 2014.
1406.2237.extracted_bib
{bishop2006pattern C.~M. Bishop and N.~M. Nasrabadi, Pattern Recognition and Machine Learning.\hskip 1em plus 0.5em minus 0.4em\relax springer New York, 2006, vol.~1.
1406.2237.extracted_bib
{Schapire1990 R.~E. Schapire, ``The strength of weak learnability,'' Machine Learning, vol.~5, pp. 197--227, 1990.
1406.2237.extracted_bib
{Servedio2003 R.~A. Servedio, ``Smooth boosting and learning with malicious noise,'' Journal of Machine Learning Research, vol.~4, pp. 633--648, 2003.
1406.2237.extracted_bib
{Gamberger2000 D.~Gamberger, N.~Lavra\v{c, and S.~D\v{zeroski, ``Noise detection and elimination in data preprocessing: Experiments in medical domains,'' Applied Artificial Intelligence, vol.~14, no.~2, pp. 205--223, 2000.
1406.2237.extracted_bib
{Smith2011 M.~R. Smith and T.~Martinez, ``Improving classification accuracy by identifying and removing instances that should be misclassified,'' in Proceedings of the IEEE International Joint Conference on Neural Networks, 2011, pp. 2690--2697.
1406.2237.extracted_bib
{John95 G.~H. John, ``Robust decision trees: Removing outliers from databases,'' in Knowledge Discovery and Data Mining, 1995, pp. 174--179.
1406.2237.extracted_bib
{Segata2009 N.~Segata, E.~Blanzieri, and P.~Cunningham, ``A scalable noise reduction technique for large case-based systems,'' in Proceedings of the 8th International Conference on Case-Based Reasoning: Case-Based Reasoning Research and Development, 2009, pp. 328--342.
1406.2237.extracted_bib
{Teng2003 C.~Teng, ``Combining noise correction with feature selection,'' in Data Warehousing and Knowledge Discovery, ser. Lecture Notes in Computer Science, 2003, vol. 2737, pp. 340--349.
1406.2237.extracted_bib
{Lawrence2001 N.~D. Lawrence and B.~Sch{\"olkopf, ``Estimating a kernel fisher discriminant in the presence of label noise,'' in In Proceedings of the 18th International Conference on Machine Learning, 2001, pp. 306--313.
1406.2237.extracted_bib
{Ng+Jordan:2001 A.~Y. Ng and M.~I. Jordan, ``On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes,'' in Advances in Neural Information Processing Systems 14, 2001, pp. 841--848.
1406.2237.extracted_bib
{Lee2011 J.~Lee and C.~Giraud-Carrier, ``A metric for unsupervised metalearning,'' Intelligent Data Analysis, vol.~15, no.~6, pp. 827--841, 2011.
1406.2237.extracted_bib
{Peterson2005 A.~H. Peterson and T.~R. Martinez, ``Estimating the potential for combining learning models,'' in Proceedings of the ICML Workshop on Meta-Learning, 2005, pp. 68--75.
1406.2237.extracted_bib
{weka2009 M.~Hall, E.~Frank, G.~Holmes, B.~Pfahringer, P.~Reutemann, and I.~H. Witten, ``The weka data mining software: an update,'' SIGKDD Explorations Newsletter, vol.~11, no.~1, pp. 10--18, 2009.
1406.2237.extracted_bib
{Kass1995 R.~E. Kass and L.~Wassermann, ``\BIBforeignlanguage{English{{A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion,'' \BIBforeignlanguage{English{Journal of the American Statistical Association, vol.~90, no. 431, pp. 928--934, 1995.
1406.2237.extracted_bib
{UCI_dataRepo \BIBentryALTinterwordspacing K.~Bache and M.~Lichman, ``{UCI machine learning repository,'' 2013. [Online]. Available: \url{http://archive.ics.uci.edu/ml \BIBentrySTDinterwordspacing
1406.2237.extracted_bib
{Demsar2006 J.~Dem\v{sar, ``Statistical comparisons of classifiers over multiple data sets,'' Journal of Machine Learning Research, vol.~7, pp. 1--30, 2006.