Background
==========

In recent years there has been a surge of interest in computer implementations of automatic beat classification algorithms. The impetus for this research stems partly from the advances and miniaturisation of electronics, which allows portable, wearable and implantable devices to perform greater functionality than was achievable in the past, but also from the desire to automate tasks currently performed by intensive care and operating room staff.

There are several principles on which classifiers operate, and many variations and implementations of each. With the large number of published algorithms comes the need to analyse and compare their performance. The literature to date, compares techniques on a low level, parameter-by-parameter basis \[[@B1]-[@B3]\]. ANSI standard EC57 \[[@B4]\] attempts to formalise methods for reporting such comparisons. These comparisons are of interest to those working on the development of new algorithms or the enhancement of existing ones, but are of little interest to a clinician when making a decision about which algorithm suits his purpose.

Problems with current classification methods
--------------------------------------------

The contemporary method for reporting the performance of a beat classification algorithm involves beat-by-beat comparisons between the class as indicated by the algorithm and as indicated by some reference. The MIT-BIH database \[[@B5]\] is a commonly used reference source. Performance is reported either by a table giving counts of correctly and incorrectly classified beats, or by way of statistics inferred from such a table. Common statistics are sensitivity (Se), specificity (Sp) and positive predictivity (+P). These are defined as:

$$\begin{matrix}
{\text{Se} = \frac{\text{TP}}{\text{TP} + \text{FN}},} \\
{\text{Sp} = \frac{\text{TN}}{\text{TN} + \text{FP}},} \\
{+ \text{P} = \frac{\text{TP}}{\text{TP} + \text{FP}},} \\
\end{matrix}$$

where:

TP = number of true positives

FP = number of false positives

TN = number of true negatives

FN = number of false negatives.

However these values are defined only for *binary*classification, and do not readily lend themselves to problems involving more than two classes, {*ω*~1~, *ω*~2~\... *ω*~*n*~}. Nevertheless, in the literature, one often sees beat classifier performance reports where sensitivity and specificity are freely quoted. Whilst the definitions of these measures for the context are normally not given, they appear to use the following extended definitions:

TP~*j*~= number of beats correctly identified as belonging to class *ω*~*j*~

FP~*j*~= number of beats incorrectly identified as belonging to class *ω*~*j*~

TN~*j*~= number of beats correctly identified as belonging to a class other than *ω*~*j*~

FN~*j*~= number of beats incorrectly identified as belonging to a class other than *ω*~*j*~

and hence Se~*j*~, Sp~*j*~and +P~*j*~can be defined accordingly.

A number of problems become apparent when using such statistics to evaluate the performance of beat classifiers:

1\. They do not take into account the *a priori*probabilities of the beat classes.

2\. They do not take into account the relative costs of false classification.

3\. They can be presented only as a multi-dimensional value, even where only two classes are being considered. There is no obvious single ordinal value.

Problem 1 has been recognised in the medical literature \[[@B6]\]. We are not aware of any previous attempt to deal with problem 2. Problem 3 makes these reports particularly unhelpful from the point of view of the clinician trying to compare systems with a view to adopting one for use. For an *n*class classifier, there are 2*n*scalar quantities, so ranking classifiers using these quantities is not possible.

We propose a new method, which overcomes these problems and aims to be generally useful for the quantitative comparison of beat classification schemes.

Proposed methodology
--------------------

A system\'s utility as a prognostic medical tool is a measure of the benefit afforded by selecting it against other alternatives. Choosing a system involves maximising the benefit, or alternatively, minimising the risk. A measure for the overall risk associated with making a decision based upon the output of a beat classifier is a useful measure of its performance. Risk is characterised by the probability of error and the costs associated with making a decision based upon the erroneous classification. We have used Bayesian decision theory to determine a method of calculating the risk associated with a beat classifier.

### Introduction to Bayesian risk

Bayesian decision theory is presented in many texts on statistics and classification theory \[[@B7],[@B8]\] and will be introduced here only briefly. In a system which is claimed to recognise *n*different classes of beats {*ω*~1~, *ω*~2~\...*ω*~*n*~}, there are *n*possible outputs, {*α*~1~, *α*~2~\... *α*~*n*~}. Bayes rule states:

$$\left. P(\omega_{j} \middle| \alpha_{k}) = \frac{\left. P(\alpha_{k} \middle| \omega_{j})P(\omega_{j}) \right.}{P(\alpha_{k})}. \right.$$

Since {*α*~1~, *α*~2~\... *α*~*n*~} are mutually exclusive and $P( \cup_{i = 1}^{n}\alpha_{i}) = 1$ then

$$P(\alpha_{k}) = {\sum\limits_{i = 1}^{n}\left. P(\alpha_{k} \middle| \omega_{i})P(\omega_{i}) \right.}.$$

The quantity *P*(*ω*~*j*~) is called the *a priori*probability, *P*(*α*~*k*~\|*ω*~*j*~) the *likelihood*or *class conditional*probability and *P*(*ω*~*j*~\|*α*~*k*~) the *a posteriori*probability. Note that *P*(*α*~*j*~\|*ω*~*j*~) ≡ Se~*j*~. From (1) and (2) we can write:

$$\left. P(\omega_{j} \middle| \alpha_{k}) = \frac{\left. P(\alpha_{k} \middle| \omega_{j})P(\omega_{j}) \right.}{\sum\limits_{i = 1}^{n}\left. P(\alpha_{k} \middle| \omega_{i})P(\omega_{i}) \right.}. \right.$$

Let *λ*(*α*~*k*~\|*ω*~*j*~) be the cost incurred for making decision *α*~*k*~when *ω*~*j*~is the true beat class. Therefore, the risk of making decision *α*~*k*~based upon the classifier\'s output (the risk of reliance) is:

$$R(\alpha_{k}) = {\sum\limits_{j = 1}^{n}\left. \lambda(\alpha_{k} \middle| \omega_{j})P(\omega_{j} \middle| \alpha_{k}) \right.}.$$

Combining the above gives

$$R(\alpha_{k}) = \frac{\sum\limits_{j = 1}^{n}\left. \lambda(\alpha_{k} \middle| \omega_{j})P(\alpha_{k} \middle| \omega_{j})P(\omega_{j}) \right.}{\sum\limits_{i = 1}^{n}\left. P(\alpha_{k} \middle| \omega_{i})P(\omega_{i}) \right.}.$$

The overall risk of relying on a classifier is

$$\mathcal{R} = {\sum\limits_{k = 1}^{n}{R(\alpha_{k})P(\alpha_{k})}},$$

or equivalently

$$\mathcal{R} = {\sum\limits_{k = 1}^{n}{{\sum\limits_{j = 1}^{n}\left. \lambda(\alpha_{k} \middle| \omega_{j})P(\alpha_{k} \middle| \omega_{j})P(\omega_{j}) \right.}.}}$$

We propose that {*R*(*α*~1~), *R*(*α*~2~) \... *R*(*α*~*n*~)} be used in the consideration of a classifier\'s utility, and that $\mathcal{R}$ be used for overall rating of classifiers. $\mathcal{R}$ has the range (0, ∞) and its units are dollars (or whatever units have been chosen for *λ*(*α*~*k*~\|*ω*~*j*~)). We envisage that a unitless measure, having the range (0, 1) is more useful in many circumstances. Accordingly, we also propose a normalised metric:

$$\hat{\mathcal{R}} = \frac{\mathcal{R}}{\mathcal{R}_{\max}},$$

where $\mathcal{R}_{\max}$ is the value obtained from equation (5) when the class conditional probabilities are set to

$$\left. P(\alpha_{k} \middle| \omega_{j}) = \begin{cases}
1 & \left. \text{if~}\lambda(\alpha_{k} \middle| \omega_{j}) = \right. \\
 & \left. \underset{i}{\max}\lambda(\alpha_{i} \middle| \omega_{j}) \right. \\
0 & \text{otherwise} \\
\end{cases}. \right.$$

Thus, a perfect classifier has a $\hat{\mathcal{R}}$ value of zero and, at the opposite extreme, unity.

Results
=======

ANSI EC57 section 4.3 identifies 5 classes of beats which are recommended in performance reports, *viz:*normal beats, Supra-Ventricular Ectopic Beats, Ventricular Ectopic Beats, fusions of normal and Ventricular Ectopic Beats and other unclassified beats. From secondary data sources, we derived *a priori*probabilities and costs of decisions for these classes. A detailed description of the derivation and the data sources are below.

Table [1](#T1){ref-type="table"} shows the *a priori*probabilities. Table [2](#T2){ref-type="table"} shows the costs. Together with the values of *P*(*α*~*k*~\|*ω*~*j*~), these tables enable the risk to be calculated. Unfortunately, in many cases the literature presents neither the values for *P*(*α*~*k*~\|*ω*~*j*~), nor the table of beat-by-beat comparisons from which they could be deduced. In the literature, we were able to find only two classifiers for which this data was reported. These are the classifiers of de Chazal *et al*. \[[@B9]\] and of Melo *et al*. \[[@B10]\]. Melo *et al*. publishes separate results for aberrated atrial premature beats. For the purposes of comparison, we have regarded aberrated and non-aberrated atrial premature beats as a single class (*ω*~SVEB~).

###### 

A priori probabilities derived from the MIT-BIH Arrhythmia Database.

  *j*        *ω*~*j*~                         *c*~*j*~   *P*(*ω*~*j*~)
  ---------- -------------------------------- ---------- ---------------
  0          Not a Beat                       577        .
  1 (N)      Normal Beat                      46097      0.9670
  2 (SVEB)   Supra Ventricular Ectopic Beat   192        0.0040
  3 (VEB)    Ventricular Ectopic Beat         1345       0.2800
  4 (F)      Fusion of Normal and VEB         13         0.0002
  5 (Q)      Unclassified Beat                0          0.00

###### 

Costs of false classification in 1000s of AUD

  *λ*         *ω*~N~    *ω*~SVEB~   *ω*~VEB~    *ω*~F~      *ω*~Q~
  ----------- --------- ----------- ----------- ----------- --------
  *α*~N~      0         38.63^1^    170.19^2^   170.19^3^   0
  *α*~SVEB~   0         0           170.19^4^   170.19^5^   0
  *α*~VEB~    2.15^6^   2.15^7^     0           0           0
  *α*~F~      2.15^8^   2.15^9^     0           0           0
  *α*~Q~      0         0           0           0           0

^1^Datum 5

^2^Datum 13

^3^Datum 13

^4^Datum 13

^5^Datum 13

^6^Datum 14

^7^Datum 14

^8^Datum 14

^9^Datum 14

The results are shown in Table [3](#T3){ref-type="table"}. In these results, the overall risk $\mathcal{R}$ is significantly lower for the Melo classifier, and the risks of reliance *R*(*α*~*i*~) is also lower for all *i*. In other words, this classifier dominates in all respects. In general however this may not be the case, and one classifier may have a lower risk of reliance for one decision whilst having a higher risk of reliance for another.

###### 

Comparative performance of two classifiers

                                                                                                                                        Chazal *et al*.   Melo *et al*.
  ------------------------------------------------------------------------------------------------------------------------------------- ----------------- ---------------
  *R*(*α*~N~)                                                                                                                           2.05              1.67
  *R*(*α*~SVEB~)                                                                                                                        65.46             17.91
  *R*(*α*~VEB~)                                                                                                                         0.07              0.03
  *R*(*α*~F~)                                                                                                                           1.79              0.71
  $R(\alpha_{k}) = {\sum\limits_{j = 1}^{n}\left. \lambda(\alpha_{k} \middle| \omega_{j})P(\omega_{j} \middle| \alpha_{k}) \right.}.$   6.67              1.71
  $\hat{\mathcal{R}}$                                                                                                                   0.126             0.033

Discussion
==========

We do not presume that the costs presented herein, or the propositions used in their calculation are universally applicable. Rather, we seek to demonstrate how, given the class conditional matrix, an ordinal measure for a beat classifier may be determined, applicable to any particular situation. Others might disagree with our cost calculation methods, or the application might demand consideration for classes other than those we have investigated. Whilst we have used monetary units to measure costs, we recognise the ethical issues raised by doing so, and our methodology imposes no requirement on the nature of the units of cost. Any unit acceptable to the community of interest may be used. In such cases, given the class conditional matrix, potential users may conduct their own studies and assess the performance of the classifier using the method we have described.

To facilitate such studies, we urge biomedical engineers to report more than the relative numbers of correct versus incorrectly classified beats, but also the identity of the misclassified beats in the form of a class conditional matrix. ANSI EC57 describes how to compile such a matrix, but makes no recommendation for its publication. It is trivial to calculate sensitivity and specificity from such a matrix if desired, and allows for more useful measures of performance as described herein. We recommend publication of the class conditional matrix and/or a table of beat-by-beat comparisons.

Conclusion
==========

The utility of a beat classifier cannot be fully quantified in terms of the number of correct and incorrect beats. Instead, the number of misclassifications *for each class*is required. Together with the *a priori*probabilities and the costs of misclassification, quantitative measures of a classifier\'s utility can be determined.

A system which claims to classify beats into more than two classes is not a binary classifier, and performance should not be reported as if it is. Instead of reporting sensitivity/specificity/predictivity for each of the *n*classes, a *n*× *n*matrix of beat classifications (the class conditional frequencies) should be reported.

Clinicians wishing to assess a classifier need to obtain estimates for the costs of misclassification, and calculate the overall risk of reliance.

Methods
=======

Equation (5) comprises the terms *P*(*α*~*k*~\|*ω*~*i*~), *P*(*ω*~*j*~) and *λ*(*α*~*k*~\|*ω*~*j*~). *P*(*α*~*k*~\|*ω*~*i*~) are parameters of the classifier and can be tested experimentally. *P*(*ω*~*j*~) and *λ*(*α*~*k*~\|*ω*~*j*~) are parameters of the classes of interest. They are respectively the *a priori*probabilities and the costs of making decisions. In this section we examine a number of secondary sources to determine values for *P*(*ω*~*j*~) and *λ*(*α*~*k*~\|*ω*~*j*~).

A priori probabilities
----------------------

We used the MIT-BIH Arrhythmia Database to extract *a priori*probabilities. The records chosen were the first group of records (numbers 100--124) from the database. We omitted the second group (numbers 200--234), since these were deliberately selected by the authors of the database to contain \"rare but clinically important phenomena\", whereas the first group was randomly selected so as to \"serve as a representative sample of the variety of waveforms and artifact that an arrhythmia detector might encounter in routine clinical use\".

Table [1](#T1){ref-type="table"} shows the data extracted from the database. Class 0 was disregarded, since these annotations are not beats, but are used to mark other interesting features in the signal. *c*~*j*~are the counts of beats of class *ω*~*j*~. *P*(*ω*~*j*~) was calculated by dividing *c*~*j*~by $\sum_{1}^{5}c_{j}$, Note that *c*~5~= 0 and we therefore conclude that beat classes other than 1--4 are sufficiently rare to have negligible effect on the utility of a system.

Extrapolation of data
---------------------

Where possible, we referred to longitudinal studies, giving data gathered over a period of 10 years or longer. In many instances, the data was available only in graphical form in which case trapezoidal approximation of integrals was used. Where data over a 10 year period was not available, we used data gathered over a shorter period and extrapolated by the following method.

We presume survival to be described by an exponential expression

where *κ*and *β*are constants for which *β*\< 0 and 0 \<*κ*≤ 1. Equation (10) implies that the mortality fraction *m*is

Integrating *m*over 10 years gives the total loss:

$${\int_{0}^{10}{1 - \kappa e^{\beta t}}}dt,$$

which can be expressed as

$$10 - \frac{e^{X}}{\beta}\lbrack e^{\beta \times 10} - 1\rbrack,$$

where *X*= ln *κ*. Thus equation (10) implies

Hence, *X*and *β*can be found from the data by linear regression of *t*against ln(*s*), and the total expected mortality over 10 years from equation (13).

Costs of incorrect classification
---------------------------------

In determining costs for incorrect decisions, we have presumed standard clinical treatment of abnormal beats, or non-treatment of normal beats according to the system\'s output. We have then investigated the costs, in monetary terms, of taking that course of action under each state of nature. We have endeavoured to report \'costs\' as the general cost to society, rather than the cost to any particular entity. A summary of these figures is presented in Table [2](#T2){ref-type="table"}. All costs have been normalised the year 2006, and are in Australian dollars except where otherwise noted.

Each application however may have different ancillary parameters, and these may affect the costs involved. The methods and figures provided in this study reflect the most general situation, as best as we could determine. Propositions we have made in calculations of costs we have stated herein, and these should be examined when applying the figures.

**Proposition 1***The clinical treatment for fusions of normal and Ventricular Ectopic Beats (class ω*~F~*) is identical to that for Ventricular Ectopic Beats*.

We make this proposition on the basis that sustained Ventricular Ectopic Beats are potentially life threatening, and must be treated. To a clinician, the fact that the polarisation of the waveform coincides with the preceding beat is merely incidental.

**Proposition 2***The maximum future projection which may affect costs of misclassification is 10 years*.

10 years is chosen as a reasonable period beyond which the advances in medical technology can be expected to invalidate the results of future prediction.

**Datum 1***The expected loss of life of a healthy subject, projected over the next 10 years, is 1.21 years*.

This datum is from the control group of Benjamin *et al*. \[[@B11]\]. This was a longitudinal study which investigated the mortality of subjects who had developed Atrial Fibrillation. We integrated the survival results presented by Figure A of that paper to determine the expected number of years of life lost by a healthy subject.

**Datum 2***The probability that a subject with Supra-Ventricular Ectopic Beats will develop Atrial Fibrillation is 0.324*.

Datum 2 comes from the results of Frost *et al*. \[[@B12]\].

**Datum 3***The expected loss of life due to a person suffering from Atrial Fibrillation is 2.69 years (projected over 10 years)*.

Datum 3 is determined from Benjamin *et al*. in a similar fashion to Datum 1: The calculations are presented in Table [4](#T4){ref-type="table"}.

###### 

Expected Loss of life due to Atrial Fibrillation

  ----------------------------------- ------
  Women without Atrial Fibrillation   0.87
  Men without Atrial Fibrillation     1.55
  Mean Average                        1.21
  Women with Atrial Fibrillation      3.75
  Men with Atrial Fibrillation        4.05
  Mean Average                        3.90
  Expected Loss Due to AF             2.69
  ----------------------------------- ------

**Datum 4***A person\'s contribution to society is \$44,320 per annum*.

Datum 4 is the mean average wage in Australia for the year 2006\[[@B13]\].

**Datum 5***The cost of misdiagnosing a Supra-Ventricular Ectopic Beat as a normal beat, λ*(*α*~N~\|*ω*~SVEB~) *is \$38,627*.

This figure is the product of Data 2, 3 and 4.

**Datum 6***The probability of initially surviving Ventricular Fibrillation is*$\frac{146}{886} = 0.164$.

From the text of Baum *et al*.\[[@B14]\], we know that, in a 3 year study of Ventricular Fibrillation cases, 146 patients out of 886 initially survived Ventricular Fibrillation.

**Datum 7***The total expected loss of life by a person who initially survives Ventricular Fibrillation is 6.33 years (projected over 10 years)*.

To derive Datum 7 we used the results of Baum *et al*.\[[@B14]\]. In Figure 2 of that paper, survival curves are presented for subjects who initially survived Ventricular Fibrillation. Survival data are presented for only 24 months. We extrapolated survival data over a 10 year period by the method described above.

**Datum 8***The expected loss of life, attributable to Ventricular Fibrillation, by a subject who suffers Ventricular Fibrillation is 8.19 years (projected over 10 years)*.

We know from Datum 3 that the expected loss of life of healthy subjects is 1.21 years. Hence we can calculate the loss due to Ventricular Fibrillation as per Table [5](#T5){ref-type="table"}.

###### 

Expected Loss of life due to Ventricular Fibrillation

  Group                  P             loss of life       Expectation
  ---------------------- --------- --- -------------- --- -------------
  Survival               0.16^1^   ×   6.36^2^        =   1.04
  Non-survival           0.84      ×   10             =   8.40
  Healthy subject                                         (1.21)^3^
  Total loss due to VF                                    8.19

^1^Datum 6

^2^Datum 7

^3^Datum 1

**Datum 9***The expected loss of life, attributable to Ventricular Tachycardia by a subject who suffers Ventricular Tachycardia is 2.59 years (projected over 10 years)*.

This datum was obtained from Doval *et al*.\[[@B15]\] by the data extrapolation method described previously. In that study of 516 subjects both with and without non-sustained Ventricular Tachycardia, the projected loss over 10 years for the group with Ventricular Tachycardia was 9.24 years, whereas the projected loss for the group without Ventricular Tachycardia was 6.65 years. The difference is 2.59 years.

**Datum 10***The probability that a subject who experiences one or more episodes of Ventricular Ectopic Beats, will develop Ventricular Fibrillation is 0.34*.

**Datum 11***The probability that a subject who experiences one or more episodes of Ventricular Ectopic Beats, will develop Ventricular Tachycardia is 0.41*.

Datum 10 and Datum 11 are implied from Carrim and Khan \[[@B16]\]. In that study of 44 subjects exhibiting Ventricular Ectopic Beats, *V T*is the set of subjects developing Ventricular Tachycardia and *V F*is the set of subjects developing Ventricular Fibrillation. From their data, we are given:

$$\left| {VT \cap \overline{VF}} \right| = 6,$$

and

$$\left| {VF \cap \overline{VT}} \right| = 3,$$

from which we can deduce that \|*V T*∩ *V F*\| = 12, \|*V T*\| = 18 and \|*V F*\| = 15.

Thus,

$$\left. P(\text{VEB}\rightsquigarrow VT) = \frac{\left| {VT} \right|}{\left| \mathcal{E} \right|} = \frac{18}{44} \right.$$

and

$$\left. P(\text{VEB}\rightsquigarrow VF) = \frac{\left| {VF} \right|}{\left| \mathcal{E} \right|} = \frac{15}{44}. \right.$$

**Datum 12***The expected loss of life, when a Ventricular Ectopic Beat is misdiagnosed as a normal beat is 3.84 years*.

Datum 12 is derived from Data 10, 11, 8 and 9. The derivation is shown in Table [6](#T6){ref-type="table"}.

###### 

Expected loss of life due to Ventricular Ectopic Beats

  Group          P             loss of life       Expectation
  -------------- --------- --- -------------- --- -------------
  Tachycardia    0.41^1^   ×   2.59 ^2^       =   1.06
  Fibrillation   0.34^3^   ×   8.19 ^4^       =   2.78
                                                  
  Total                                           3.84

^1^Datum 11

^2^Datum 9

^3^Datum 10

^4^Datum 8

**Datum 13***The cost of misdiagnosing a Ventricular Ectopic Beat as a normal beat, λ*(*α*~N~\|*ω*~VEB~) *is \$170,189*.

This is the product of Datum 12 and Datum 4. By Proposition 1 we attribute the same cost to *λ*(*α*~N~\|*ω*~F~).

### Costs of misclassification of a normal beat as abnormal

Ventricular Ectopic Beats are of potential concern to a physician. If a system misclassifies a normal beat (*ω*~N~), or a Supra-Ventricular Ectopic Beat (*ω*~SVEB~) as Ventricular Ectopic Beat or a fusion beat (decisions *α*~VEB~and *α*~F~), the likely result is that the patient will be unnecessarily detained in observation, pending further examination. Thus, the cost of misclassification, in this case, is the cost of retaining a patient in intensive care for one day.

**Datum 14***The cost of retaining a patient in intensive care for 1 day is \$2146*

This datum is the mean average of figures obtained from two independent Australian health insurance providers. A case study by Rechner and Lipman\[[@B17]\] for the year 2003 cites the figure \$2670. This study however was conducted within a teaching hospital, and we therefore expect the costs to be higher than average.

We use the cost of retaining a patient in intensive care for one day as the cost of misclassification of a normal beat as the penalty in such instances. Thus *λ*(*α*~VEB~\|*ω*~N~) = *λ*(*α*~F~\|*ω*~N~) = *λ*(*α*~VEB~\|*ω*~SVEB~) = *λ*(*α*~F~\|*ω*~SVEB~) = \$2146.

From Proposition 1 we conclude that *λ*(*λ*(*α*~F~\|*ω*~VEB~) = *λ*(*α*~VEB~\|*ω*~F~) = 0.

### Erroneous classification of beats as Supra-Ventricular Ectopic Beats

Unlike Ventricular Ectopic Beats, Supra-Ventricular Ectopic Beats are typically not life threatening, and are therefore not normally treated unless recurrent \[[@B18]\]. Thus, the cost of misclassification as a normal beat (*λ*(*α*~SVEB~\|*ω*~N~)) is zero. By the same token, misclassification of Ventricular Ectopic Beats or fusion beats as Supra-Ventricular Ectopic Beats bears the same penalty as an erroneous classification as a normal beat. Thus *λ*(*α*~SVEB~\|*ω*~VEB~) = *λ*(*α*~N~\|*ω*~VEB~) and *λ*(*α*~SVEB~\|*ω*~F~) = *λ*(*α*~N~\|*ω*~F~).

Upper bound of risk
-------------------

From Tables [1](#T1){ref-type="table"} and [2](#T2){ref-type="table"}, the value of $\mathcal{R}_{\max}$ was calculated as described in section as \$52,817.

Competing interests
===================

The author(s) declare that they have no competing interests.

Authors\' contributions
=======================

JMD conceived the study, participated in the acquisition of data, and drafted the manuscript. LH assisted with the design of the study, made substantial contribution to the acquisition of secondary data and the interpretation of data. Both authors read and approved the final manuscript.

Pre-publication history
=======================

The pre-publication history for this paper can be accessed here:

<http://www.biomedcentral.com/1472-6947/8/7/prepub>

Acknowledgements
================

The authors would like to express their thanks to Amitava Datta for his advice during this study.
