Abstract: Post-marketing detection and surveillance of potential safety hazards are crucial tasks in pharmacovigilance. To uncover such safety risks, a wide set of techniques has been developed for spontaneous reporting data and, more recently, for longitudinal data. This paper gives a broad overview of the signal detection process and introduces some types of data sources typically used. The most commonly applied signal detection algorithms are presented, covering simple frequentistic methods like the proportional reporting rate or the reporting odds ratio, more advanced Bayesian techniques for spontaneous and longitudinal data, e.g., the Bayesian Confidence Propagation Neural Network or the Multi-item Gamma-Poisson Shrinker and methods developed for longitudinal data only, like the IC temporal pattern detection. Additionally, the problem of adjustment for underlying confounding is discussed and the most common strategies to automatically identify false-positive signals are addressed. A drug monitoring technique based on Wald’s sequential probability ratio test is presented. For each method, a real-life application is given, and a wide set of literature for further reading is referenced.
Before releasing a newly developed drug to the market, a thorough assessment of the benefit-risk profile of the drug is made. Despite the often vast efforts to get a clear understanding of a compound’s impact on the body system, not every adverse drug reaction (ADR) may be spotted in pre-market studies. Post-marketing strategies to detect unknown, and to monitor suspected, risks are of high importance and have therefore led to the development of a wide variety of methodological approaches to detect and follow up drug safety signals in recent decades. According to the World Health Organization (WHO), a safety signal is defined as reported information on a possible causal relationship between an adverse event and a drug, the relationship being unknown or incompletely documented previously . A more recent definition was given by the Council for International Organizations of Medical Sciences (CIOMS) : “Information that arises from one or multiple sources (including observations and experiments), which suggests a new potentially causal association, or a new aspect of a known association, between an intervention and an event or set of related events, either adverse or beneficial, that is judged to be of sufficient likelihood to justify verificatory action.” Here, terminology is used that addresses the inclusion of other sources of information besides ADR reporting, reinforces the notion that a possible safety risk is only suggested, and takes into account the needed verification of the potential association between drug exposure and clinical event . Both definitions interpret the term “signal” as a “signal of disproportionate reporting” as proposed by Hauben and Reich , not a signal or “alert” as known from a clinical context, where an underlying causality is strongly suspected and has undergone clinical review. This interpretation will also be used throughout this paper.
To overcome some of these limitations, and triggered by several severe safety issues in the last years [10,11,12,13], programs have been initiated to make beneficial use of large data pools besides spontaneous reports. An early pioneer in using electronic health care data for routine safety surveillance is the Vaccine Safety Datalink (VSD)  that started to study the adverse side effects of vaccines in 1990. In 2007, the U.S. Food and Drug Administration (FDA) has started to establish the Sentinel Initiative [15,16], which aims at the provision of electronic healthcare data consisting of multiple sources like administrative and insurance claims databases. The target of the Sentinel system is to capture data on more than 100 million individuals for active drug safety surveillance. In the wake of this initiative, the Observational Medical Outcomes Partnership (OMOP) [16,17], a public–private partnership with the Pharmaceutical Research and Manufacturers of America (PhRMA) and the FDA, launched by the Foundation for the National Institutes of Health, strives for improvements in monitoring drugs for safety by researching methods that are feasible and useful to analyze existing healthcare databases. As European counterpart the EU-wide “IMI-PROTECT Project”, a large consortium involving the European Medicines Agency (EMA), academic institutions and the pharmaceutical industry, pursues the goal of strengthening the benefit–risk monitoring of medicines in Europe [18,19]. This endeavor includes the improvement of early and proactive signal detection from spontaneous reports, electronic health records and clinical trials. Moreover, the European Union (EU) has funded the EU-ADR project, where techniques were developed that allow mining for adverse drug reactions in electronic health records (EHR) across European countries [20,21].
A variety of data-mining methods to detect unknown safety signals in SR databases has been developed and established over the last decades, and new methodology that takes advantage of the available longitudinal observational data is nascent. Furthermore, a multitude of drug surveillance techniques has been proposed and tested. The goal of this paper is (1) to give a brief overview of the signal detection process; (2) to describe how different sources of data can be used; (3) to present data-mining methods widely used today, to describe sophisticated methods that were recently developed especially for the use in longitudinal data and to highlight some of the surveillance techniques; (4) to give an outlook on further methods and resources and (5) to discuss some of the real-world applications of the presented methods.
Safety signals may also arise from literature reviews or from findings of pharmacoepidemiological studies, e.g., cohort studies. Such studies are beyond the scope of this paper although they may be set up on pharmacoepidemiological databases.
This paper is organized as follows: Section 2 gives an overview of the workflow in the signal detection process. Section 3 does not only present well-established data sources and data sources that recently became of interest for drug safety research, but also some strategies to prepare the different data sources to make them suitable for signal detection. In Section 4, we discuss the most common signal detection algorithms. We start by introducing simple frequentistic methods designed for SR data and highlight their shortcomings, especially when applied to data with low event counts. These shortcomings can be overcome by more advanced Bayesian techniques especially when frequency counts are low. These techniques, however, ignore possible interactions between several drugs and potential safety hazards, but they can be modified to also cope with this situation as will be shown. Next, the Bayesian measures will be extended to longitudinal data by taking advantage of the information on the duration of drug exposure in the data. Finally, some techniques are presented that do not originate from the SR context and have been developed to take full advantage of longitudinal information in the data. For each of the introduced methods, an application is presented and described briefly. After discussing the problem of confounder adjustment in Section 5, we address the “triage” step, where strategies are applied to automatically separate false-positive signals from those signals that may indicate a safety hazard. Section 7 gives an outlook on techniques on how to monitor potential safety signals over time, before a definite decision can be made whether to discard the signal or to pursue it further. We close with a discussion in Section 8.
2. General Workflow
The process of signal detection comprises several phases (cf. Figure 1). The basis and one of the most crucial parts is the collection and preparation of the data to be analyzed. Different types of data with a variety of available information can be used, ranging from spontaneous reports of ADRs to detailed information from Electronic Medical Records (EMRs). Based on these data, signal detection analyses are conducted. We can coarsely distinguish between two different strategies to detect safety hazards:
Data-mining techniques that strive to uncover so far unknown and unsupected associations. These methods are usually applied to a broad range of combinations of drug exposures and subsequent adverse events, often without limiting the search to pre-defined drug classes or specific medical conditions. They can be regarded as a broad search over the whole spectrum of drug-event combinations (DECs) in the underlying dataset.
If the data-mining search has indicated a possible health risk with a certain DEC, it may be advised to closely monitor this DEC over time to decide whether it should be considered further in confirmatory studies. Surveillance techniques have been developed to consolidate knowledge on these already suspected DECs and are often applied after the first data-mining step.
After the detection of potential signals in the data-mining process, they have to be adjudicated thoroughly to identify all DECs that (a) are already known and well documented; (b) occur very seldom or (c) are highly implausible from a medical perspective and thus can be regarded as artificial false-positive signals. This triage process is crucial in the entire signal detection process as, on the one hand, it can drastically reduce the workload in the following steps, but on the other hand, it can also lead to the dismissal of correctly identified safety signals. Subsequently, each remaining potential signal has to be classified regarding its safety risk. This can either lead to immediate action like confirmatory studies that can result in halt of marketing or even withdrawal of the drug, or given a non-negligible but not critical risk—the decision to closely monitor the DEC via surveillance analysis techniques, or to discard the potential signal as non-hazardous.
3. Data Sources
During the last years, the awareness of the need for extensive drug safety surveillance increased. Programs were set up for a multitude of different data sources and studies were conducted to ascertain whether these data could be of use to monitor unknown effects of drug exposure with signal detection methods.
A wide variety of data sources is considered today , such as
spontaneous reporting databases, like the WHO International Database , maintained at the Uppsala Monitoring Centre (UMC) in Uppsala, Sweden; the European Union Drug Regulating Authorities Pharmacovigilance database (EudraVigilance) or a multitude of national databases, e.g., the Adverse Event Reporting System (AERS) as part of MedWatch, the FDA Safety Information and Adverse Event Reporting Program, or the Vaccine Adverse Event Reporting System (VAERS);
captured data of drug dispensing, e.g., by the New Zealand Intensive Medicines Monitoring Programme ;
longitudinal administrative or claims databases from health insurance institutions, like the Medicare database, based on the social insurance program in the U.S., or the German Pharmacoepidemiological Research Database (GePaRD) ;
EMRs databases, like the General Practice Research Database (GPRD) in the UK, or data from EHRs, or the Vaccine Safety Datalink (VSD) project .
Additionally, types of data that were not available for signal detection before can now be exploited, such as laboratory measurements stored in EMRs. In a recent study by Park et al. , the authors presented a novel algorithm for detecting signals of ADRs using EMR data with focus on laboratory abnormalities after drug exposure, and to evaluate the potential use of this method as a signal detection tool. Moreover, information sources like medical internet forums or text mining in biomedical literature  are becoming of interest. In general, it can be stated that nearly every data source that contains information on health status and drug exposure can be of beneficial use for signal detection purposes.
3.2. Spontaneous vs. Longitudinal Data
SR data are a valuable source of information, as the existing data collections cover very long time periods. Furthermore, national spontaneous reporting systems are installed in numerous countries besides the international data collections, such that a wide range of coverage is reached with these data, both geographically and population-wide. Nonetheless, SR data can only shed light on small sections of the subject’s medical history; the information contained in SR is punctual, that is, usually focused on the day of the ADR only. Hence, SR data are merely a glimpse through a keyhole without the possibility of seeing the full picture (cf. Figure 2). Moreover, as already stated, SR data bear a list of well-known limitations. Since the SR system relies on the reporting of adverse drug reactions in the first place, mainly by physicians, but also the consumers, the data from those sources inherently carry an unknown proportion of under-reporting. The reports on ADRs from studies reported through the pharmaceutical industry are considered more complete. SR data, however, also have the potential for over-reporting, i.e., reports on ADRs that are not solely triggered by the expertise and suspicion of a medical expert, but also influenced by other factors, e.g., extensive media coverage of newly suspected adverse reactions after exposure to a certain drug. The fact that multiple actors may report to the system can lead to duplicate reporting. One of the foremost problems from a methodological point of view is the inability to provide incidence rates: SR data only cover subjects with a drug exposure and a subsequent health event; they do not include data on exposed subjects without an event, so the number of subjects actually consuming the drug of interest is not known. This does not allow for risk assessment as in classical pharmacoepidemiological studies [7,8,9]. A further considerable drawback when using SR data lies in the compliance of the reporters with the reporting system regarding the timely feedback on suspected ADRs. A collection of SR data is of dramatically reduced value if ADRs are reported with a significant delay, as pointed out by Ahmad . Especially high-risk DECs need to be detected as soon as possible, which is only possible if data are provided promptly.
Spontaneous reports are usually aggregated in drug-event report tables (cf. Table 1), containing the essential information from the “raw” reports. These tables can be processed to be suited for statistical analyses, but the statistical methodology is technically not limited to this data format, as discussed later in this paper.
|Sex||Age||Drug 1||…||Drug I||Event 1||…||Event J|
The use of longitudinal observational data like EHRs or health insurance claims data might widen this narrow view considerably by providing information that is not directly connected with the actual ADR. Although of different origin and different (primary) purpose, these longitudinal data usually bear some key similarities, distinguishing them from SR data: they do not contain duplicates, are typically derived automatically and are, therefore, largely unaffected by under- or over-reporting and they include coherent information on most of the subject’s drug exposure periods (often via outpatient drug prescriptions), clinically relevant events (independently from the exposure status) and—very important—information on exposed subjects without events (cf. Figure 3). The assessment of drug exposure and comorbidity status is much more complete as compared to SR data, while this does not automatically imply that 100% of the information is collected. For instance, drug exposures usually are only available for the outpatient setting, information on medications administered during hospitalization periods is often not included. It is also well understood that these routinely collected longitudinal data typically cannot provide information that is not needed for reimbursement, e.g., on lifestyle factors like smoking behavior, alcohol consumption or anthropometric factors like body mass index. This lack of information on lifestyle factors becomes crucial when risk estimates have to be adjusted for potential confounders. A major limitation of most administrative data sources is the lack of validation of the diagnostic coding. As the primary purpose of diagnoses in, for example, claims data is reimbursement, a validation of the coded information against the original source (e.g., a medical chart) is crucial. In the Mini-Sentinel project , such techniques for the validation of health outcomes were reviewed on a wide set of outcomes . Additionally, prescription or dispensation data do not contain information of the patients’ compliance to the prescribed drug regimen; the actual drug exposure usually remains unknown.
3.3. Data Preparation
The signal detection process starts with data preparation. Until today, many techniques operate on frequencies of DECs, collated in 2 × 2 contingency tables, as given in Table 2. For each DEC ij (i.e., drug i, ADR j) in question, such a table is constructed, e.g., in a database containing information on 15,000 drugs and 15,000 ADRs, 225 million such contingency tables are produced and evaluated during the data-mining process. All subsequent formulae refer to the nomenclature given in Table 2, indices ij are omitted where convenient.
|Drug i × ADR j||Event||No event||Total|
The definition of the single cells n11 to n00 in the 2 × 2 table highly depends on the level of information contained in the underlying database.
SR data do not contain any information on the unexposed or event-free groups of the general population. Thus, the definitions for “unexposed” and “no event” have to be adapted. “Unexposed” is defined as exposure to any drug besides exposure to drug i, “no event” is similarly defined as occurrence of any event apart from event j. According to these definitions, n11 is the number of reports on DEC ij and n11 the number of reports on ADR j when exposed to other drugs than drug i. Analogously, n10 and n00 are defined as the frequencies of reports on other ADRs when exposed to drug i or not exposed to drug i, respectively (cf. Figure 4(1)). In the recent literature [30,31,32,33,34], approaches for the conversion of longitudinal data to 2 × 2 tables were proposed:
The basic and most obvious approach is to create “pseudo”-SRs, trying to simulate the exact data structure presented in Table 1. Then, n11 is defined as the number of DEC ij, n01 as the number of events j while not under exposure i, n10 denotes all exposure periods to drug i with a different ADR than event j and n00 is the number of non-j events under non-i exposure (cf. Figure 4(2)). This is a coherent and convenient definition, suffering from the major deficit that information on exposures without events and events without exposures is missing. This approach was discussed and implemented by Schuemie  and Zorych et al. .
Curtis et al.  proposed a method of converting longitudinal data to SR with the possibility to additionally include information on non-exposures and non-events by introducing temporal segmentation of the data. They considered each month per subject to be a single report, consisting of all events that the subject experienced during this specific month and all drugs that were consumed that month or the month before. Thus, reports similar to the structure shown in Table 1 can be generated, plus reports that might contain information on exposures without events or events without exposures (cf. Table 3). Here, n11 is defined as before and denotes the number of reports on DEC ij, n10 is the sum of all reports on drug i without ADR j, n01 is—viceversa—the sum of all reports on ADR j without exposure to drug i and n00 the number of reports containing neither drug i nor ADR j (cf. Figure 4(3)).
A closely related approach to take advantage of the longitudinal information, but without imitating a “reporting structure” was described by Schuemie  and Zorych et al. . Here, n11 is defined as the number of distinct DECs ij, n10 and n01 stand for the number of all exposure eras to drug i without the occurrence of ADR j or the number of ADRs j not experienced during exposures to drug i, respectively. Finally, n00 is defined as the number of all non-j events that occur during non-i exposure periods, event-free non-i exposure periods and non-j ADRs when not exposed to any drug (also cf. Figure 4(3)).
The final approach uses even more information than the one presented in c). n11 is defined as number of individuals experiencing event j while exposed to drug i, n10 and n01 are defined as number of persons with exposure to drug i and no occurrence of event j, or an experienced event j and no exposure to drug i, respectively. Finally, n00 includes all individuals that were neither exposed to any drug, nor experienced any ADR (cf. Figure 4(4)). Thus, n11 + n10 + n01 + n00 equals the number n of all subjects contained in the database.
These might not be all approaches currently discussed in the literature, but this overview gives an adequate insight in the basic ideas.
|Sex||Age||Drug 1||…||Drug I||Event 1||…||Event J|
3.4. Definition of Exposure and Event in Longitudinal Data
A crucial step in converting longitudinal data is the definition of what needs to be considered as “event” and “exposure.”
The estimation of drug exposures depends on the type of available information; often drug prescriptions are considered as a suitable surrogate if exposure information is not directly available. The estimation of the exposure duration is usually constrained by the lack of information on the actual adherence to the prescribed drug regimen. If information on dosing regimen is available, this information can be used to estimate the exposure duration, whereas the number of dispensed units and the defined daily dose (DDD) may be used as substitute ; actual drug use can also be estimated from repeated prescriptions when focusing on chronic diseases. A thorough discussion of the definition and calculation of exposed and unexposed person-time is given by Brown et al. .
An event may be defined as hospitalization due to an emerging illness, an event under drug exposure could then be defined as hospitalization with a preceding drug prescription within a fixed timeframe before the onset of the event. Although this timeframe highly depends on the drug considered, a fixed timeframe usually is chosen for all drugs and events for practical reasons. For instance, Choi et al. [31,32] and Kim et al.  consider 12 weeks before the event to be a suitable timeframe to look for the target drug exposure.
4. Analysis Techniques
Basically, two methodological approaches to detect safety signals can be identified in the recent literature, First, there are the data-mining techniques that operate cross-sectionally on a single data snapshot (non-sequential methods), where the disproportionality analysis methods comprise the most widely applied class of analytical techniques . These methods will be discussed in this section. Second, there are the surveillance methods that sequentially calculate a cumulated risk estimate over a sequence of data snapshots, taken at different points in time (sequential methods), where approaches based on Wald’s Sequential Probability Ratio Test (SPRT) [39,40] are among the applied methods. These methods are typically not applied to detect yet-unknown safety hazards, but to monitor DECs where a health risk is of concern. These surveillance techniques will be reviewed briefly in Section 7 of this paper.
Besides these highly automated methods, traditional approaches like reviews of single cases or case series, literature reviews or results from pharmacoepidemiological studies (e.g., cohort designs or case-control designs) can yield safety signals, but these methods usually need manual input, such as appropriate confounder selection, and are not suited to automatically process large quantities of different drug exposures and ADRs in a reasonable timeframe and are thus not considered here.
As part of the large governmentally funded initiatives mentioned before, development of new methodology in drug safety has taken a big leap forward after the start of the Sentinel Initiative and the OMOP collaborative. Extensive methods libraries with accompanying documentation material were set up, most prominently the documentation regarding the single working areas in the Mini-Sentinel [16,41] pilot program or the OMOP methods library . The latter profited largely from the so called “OMOP cup,” a competition to promote methods development and evaluation of novel algorithms for identifying drug safety issues in observational healthcare data, proclaimed in 2009.
4.2. Disproportionality Analysis Measures for Spontaneous Reporting Data
Disproportionality analysis measures are constructed to identify combinations of drug exposures and ADRs that occur disproportionately often, compared to other drug-event combinations. Originally developed for SR data, disproportionality measures can also be directly applied to longitudinal data. Several different disproportionality measures have been proposed in the literature [35,43,44,45,46,47,48], which can generally be divided into two categories: frequentistic and Bayesian, both relying on the aforementioned 2 × 2 contingency tables. The most popular frequentistic methods are the proportional reporting rate (PRR)  and the reporting odds ratio (ROR) ; among the Bayesian approaches the Bayesian Confidence Propagation Neural Network (BCPNN)  and the Gamma-Poisson Shrinker (GPS)  (respectively, its extension, the Multi-item Gamma-Poisson Shrinker, MGPS ) are the most prominent and widely used techniques.
As the results of signal detection analyses are usually referred to as “risk” estimates, one might assume that they can be directly compared to risk estimates obtained from case-control or cohort studies in pharmacoepidemiology. This is, however, not the case, as SR data do not contain information on subjects under drug exposure who did not experience a health event, so that “risks” cannot be appropriately estimated. The “risk” estimates obtained from SR data merely serve as a tool to identify the most suspicious signals. These signals then need to be examined further in the signal adjudication step.
4.2.1. PRR and ROR—Simple Measures
The most basic frequentist disproportionality analysis measures that are widely used are the PRR and the ROR. Their estimators are defined similarly to the estimators of the relative risk (RR) and the odds ratio (OR) with:
The limits of the 95% confidence intervals (CI) of PRR and the ROR are usually obtained via an approximation of the normal distribution as
The estimates for PRR and ROR are easy to calculate, but the results tend to become unstable when the number of events is small, resulting in large estimates with wide confidence intervals [3,45,50], thus leading to many false-positive signals for very rare events. To uncover these false-positive signals, for instance, the biological plausibility has to be examined and/or confirmatory studies to re-assess the found signals using additional data sources have to be conducted (see also Section 6 of this paper).
As pointed out by van Puijenbroeck et al.  and Bousquet et al. , other statistical methods usually applied in cross-classification tables can also be exploited, such as the X2-test with one degree of freedom (with or without Yates’s correction ), where a relevant signal would then require a X2-value greater than the corresponding 95% quantile 3.85 . Further, rarely applied methods include the crude relative risk (cRR) or Yule’s Q-test . The problems that may arise when using such fixed thresholds are discussed in Section 6 below.
4.2.2. BCPNN and GPS—Bayesian Shrinkage
The instability of the above estimators when applied to low drug-event counts led to the development of more advanced Bayesian shrinkage techniques. The two methods mainly used today are the BCPNN, applied at the UMC to analyze the WHO database, and the MGPS, based on the GPS and deployed on the SR data of the FDA.
Both methods are based on the relative reporting ratio (RRR), defined as
The can easily be interpreted: it simply is the ratio of how many ADRs under exposure were actually observed over the number of expected events under the assumption that ADR and drug exposure were independent.
Based on this simple definition of the RRR, the BCPNN estimates the information component (IC), a measure of mutual information between two variables, originating from information theory . It is defined as
The Bayesian approach underlying the theoretical concept of the BCPNN assumes binomial distributions for both variables and a multinomial distribution for the joint probability. Due to the convenient analytical characteristics of conjugate distributions in Bayesian theory , the prior distributions are assumed to be the conjugate priors of binomial and multinomial variables, namely beta distributions. Let and be random variables to describe the occurrence of drug exposure i and ADR j, respectively. The a priori expectations of X and Y are obtained as:
The distribution of the joint occurrence X and Y is also given as a Beta distribution with
After inclusion of the information given in the SR data, a closed form of the a posteriori distribution of the IC cannot be given, but the a posteriori expectation and the variance of the IC can be derived as
An estimate for the credibility interval (CredI) of the IC using Monte-Carlo simulations is given by Norén et al. .
As an alternative, DuMouchel proposed the so-called Gamma-Poisson Shrinker (GPS) algorithm . Here, the occurrence of the target DEC is considered as rare event such that the observed DEC count n11 may be assumed as realization of a Poisson-distributed random variable. According to the GPS, the relative reporting rate is defined as
Following a Bayesian approach, μ is not considered as a fixed parameter, but as random, hence λ is also considered to be a random variable. The assumption of an underlying Poisson distribution for n11 leads to a gamma distribution as conjugate prior for λ. To add flexibility to the model, a mixture of two gamma distributions with initially unknown mixture parameter p is assumed as distribution of λ:
The EBGM is then estimated by a plug-in approach, where E is estimated as shown in Equation 11, and the parameters α1,α2,β1,β2 and p are obtained from the data using an empirical Bayes approach. The fifth percentile of the posterior distribution of λ is denoted as “EB05” and interpreted as the lower one-sided 95% confidence limit for the EBGM, the upper one-sided 95% CI is defined analogously as 95th percentile. This Bayesian estimator gives more conservative risk estimates when event counts are small, i.e., the risk estimates are considerably smaller and the CIs less wide, hence the denomination “shrinkage estimate.” While this shrinkage might obfuscate a real signal by reducing it to a non-conspicuous level, it helps to eliminate false-positive signals, which otherwise would have to be adjudicated subsequently.
4.2.3. Extension of the GPS: the MGPS
The techniques discussed so far assess the risk of 2-way DECs, i.e., one drug and one ADR. Another serious concern is due to potential interactions between several drugs taken simultaneously in relation to the occurrence of an ADR. For the sake of simplicity, let us assume that we are interested in a specific ADR, denoted as A, and in two drugs D1, D2, where neither exposure to D1 nor D2 alone results in an elevated risk for A. If the joint exposure to both drugs poses a safety risk, this risk would not be detected in two-way analyses. A famous example is the interaction of cerivastatin and gemfibrozil, leading to an elevated risk of rhabdomyolysis and resulting in the withdrawal of cerivastatin [11,60] from the worldwide market in 2001.
DuMouchel and Pregibon  introduced the Multi-item Gamma-Poisson Shrinker (MGPS) as an extension of the GPS algorithm in 2001 to deal with multi-item sets of a size n > 2 (e.g., n = 3; drug-drug-event interactions). The basic idea is to assess how much of the observed frequency of the joint occurrence of the multi-item-set can be explained by the occurrence of all possible two-way interactions in the set of (n − 1) drugs under inspection and the event A of interest. Given the above set of two drugs (D1, D2) and one ADR (A), the number of reports on A after simultaneous exposure to D1 and D2 is considered to be “interesting” if the number of reports involving the two-way interactions (i.e., D1 × D2, D1 × A and D2 × A) does not explain the observed count of the triplet. A log-linear analysis can be conducted to determine if any of the observed frequencies of the two-way combinations depends on the third item. From this analysis one obtains an estimate eAII2F of the frequency of reports on the joint occurrence of D1, D2 and A if all associations were strictly pairwise and independent from the third item. DuMouchel and Pregibon define the EXCESS2 value as number of excess reports on D1, D2 and A over what might be expected if all associations were only pairwise:
Thus, high EXCESS2 values of an examined triplet might indicate that a safety risk is given under combined exposure to D1 and D2. This approach may also be extended to higher interaction levels . A variety of approaches to properly handle multi-item associations can be found for example in von Puijenbroek et al. [61,62], Norén et al. [57,63], Almenoff et al.  or Madigan et al. .
4.3. Analysis Techniques for Longitudinal Data
4.3.1. Adaptation of the MGPS for Longitudinal Data: the LGPS
For the analysis of longitudinal observational data (e.g., claims data), one option is to convert the data structure to match the structure of SR data, so that the aforementioned techniques can be applied directly. An additional option is to modify the algorithms to better fit the structure of longitudinal data and take full advantage of the available information.
Key information in longitudinal studies includes the number of days a patient was under risk, i.e., the number of days the patient was exposed to the target drug. Let t1 denote the number of days the patient was under risk, t0 the number of days the patient was observed without being under risk, and n01 the number of ADRs j when not exposed to drug i. Then, according to Schuemie  the expected number (E) of DECs ij can be estimated as:
4.3.2. Self-Controlled Case Series
The “self-controlled case series” (SCCS), introduced by Farrington  in 1995, can be used to study the temporal association between a time-varying drug exposure and an adverse event using data from cases only. The basic idea of the SCCS is to compare the incidence rate of the event of interest in periods when the individual was exposed to the drug of interest to periods without such exposure. Each case acts as its own control, thereby controlling for both measured and unmeasured confounding variables that do not vary over time, which is the key advantage of the method. Initially developed to study the risk profile of vaccines, it gained wide recognition when used to examine the effect of vaccination for mumps, measles and rubella (MMR) on autism .
For each individual k, risk periods m,m = 1,2,..., are identified from the data, i.e., windows of time either during or directly after drug exposure. Any other time periods in the observation period for subject k are considered to constitute the control periods, indexed by m = 0. Moreover, the observation period for individual k is split into age groups l to incorporate the effect of aging. The occurrence of events is assumed to follow a non-homogeneous, age-dependent Poisson process.
Let nklm be the realization of a Poisson-distributed random variable Nklm and denote the number of target events experienced by individual k while having spent the time eklm in age group l and risk period m. The incidence rate in each such interval, denoted by λklm, is assumed to be constant and is given by:
The number of events Nklm occurring within an interval of length eklm is assumed to be Poisson-distributed with rate eklmλklm. The exponentiated quantities are referred to as relative incidences and are a measure of incidence in risk period m relative to the control period m = 0.
Conditioning on the total number of events observed for the individual k during the observation period, the log likelihood is multinomial:
Hocine et al.  based a technique for near real-time drug surveillance on the SCCS (see Section 7.3). Besides the SCCS, other case-only methods are also used in pharmacovigilance, but not as prominently as the SCCS. A variety of case-only approaches was discussed by Maclure et al.  with respect to their usefulness for safety monitoring.
4.3.3. IC Temporal Pattern Discovery
Norén et al.  introduced a technique to identify patterns in the temporal association between the prescription of a drug and the occurrence of a medical event, the “IC temporal pattern discovery” (ICTPD). The method is, very similar to the SCCS, based on the intra-personal comparison of a risk period and a preceding control period. A main difference to the SCCS technique, though, lies in the additional use of information from non-cases, as the ICTPD is mainly focused on the exposure to a certain drug. The methodology is based on a disproportionality approach such as presented in the previous section, where the observed number of ADRs j in a certain time period t is contrasted to an expected number based on the overall frequency of the ADR relative to other drugs, which is analogous to the approach in Equation 5, but dependent on the time-window t. Thus, let further denote the number of prescriptions of drug i with a subsequent event j in the time-window t, the number of prescriptions to any drug with a subsequent event j in time-window t, the number of prescription of drug i and any subsequent event in time period t, and the number of any prescription, followed by any event in time period t. Then the IC, defined in Equation 7, for the time window t can be estimated as
Based on this definition of the IC, the authors constructed a measure of temporal association. Let u be the time period of primary interest, i.e., the time period after a current drug prescription of drug i, and v a control time period, against which u is to be contrasted. Considering the difference
For further information, e.g., on choices of u and v, the possibility of stratification to adjust for confounding and a sophisticated graphical representation of the ICΔ estimates (the so-called “chronographs”) and their interpretation, we refer to Norén et al. .
In the following, sample applications for each analysis technique are presented.
4.4.1. Application of the Disproportionality Measures
Several recent studies used data from the Korean Health Insurance Review and Assessment Service (HIRA), a database including all claims data and prescriptions from approximately 50 million Koreans. In 2010, Choi et al.  examined the risk profile of rosuvastatin compared to other statins in a population of approximately 1.1 million elderly patients (≥65 years), based on nearly 12 million drug prescriptions by calculating the crude relative risk (cRR) as signal detection method. Target outcomes were several clinical conditions associated with statin use, divided in ADRs for rosuvastain and all other statins. They found 25 signals for rosuvastatin, including 8 that were expected only for this drug, 11 that were expected only for other statins and 6 that were not expected for any of the statins investigated. They considered the cRR measure to be an efficient data-mining tool.
In 2011, Choi et al.  used the same data to assess the risk of analgesics and anti-inflammatory drugs (AAIDs) in the same cohort of elderly patients. A wider set of signal detection methods was applied: PRR, ROR, BCPNN and the cRR. According to their results, the cRR seems to be the measure with the highest sensitivity and specificity.
Zorych et al.  used simulated data generated with the Observational Medical Dataset Simulator (OSIM) [71,72] plus anonymized Thomson Reuters MarketScan databases : the lab data (MSLR), the Medicaid Multi-State database (MDCD), the Medicare Supplemental and Coordination of Benefits database (MDCR), and the Commercial Claims and Encounters database (CCAE). They did not focus on specific events or drugs, but ran database-wide analyses across all events and drugs present in the data using the four most common data-mining techniques: PRR, ROR, BCPNN and MGPS. According to the results from the simulated as well as from the real data Bayesian techniques showed better performances than PRR and ROR.
4.4.2. Application of the LGPS
Schuemie  also used data generated from the OSIM and real-life data from the Integrated Primary Care Information (IPCI)  database from the Netherlands. He focused on detecting the signals that were artificially defined during the simulation process. He applied the longitudinal LGPS algorithm in comparison to other variants of the GPS, and LGPS showed the best performance. Schuemie considered his approach to be highly suitable for signal detection, which is underpinned by the fact that he won the aforementioned OMOP cup in 2010 with the LGPS approach. The LGPS is also successfully applied by the EU-ADR consortium, that examines the combined healthcare data of eight European databases [21,75] for safety signals.
4.4.3. Comparative Studies
Two recent studies by Schuemie et al.  and Ryan et al.  compared a wide set of methodological approaches for risk identification in longitudinal data. Schuemie et al. assessed the performance of ten different methods, ranging from the simple PRR and ROR to elaborated techniques like the LGPS or SCCS on a dataset comprising information from seven European health care databases, including over 20 million individuals. They examined for each method its ability to detect pre-defined known signals and associations unlikely to represent an ADR (negative controls). Additionally they applied a technique to assess protopathic bias (“LEOPARD”, presented by Schuemie , which is briefly discussed in section 6 of this paper) in combination with each of the ten signal detection techniques. They found that the performance of the methods did not differ too much, but that the LGPS and a matched case-control design outperformed the other approaches, whereas the techniques originally designed for SR data fell behind slightly. The filtering for protopathic bias increased the performance for each of the ten methods under inspection.
An analogous approach was made by Ryan et al. , who compared eight out of 13 analytical methods that were implemented during the OMOP, e.g., disproportionality measures adapted from the SR data analysis, ICTPD, univariate SCCS or a cohort design based on a high-dimensional propensity score (HDPS) stratification [77,78] (cf. Section 5 of this paper for details on the HDPS technique). The analyses were based on a combination of various health care databases, covering about 130 million patients. The authors assessed a total of 53 DECs, where nine were true signals and 44 negative controls. For each analysis technique, a variation of method-dependent parameter settings was studied. A set of performance measures was included to rate the performance of the methods, e.g., the area under the receiver operating characteristic (ROC) curve (AUC)  or the average precision (AP) . The authors found the cohort design based on HDPS adjustments to be the most predictive, followed by the ICTPD and the univariate SCCS. Similar to the results of Schuemie et al., the disproportionality measures originating from the SR data analysis were among the least predictive models. All assessed methods that were developed in the OMOP are freely accessible in the OMOP methods library .
5. Confounder Adjustment
A largely unsolved problem is the adjustment for confounding, as pointed out by Lu  or Hauben and Bate [9,50]. It is important to ensure that confounding does not produce spurious safety signal warnings of non-existing hazards. With longitudinal data, information on some relevant confounding factors is at hand (e.g., concomitant medication or comorbidities), although other potential confounders are usually still lacking, such as smoking behavior or alcohol consumption. Adjustment for underlying confounding is not per se part of signal detection techniques, especially not of the simpler ones (PRR, ROR), but can partially be achieved by stratification for certain variables, like sex and/or age (groups). The disproportionality measures can then be calculated per stratum, although stratification usually comes at the cost of more sparse data tables, thus jeopardizing the stability of the statistical analyses if event counts are low.
To adjust study results for confounders, a lot of sophisticated approaches exist for pharmacoepidemiological study designs, which can, however, not easily be exploited for signal detection methods in the pharmacovigilance context. As a major problem, the selection of the appropriate confounders that have an impact on the outcome is usually a manual process, involving expertise from various scientific disciplines. For high-throughput signal detection methods this is not feasible, and automated solutions would be needed. This problem was addressed by Schneeweiss et al. with the introduction and application of high-dimensional propensity scores (HDPS) [77,78], a technique based on the propensity score (PS) introduced by Rosenbaum and Rubin . The HDPS algorithm automates the selection of potential confounders to be considered in a multi-step approach. Briefly, the HDPS algorithm
requires the identification of the different data dimensions (e.g., hospitalization data, outpatient care data, outpatient drug dispensation data) in the database;
identifies a pre-specified number of the top most prevalent codes, e.g., ICD or ATC codes (ICD = international statistical classification of diseases and related health problems, ATC = anatomical therapeutic chemical classification system) in each data dimension as candidate covariates;
ranks candidate covariates based on their recurrence (the frequency that the codes are recorded for each individual during the baseline period);
ranks covariates across all data dimensions by their potential for control of confounding based on the bivariate associations of each covariate with the treatment and with the outcome;
selects a pre-specified number of covariates from Step 4 (e.g., 500) for PS modeling, and
estimates the PS based on multivariable logistic regression using the selected covariates plus any pre-specified covariates.
This technique theoretically allows for a fully automated selection of confounders for each signal detection analysis. Rassen and Schneeweiss  applied the HDPS to control for confounding in sequential database cohort studies. They concluded that HDPS offers substantial advantages over non-automated alternatives in active product safety monitoring systems. In the comparative study by Ryan et al. , the HDPS stratified cohort design turned out to lead to the best predictive model. Garbe et al.  compared the performance of the HDPS vs. conventional PS with manual confounder selection, using data from the German Pharmacoepidemiological Research Database (GePaRD). Here, the comparison of HDPS and conventional PS matching resulted in improved point estimates for the HDPS when studying an intended treatment effect of coxibs versus traditional non-steroidal anti-inflammatory drugs, benchmarked against results from randomized controlled trials.
A common approach to adjust for confounders in pharmacoepidemiological studies is the use of a multiple logistic regression model . In the pharmacovigilance setting with often >10,000 covariates present, confounding theoretically can be introduced by any of these covariates. Regression against all of these possible confounders bears (a) theoretical problems and (b) until recently also technical problems in terms of necessary computing time. Although methodology for automated selection of confounders is developing (e.g., HDPS), advanced logistic regression models are also considered. Genkin et al.  proposed a Bayesian logistic regression (BLR), where millions of confounders can be included in the analysis. The BLR was applied to the WHO SR database at the UMC by Caster et al. , and they concluded that the BLR does offer practical advantages, as it can eliminate false-positives and false-negatives due to other covariates, and it identifies some established drug safety issues earlier than a measure based on contingency tables.
A different approach to adjust for unmeasured and measured time-invariant confounding is taken in the case-only methods, such as SCCS (cf. Section 4 of this paper, see also especially Hocine et al.  and Maclure et al. ). Here, as already mentioned, each case is its own control, thereby controlling for both measured and unmeasured confounding variables that do not vary over time, such as sex, genetic predispositions or general state of health.
6. Triage—Adjudication of Potential Signals
When the potential signals have been identified by one of the described analytical methods, the impact and the importance of each identified potential signal have to be evaluated. The aim of this adjudication process is the identification of those signals that are likely to indicate a yet-unidentified safety hazard, and the elimination of false-positives from the results. The sheer number of DECs usually under examination, especially when data-mining methods are applied, makes a structured adjudication of the results a necessity.
First, the plethora of potential signals needs to be reduced by application of a set of basic rules. In EudraVigilance, only signals that are based on three cases or more, show a risk estimate of 2 or higher and have a lower confidence limit greater than 1 are considered relevant. A different approach may be based on a X2-test with one degree of freedom, where a relevant signal would then require a X2-value greater than 3.85 . Such simple rules can drastically reduce the amount of potential signals to be processed. A different approach to reduce the number of potential signals is not to focus on the risk estimates from the signal detection, but on a fixed number R of potential signals that will remain. The list of analyzed DECs is ordered by the size of the respective risk estimates and cut after the top R entries. More sophisticated approaches were suggested by Ahmed et al. [87,88] where “rules of thumbs,” e.g., any reliance on fixed thresholds and the estimated 95% confidence interval were avoided. Here, the decision criterion relies on the estimation of the false discovery rate (FDR) . Bayesian and non-Bayesian FDR-based methods were proposed that address the arbitrariness of thresholds and allow for a built-in estimate of the FDR. Simulation studies indicated that these methods can be suitable alternatives to the currently used methods; see  for more details.
After having reduced the number of potential signals by mere technical restrictions, the remainders need to be assessed on a qualitative level. A common step is to exclude—automatically if possible—all known and well-documented risks and to focus on the unknown or unexpected identified signals. The exact layout of this part of the triage highly depends on a number of factors, including the underlying data structure, the signal detection method used and personnel resources, as in-depth medical and pharmacological knowledge is necessary. No globally standardized modus operandi can be defined, but many institutions assessing drug safety data utilize a defined set of triage criteria. For instance, the WHO criteria applied at UMC have been well documented by Ståhl et al. , Lindquist  and Hauben and Bate  and include several different aspects like the public health impact of the outcome, novelty of the drug in the market or comparison of the results with prior analyses.
Besides these qualitative assessments, other sources of false-positive signals can be identified, for instance cases of protopathic bias , when a drug is administered to treat the condition in question, thus leading to reverse causality in the signal detection process. Schuemie  proposed a framework to identify signals caused by such a bias, the Longitudinal Evaluation of Observational Profiles of Adverse Events Related to Drugs (LEOPARD). It is based on the comparison of prescription rates of the drug under consideration in time windows before and after the event, and a potential signal is considered to be protopathic if the number of prescriptions after the event is higher relative to the prescriptions before the event.
Once the triage is completed, the safety risk of every remaining signal needs to be rated to decide whether (a) impact analyses and subsequent confirmatory analyses need to be induced; (b) the signal should be monitored to sharpen the risk profile or (c) the signal can be discarded because of low potential risk. Schneeweiss  proposed a systematic approach for these confirmatory steps.
7. Near Real-Time Surveillance Techniques
If a certain DEC is suspected to bear a safety hazard after the data-mining process, but no certain assertion for an elevated risk can be made yet, it might be advisable to monitor the respective DEC closely. It would be most preferable to assess the risk profile of the active agent of concern in real-time, i.e., whenever new information is available, or in near real-time, i.e., at regular short intervals. For this purpose, classical sequential tests might be applied as e.g., Wald’s Sequential Probability Ratio Test (SPRT). Wald’s approach is based on the simple null hypothesis H0:RR = 1 vs. the simple alternative H1:RR = r with a known fixed r > 1. Let Xt be a Poisson-distributed random variable representing the number of observed adverse events at time t with means λt and rλt under the null and the alternative hypothesis, respectively, xt the realization of Xt. Wald’s approach is based on the repeated assessment of the likelihood ratio test statistic
This approach is of limited practical value for the assessment of an unknown drug risk for two main reasons. First, the simple alternative H1:RR = r requires the magnitude r of the elevated risk to be known, which is usually not the case as one is merely interested in an unknown elevated risk RR > 1. Second, the baseline frequency λt also needs to be known as it is part of the test statistic. This is not always the case.
To overcome the first of these two restrictions, Kulldorff et al.  introduced the MaxSPRT, extending Wald’s original approach by allowing for a composite alternative hypothesis . This leads to a modified test statistic, where the denominator is still given by the simple likelihood of the null hypothesis, but the numerator now is given as the maximum likelihood under the composite alternative hypothesis. The likelihood ratio based test statistic is then
To handle the second of the above restrictions, i.e., the often unknown baseline frequency λt, one needs access to historical data that were collected before the start of the current surveillance in a cohort not exposed to the drug of interest. From these data, λt can be estimated and this estimate can be used in the test statistic via a plug-in approach. While this seems to be reasonable, it inherently creates a new problem: now used in the test statistic is just an estimate, not the real baseline frequency, and as such it has a variability of its own. If we neglect this variability by simply treating as if it was the real value, we could spoil the results by assigning a level of certainty it does not possess.
Li et al.  thus introduced the so-called Conditional MaxSPRT (CMaxSPRT). λh and λs denote the event frequencies in the historical data and the surveillance data, respectively. The question that needs to be answered is whether λh and λh are equal or not, leading to the hypotheses H0:λs = λh = λ0 vs. H1:λs > λh. The joint likelihood of the historical data and the surveillance data after the kth event is given as:
7.2. Application of the SPRT and MaxSPRT
An application of MaxSPRT was presented by Brown et al.  who conducted a proof of principle evaluation. They assessed well-documented risks of several drugs (the NSAIDs celecoxib, rofecoxib, valdecoxib, the ACE-inhibitor lisinopril and the statin cerivastatin) plus comparator drugs and two negative controls (cetirizine, clemastine) using data from multiple health plans involved in the Research Network’s Center for Education and Research on Therapeutics (CERT)  of the US Health Maintenance Organization (HMO). Analyzing data from approximately eight million insurance members covering a six-year study period, the drug-event associations known from literature were found for four of five drugs, whereas the negative controls triggered no signal. Brown et al. discussed several methodological problems using healthcare data for signal detection, they endorsed the use of the MaxSPRT technique in the monitoring of drug safety. Li , who also used the CERT data, assessed the risk of acute myocardial infarction (AMI) after intake of rofecoxib compared to diclofenac and naproxen using the CSSP method. The obtained results were consistent with those reported by Brown et al. .
Davis et al.  applied a SPRT analysis to the Vaccine Safety Datalink (VSD) data to assess the risk of intussusception after rotavirus vaccination and risks for fever, seizures, and other neurologic adverse events after the change from whole cell diphtheria-tetanus pertussis (DTPw) to a cellular DTP vaccine (DTPa). They detected an increase in intussusception after 2589 vaccine doses of rotavirus vaccine, which was an expected signal, and decreases in fever rates, febrile seizures and other neurologic events with DTPa, where such an association has been suggested by prelicensure trials. The authors concluded that active and prospective surveillance analysis of VSD data provides a valuable, population-based early warning system.
Near real-time safety surveillance and sequential analysis were pioneered for and have been mostly used for vaccine safety surveillance. Yih et al.  reviewed a large number of post-market vaccine safety surveillance studies using the Poisson-based maxSPRT described in Section 7.1 and a binomial-based maxSPRT (not described in this paper, see  for details).
7.3. Further Approaches
Further approaches can be found in the literature as for instance if sequential tests should be applied on a regular basis, say weekly or monthly, where group sequential testing methods might be more appropriate as proposed by Li  and Li et al. .
Hocine et al.  presented a sequential case-series analysis, combining the idea of the SPRT with the SCCS design. They applied the SPRT, implemented as a group sequential test, to the log-likelihood of the self-controlled case series model.
Furthermore, Jin et al. introduced algorithms that basically strive to rate the “unexpectedness” of an event [102,103]. Alternative techniques for visual inspection have been suggested in [104,105,106,107].
In this paper, we gave an overview of the process of signal detection, including available data sources and the preparation of the data, various analysis techniques for SR data, how some of them can be extended to make beneficial use of additional information available in the data and techniques specifically developed for longitudinal data. We briefly addressed the adjustment for underlying confounding as well as the decision making during the triage before presenting methods for drug surveillance.
Looking back on the past decade, Hauben and Bate , Hauben and Norén  and Lu  stated unanimously that disproportionality analyses for signal detection purposes are a worthwhile addition to pharmacovigilance and may be of great value when applied with care, i.e., knowing that the techniques and the underlying data have their limitations. While the usage of longitudinal data for “classical” pharmacoepidemiological studies is routine today , the usage of such data for pharmacovigilance is still an emerging field. Impressive amounts of data are potentially available for analysis [21,35,76], and the secondary use of healthcare data for signal detection research is endorsed by many [3,35,109]. It seems plausible to extend the utilization of disproportionality measures from SR to longitudinal data as a first step. Large-scale databases offer vast amounts of information, which can allow for more refined analyses, addressing some of the shortcomings of SR data . To apply methods known from SR data, the longitudinal data need to be transformed to be digestible for the algorithms. As a second step, methods known from SR data can be modified to take advantage of the longitudinal data structure directly. Enhancements especially of the Bayesian techniques have been shown to yield results of higher relevance than the methods for SR data, as indicated by the findings of Schuemie  and the combination of the LGPS and LEOPARD algorithms. However, following the results of Schuemie et al.  and Ryan et al. , the techniques that originate from the SR data realm often fall short compared to the newly developed techniques for longitudinal data, e.g., those developed or refined in the wake of the OMOP, like ICTPD by Norèn et al.  or the application of SCCS or HDPS in pharmacovigilance [68,77,78].
Madigan and Ryan  proposed a set of simple questions, comprising central problems: what methods should be used to yield the best results, how can disparate data collections be combined beneficially, when does a signal need to be considered as a risk, what to do with newly identified risks and how often do the detection methods fail to deliver the correct result? For SR data, some questions seem to be answered: regarding the choice of methods, the “big four” (namely PRR, ROR, BCPNN, MGPS) seem to be the dominant data-mining methods, and according to Madigan et al. , Bayesian methods have shown clear advantages when applied to SR data and are the de facto standard for SR data analysis worldwide today. For drug surveillance, MaxSPRT has gained a lot of attention, as well as case-only methodology . For signal detection on longitudinal data, it is yet unclear which of the existing techniques yields the best result, but the findings of Schuemie et al.  and Ryan et al.  suggest that the techniques used in the OMOP, like the combination of LGPS and LEOPARD or the HDPS methodology are likely to provide results of high relevance. However, the methodology for longitudinal data is still nascent, and according to Madigan and Ryan  only extensive empirical experimentation can bring answers to the question of the optimal analysis strategy.
Although one might state that the applications presented above showed reasonable results, a few limitations regarding automated analyses should be kept in mind. Generally, Ray  claimed that a high level of awareness for the inherent complexity of the studies and the data is needed. A major issue arises from the fact that no true gold standard exists to assess the quality of the results obtained from data-mining analyses [36,80]. One possible way out would be to compare results from a data-mining-study with those from a pharmacoepidemiological study regarding some well-known safety hazards to assess the performance of the applied data-mining technique, although this is, of course, no proper validation. To validate a specific result, the same study could be run on a second database to confirm the first finding. If, however, several large databases are pooled to detect even rare signals, such a finding could possibly not be replicated since equally large databases allowing such a study might not be available .
The challenging research field of signal detection comprises many more facets and not all could be discussed in this paper such as the merging of several databases which requires the development of a common data model , the mapping of terminology from different coding schemes (diagnostic codes, drug coding, etc.) which needs to be adhered to [21,113], the “innocent bystander” effect [9,45], and last, but not least, data protection issues or other legal questions that may arise when different databases containing sensitive patient information are combined .
- Bresalier, R.S.; Sandler, R.S.; Quan, H.; Bolognese, J.A.; Oxenius, B.; Horgan, K.; Lines, C.; Riddell, R.; Morton, D.; Lanas, A.; et al. Cardiovascular events associated with rofecoxib in a colorectal adenoma chemoprevention trial. N. Engl. J. Med. 2005, 352, 1092–1102. [Google Scholar] [CrossRef]
- Furberg, C.D.; Pitt, B. Withdrawal of cerivastatin from the world market. Curr. Control. Trials Cardiovasc. Med. 2001, 2, 205–207. [Google Scholar] [CrossRef]
- Schneeweiss, S.; Seeger, J.D.; Landon, J.; Walker, A.M. Aprotinin during coronary-artery bypass grafting and risk of death. N. Engl. J. Med. 2008, 358, 771–783. [Google Scholar] [CrossRef]
- Nissen, S.E.; Wolski, K. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. N. Engl. J. Med. 2007, 356, 2457–2471. [Google Scholar]
- DeStefano, F. The Vaccine Safety Datalink project. Pharmacoepidemiol. Drug Saf. 2001, 10, 403–406. [Google Scholar] [CrossRef]
- U.S. Food and Drug Administration (FDA). FDA’s Sentinel Initiative. Available online: http://www.fda.gov/Safety/FDAsSentinelInitiative/default.htm (accessed on 3 February 2012).
- Robb, M.A.; Racoosin, J.A.; Sherman, R.E.; Gross, T.P.; Ball, R.; Reichman, M.E.; Midthun, K.; Woodcock, J. The US Food and Drug Administration’s Sentinel Initiative: Expanding the horizons of medical product safety. Pharmacoepidemiol. Drug Saf. 2012, 21, 9–11. [Google Scholar] [CrossRef]
- Foundation for the National Institutes of Health. Observational Medical Outcomes Partnership (OMOP). Available online:  (accessed on 13 July 2011).
- European Medicines Agency. The IMI-PROTECT project (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium). Available online:  (accessed on 13 July 2011).
- Arlett, P.R.; Kurz, X. New approaches to strengthen pharmacovigilance. Drug Discov. Today Technol. 2011, 8, e15–e19. [Google Scholar] [CrossRef]
- Trifirò, G.; Pariente, A.; Coloma, P.M.; Kors, J.A.; Polimeni, G.; Miremont-Salamé, G.; Catania, M.A.; Salvo, F.; David, A.; Moore, N.; et al. Data mining on electronic health record databases for signal detection in pharmacovigilance: Which events to monitor? Pharmacoepidemiol. Drug Saf. 2009, 18, 1176–1184. [Google Scholar]
- Trifirò, G.; Patadia, V.; Schuemie, M.J.; Coloma, P.M.; Gini, R.; Herings, R.; Hippisley-Cox, J.; Mazzaglia, G.; Giaquinto, C.; Scotti, L.; et al. EU-ADR healthcare database network vs. spontaneous reporting system database: Preliminary comparison of signal detection. Stud. Health. Technol. Inform. 2011, 166, 25–30. [Google Scholar]
- Olsson, S. The role of the WHO programme on international drug monitoring in coordinating worldwide drug safety efforts. Drug Saf. 1998, 19, 1–10. [Google Scholar] [CrossRef]
- Coulter, D.M. The New Zealand intensive medicines monitoring programme in pro-active safety surveillance. Pharmacoepidemiol. Drug Saf. 2000, 9, 273–280. [Google Scholar] [CrossRef]
- Pigeot, I.; Ahrens, W. Establishment of a pharmacoepidemiological database in Germany: methodological potential, scientific value and practical limitations. Pharmacoepidemiol. Drug Saf. 2008, 17, 215–223. [Google Scholar] [CrossRef]
- Park, M.Y.; Yoon, D.; Lee, K.; Kang, S.Y.; Park, I.; Lee, S.-H.; Kim, W.; Kam, H.J.; Lee, Y.-H.; Kim, J.H.; et al. A novel algorithm for detection of adverse drug reaction signals using a hospital electronic medical record database. Pharmacoepidemiol. Drug Saf. 2011, 20, 598–607. [Google Scholar] [CrossRef]
- Harpaz, R.; DuMouchel, W.; Shah, N.H.; Madigan, D.; Ryan, P.; Friedman, C. Novel data-mining methodologies for adverse drug event discovery and analysis. Clin. Pharmacol. Ther. 2012, 91, 1010–1021. [Google Scholar] [CrossRef]
- Ahmad, S.R. Adverse drug event monitoring at the Food and Drug Administration. J. Gen. Intern. Med. 2003, 18, 57–60. [Google Scholar] [CrossRef]
- Platt, R.; Carnahan, R.M.; Brown, J.S.; Chrischilles, E.; Curtis, L.H.; Hennessy, S.; Nelson, J.C.; Racoosin, J.A.; Robb, M.; Schneeweiss, S.; et al. The U.S. Food and Drug Administration’s Mini-Sentinel program: Status and direction. Pharmacoepidemiol. Drug Saf. 2012, 21, 1–8. [Google Scholar]
- Carnahan, R.M. Mini-Sentinel’s systematic reviews of validated methods for identifying health outcomes using administrative data: Summary of findings and suggestions for future research. Pharmacoepidemiol. Drug Saf. 2012, 21, 90–99. [Google Scholar] [CrossRef]
- Schuemie, M.J. Methods for drug safety signal detection in longitudinal observational databases: LGPS and LEOPARD. Pharmacoepidemiol. Drug Saf. 2011, 20, 292–299. [Google Scholar] [CrossRef]
- Choi, N.-K.; Chang, Y.; Choi, Y.K.; Hahn, S.; Park, B.-J. Signal detection of rosuvastatin compared to other statins: Data-mining study using national health insurance claims database. Pharmacoepidemiol. Drug Saf. 2010, 19, 238–246. [Google Scholar] [CrossRef]
- Choi, N.-K.; Chang, Y.; Kim, J.-Y.; Choi, Y.-K.; Park, B.-J. Comparison and validation of data-mining indices for signal detection: using the Korean national health insurance claims database. Pharmacoepidemiol. Drug Saf. 2011, 20, 1278–1286. [Google Scholar] [CrossRef]
- Kim, J.; Kim, M.; Ha, J.-H.H.; Jang, J.; Hwang, M.; Lee, B.K.; Chung, M.W.; Yoo, T.M.; Kim, M.J. Signal detection of methylphenidate by comparing a spontaneous reporting database with a claims database. Regul. Toxicol. Pharmacol. 2011, 61, 154–160. [Google Scholar] [CrossRef]
- Curtis, J.R.; Cheng, H.; Delzell, E.; Fram, D.; Kilgore, M.; Saag, K.; Yun, H.; DuMouchel, W. Adaptation of Bayesian data mining algorithms to longitudinal claims data: Coxib safety as an example. Med. Care. 2008, 46, 969–975. [Google Scholar] [CrossRef]
- Zorych, I.; Madigan, D.; Ryan, P.; Bate, A. Disproportionality methods for pharmacovigilance in longitudinal observational databases. Stat. Methods Med. Res. 2011. [Google Scholar] [CrossRef]
- Coloma, P.M.; Schuemie, M.J.; Trifirò, G.; Gini, R.; Herings, R.; Hippisley-Cox, J.; Mazzaglia, G.; Giaquinto, C.; Corrao, G.; Pedersen, L.; et al. Combining electronic healthcare databases in Europe to allow for large-scale drug safety monitoring: the EU-ADR Project. Pharmacoepidemiol. Drug Saf. 2011, 20, 1–11. [Google Scholar] [CrossRef]
- Brown, J.S.; Kulldorff, M.; Petronis, K.R.; Reynolds, R.; Chan, K.A.; Davis, R.L.; Graham, D.; Andrade, S.E.; Raebel, M.A.; Herrinton, L.; et al. Early adverse drug event signal detection within population-based health networks using sequential methods: Key methodologic considerations. Pharmacoepidemiol. Drug Saf. 2009, 18, 226–234. [Google Scholar] [CrossRef]
- Madigan, D.; Ryan, P.; Simpson, S.; Zorych, I. Bayesian methods in pharmacovigilance. In Bayesian Statistics 9; Bernardo, J.M., Bayarri, M.J., Berger, J.O., Dawid, A.P., Heckerman, D., Smith, A.F.M., West, M., Eds.; Oxford University Press: Oxford, England, 2011; pp. 421–438. [Google Scholar]
- Wald, A. Sequential tests of statistical hypotheses. Ann. Math. Stat. 1945, 16, 117–186. [Google Scholar] [CrossRef]
- Wald, A. Sequential Analysis; Wiley: New York, NY, USA, 1947. [Google Scholar]
- Mini-Sentinel Coordinating Center (MSCC). Statistical methods development. Available online: http://mini-sentinel.org/foundational_activities/methods_development/ (accessed on 13 February 2012).
- Observational Medical Outcomes Partnership. OMOP methods library. Available online: MethodsLibrary (accessed on 13 February 2012).
- Almenoff, J.S.; DuMouchel, W.; Kindman, L.A.; Yang, X.; Fram, D. Disproportionality analysis using empirical Bayes data mining: A tool for the evaluation of drug interactions in the post-marketing setting. Pharmacoepidemiol. Drug Saf. 2003, 12, 517–521. [Google Scholar] [CrossRef]
- Van Puijenbroek, E.P.; Bate, A.; Leufkens, H.G.M.; Lindquist, M.; Orre, R.; Egberts, A.C.G. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol. Drug Saf. 2002, 11, 3–10. [Google Scholar] [CrossRef]
- DuMouchel, W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Am. Stat. 1999, 53, 177–190. [Google Scholar] [CrossRef]
- DuMouchel, W.; Pregibon, D. Empirical bayes screening for multi-item associations. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 26–29 2001; Association for Computing Machinery: San Francisco, CA, USA, 2001; pp. 67–76. [Google Scholar]
- Bate, A.; Lindquist, M.; Edwards, I.R.; Olsson, S.; Orre, R.; Lansner, A.; de Freitas, R.M. A Bayesian neural network method for adverse drug reaction signal generation. Eur. J. Clin. Pharmacol. 1998, 54, 315–321. [Google Scholar] [CrossRef]
- Evans, S.J.W.; Waller, P.C.; Davis, S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol. Drug Saf. 2001, 10, 483–486. [Google Scholar] [CrossRef]
- Lansner, A.; Ekeberg, Ö. A one-layer feedback artificial neural network with a bayesian learning rule. Int. J. Neural Syst. 1989, 1, 77–78. [Google Scholar] [CrossRef]
- Hauben, M.; Bate, A. Decision support methods for the detection of adverse events in post-marketing data. Drug Discov. Today 2009, 14, 343–357. [Google Scholar] [CrossRef]
- Bousquet, C.; Henegar, C.; Lillo-Le Louët, A.; Degoulet, P.; Jaulent, M.-C. Implementation of automated signal generation in pharmacovigilance using a knowledge-based approach. Int. J. Med. Inform. 2005, 74, 563–571. [Google Scholar] [CrossRef]
- Yates, F. Contingency tables involving small numbers and the chi-square-test. Suppl. J. R. Stat. Soc. 1934, 1, 217–235. [Google Scholar] [CrossRef]
- Goodman, L.A. William Henry Kruskal. In Measures of Association for Cross Classification; Springer: New York, NY, USA, 1979. [Google Scholar]
- MacKay, D.J.C. Information Theory. In Inference and Learning Algorithms; Cambridge University Press: Cambridge, UK, 2003. [Google Scholar]
- Raiffa, H.; Schlaifer, R. Applied Statistical Decision Theory; Harvard University: Boston, MA, USA, 1961. [Google Scholar]
- Gould, A.L. Practical pharmacovigilance analysis strategies. Pharmacoepidemiol. Drug Saf. 2003, 12, 559–574. [Google Scholar] [CrossRef]
- Norén, G.N.; Bate, A.; Orre, R.; Edwards, I.R. Extending the methods used to screen the WHO drug safety database towards analysis of complex associations and improved accuracy for rare events. Stat. Med. 2006, 25, 3740–3757. [Google Scholar] [CrossRef]
- Ahmed, I.; Poncet, A. PhViD, Version 1.0.4. A R Package for PharmacoVigilance Signal Detection, Available online:  (accessed on 6 December,2012).
- Anonymous. The R Project for Statistical Computing. Available online:  (accessed on 13 February 2012).
- Pogson, G.W.; Kindred, L.H.; Carper, B.G. Rhabdomyolysis and renal failure associated with cerivastatin-gemfibrozil combination therapy. Am. J. Cardiol. 1999, 83, 1146. [Google Scholar] [CrossRef]
- Van Puijenbroek, E.P.; Egberts, A.C.; Meyboom, R.H.B.; Leufkens, H.G.M. Signalling possible drug-drug interactions in a spontaneous reporting system: delay of withdrawal bleeding during concomitant use of oral contraceptives and itraconazole. Br. J. Clin. Pharmacol. 1999, 47, 689–693. [Google Scholar]
- Van Puijenbroek, E.P.; Egberts, A.C.G.; Heerdink, E.R.; Leufkens, H.G.M. Detecting drug-drug interactions using a database for spontaneous adverse drug reactions: An example with diuretics and non-steroidal anti-inflammatory drugs. Eur. J. Clin. Pharmacol. 2000, 56, 733–738. [Google Scholar] [CrossRef]
- Norén, G.N.; Sundberg, R.; Bate, A.; Edwards, I.R. A statistical methodology for drug–drug interaction surveillance. Stat. Med. 2008, 27, 3057–3070. [Google Scholar] [CrossRef]
- Farrington, C.P. Relative incidence estimation from case series for vaccine safety evaluation. Biometrics 1995, 51, 228–235. [Google Scholar] [CrossRef]
- Taylor, B.; Miller, E.; Farrington, C.P.; Petropoulos, M.C.; Favot-Mayaud, I.; Li, J.; Waight, P.A. Autism and measles, mumps, and rubella vaccine: No epidemiological evidence for a causal association. Lancet 1999, 353, 2026–2029. [Google Scholar]
- Stata Statistical Software, StataCorp, College Station, TX, USA, 2011. Release 12.
- SAS, SAS Institute Inc, Cary, NC, USA, 2011. Release 9.3.
- Hocine, M.N.; Musonda, P.; Andrews, N.J.; Farrington, C.P. Sequential case series analysis for pharmacovigilance. J. R. Stat. Soc. 2009, 172, 213–236. [Google Scholar] [CrossRef]
- Maclure, M.; Fireman, B.; Nelson, J.C.; Hua, W.; Shoaibi, A.; Paredes, A.; Madigan, D. When should case-only designs be used for safety monitoring of medical products? Pharmacoepidemiol. Drug Saf. 2012, 21, 50–61. [Google Scholar] [CrossRef]
- Norén, G.N.; Hopstadius, J.; Bate, A.; Star, K.; Edwards, I.R. Temporal pattern discovery in longitudinal electronic patient records. Data Min. Knowl. Discov. 2010, 20, 361–387. [Google Scholar]
- Murray, R.E.; Ryan, P.B.; Reisinger, S.J. Design and validation of a data simulation model for longitudinal healthcare data. AMIA Annu. Symp. Proc. 2011, 2011, 1176–1185. [Google Scholar]
- OMOP. OSIM-Observational Medical Dataset Simulator. Available online: OSIM (accessed on 14 February 2012).
- Thomson Reuters. MarketScan databases. Available online:  (accessed on 14 February 2012).
- Vlug, A.E.; van der Lei, J.; Mosseveld, B.M.Th.; van Wijk, M.A.M.; van der Linden, P.D.; Sturkenboom, M.C.J.M.; van Bemmel, J.H. Postmarketing surveillance based on electronic patient records: The IPCI project. Methods Inf. Med. 1999, 38, 339–344. [Google Scholar]
- Schuemie, M.J.; Coloma, P.M.; Straatman, H.; Herings, R.M.; Trifirò, G.; Matthews, J.N.; Prieto-Merino, D.; Molokhia, M.; Pedersen, L.; Gini, R.; et al. Using electronic health care records for drug safety signal detection: A comparative evaluation of statistical methods. Med. Care. 2012, 10, 890–897. [Google Scholar]
- Ryan, P.B.; Madigan, D.; Stang, P.E.; Marc Overhage, J.; Racoosin, J.A.; Hartzema, A.G. Empirical assessment of methods for risk identification in healthcare data: Results from the experiments of the Observational Medical Outcomes Partnership. Stat. Med. 2012. in press. [Google Scholar]
- Schneeweiss, S.; Rassen, J.A.; Glynn, R.J.; Avorn, J.; Mogun, H.; Brookhart, M.A. High-dimensional propensity score adjustment in studies of treatment effects using health care claims data. Epidemiology 2009, 20, 512–522. [Google Scholar] [CrossRef]
- Rassen, J.A.; Schneeweiss, S. Using high-dimensional propensity scores to automate confounding control in a distributed medical product safety surveillance system. Pharmacoepidemiol. Drug Saf. 2012, 21, 41–49. [Google Scholar] [CrossRef]
- Fawcett, T. An introduction to ROC analysis. Pattern Recognit. Lett. 2006, 27, 861–874. [Google Scholar]
- Manning, C.D.; Raghavan, P.; Schütze, H. Introduction to Information Retrieval; Cambridge Univ Press: Cambridge, UK, 2008. [Google Scholar]
- Lu, Z. Information technology in pharmacovigilance: Benefits, challenges, and future directions from industry perspectives. Drug Healthc. Patient Saf. 2009, 1, 35–45. [Google Scholar] [CrossRef]
- Rosenbaum, P.R.; Rubin, D.B. The central role of the propensity score in observational studies for causal effects. Biometrika 1983, 70, 41–55. [Google Scholar] [CrossRef]
- Garbe, E.; Kloss, S.; Suling, M.; Pigeot, I.; Schneeweiss, S. High-dimensional versus conventional propensity scores in a comparative effectiveness study of coxibs and reduced upper gastrointestinal complications. Eur. J. Clin. Pharmacol. 2012. [Google Scholar] [CrossRef]
- Greenland, S. Regression methods for epidemiologic analysis. In Handbook of Epidemiology; Ahrens, W., Pigeot, I., Eds.; Springer: Berlin/Heidelberg, Germany, 2005; pp. 625–691. [Google Scholar]
- Genkin, A.; Lewis, D.D.; Madigan, D. Large-scale Bayesian logistic regression for text categorization. Technometrics 2007, 49, 291–304. [Google Scholar] [CrossRef]
- Caster, O.; Norén, G.N.; Madigan, D.; Bate, A. Large-scale regression-based pattern discovery: The example of screening the WHO global drug safety database. Stat. Anal. Data Min. 2010, 3, 197–208. [Google Scholar]
- Ahmed, I.; Dalmasso, C.; Haramburu, F.; Thiessard, F.; Broët, P.; Tubert-Bitter, P. False discovery rate estimation for frequentistpharmacovigilance signal detection methods. Biometrics 2010, 66, 301–309. [Google Scholar] [CrossRef]
- Ahmed, I.; Thiessard, F.; Miremont-Salamé, G.; Haramburu, F.; Kreft-Jais, C.; Bégaud, B.; Tubert-Bitter, P. Early detection of pharmacovigilance signals with automated methods based on false discovery rates: A comparative study. Drug Saf. 2012, 6, 495–506. [Google Scholar]
- Benjamini, Y.; Hochberg, Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. 1995, 57, 289–300. [Google Scholar]
- Ståhl, M.; Lindquist, M.; Edwards, I.R.; Brown, E.G. Introducing triage logic as a new strategy for the detection of signals in the WHO Drug Monitoring Database. Pharmacoepidemiol. Drug Saf. 2004, 13, 355–363. [Google Scholar] [CrossRef]
- Lindquist, M. Use of triage strategies in the WHO signal-detection process. Drug Saf. 2007, 30, 635–637. [Google Scholar] [CrossRef]
- Horwitz, R.I.; Feinstein, A.R. The problem of protopathic bias in case-control studies. Am. J. Med. 1980, 68, 255–258. [Google Scholar] [CrossRef]
- Schneeweiss, S. A basic study design for expedited safety signal evaluation based on electronic healthcare data. Pharmacoepidemiol. Drug Saf. 2010, 19, 858–868. [Google Scholar] [CrossRef]
- Kulldorff, M.; Davis, R.L.; Kolczak, M.; Lewis, E.; Lieu, T.; Platt, R. A maximized sequential probability ratio test for drug and vaccine safety surveillance. Seq. Anal. 2011, 30, 58–78. [Google Scholar] [CrossRef]
- Li, L.; Kulldorff, M. A conditional maximized sequential probability ratio test for pharmacovigilance. Stat. Med. 2010, 29, 284–295. [Google Scholar]
- Brown, J.S.; Kulldorff, M.; Chan, K.A.; Davis, R.L.; Graham, D.; Pettus, P.T.; Andrade, S.E.; Raebel, M.A.; Herrinton, L.; Roblin, D.; et al. Early detection of adverse drug events within population-based health networks: application of sequential testing methods. Pharmacoepidemiol. Drug Saf. 2007, 16, 1275–1284. [Google Scholar] [CrossRef]
- Platt, R.; Andrade, S.E.; Davis, R.L.; DeStefano, F.; Finkelstein, J.A.; Goodman, M.J.; Gurwitz, J.H.; Go, A.S.; Martinson, B.C.; Raebel, M.A.; et al. Pharmacovigilance in the HMO research network. In Pharmacovigilance; John Wiley & Sons, Ltd.: New York, NY, USA, 2002; pp. 391–398. [Google Scholar]
- Li, L. A conditional sequential sampling procedure for drug safety surveillance. Stat. Med. 2009, 28, 3124–3138. [Google Scholar] [CrossRef]
- Davis, R.L.; Kolczak, M.; Lewis, E.; Nordin, J.; Goodman, M.; Shay, D.K.; Platt, R.; Black, S.; Shinefield, H.; Chen, R.T. Active surveillance of vaccine safety: A system to detect early signs of adverse events. Epidemiology 2005, 3, 336–341. [Google Scholar]
- Yih, W.K.; Kulldorff, M.; Fireman, B.H.; Shui, I.M.; Lewis, E.M.; Klein, N.P.; Baggs, J.; Weintraub, E.S.; Belongia, E.A.; Naleway, A.; et al. Active surveillance for adverse events: The experience of the Vaccine Safety Datalink project. Pediatrics 2011, 127, 54–64. [Google Scholar]
- Li, L.; Kulldorff, M.; Nelson, J.; Cook, A. A propensity score-enhanced sequential analytic method for comparative drug safety surveillance. Stat. Biosci. 2011, 3, 45–62. [Google Scholar] [CrossRef]
- Jin, H.W.; Chen, J.; He, H.; Williams, G.J.; Kelman, C.; O’Keefe, C.M. Mining unexpected temporal associations: applications in detecting adverse drug reactions. IEEE Trans. Inf. Technol. Biomed. 2008, 12, 488–500. [Google Scholar] [CrossRef]
- Jin, H.W.; Chen, J.; He, H.; Kelman, C.; McAullay, D.; O’Keefe, C.M. Signaling potential adverse drug reactions from administrative health databases. IEEE Trans. Knowl. Data Eng. 2010, 22, 839–853. [Google Scholar]
- Grigg, O.; Farewell, V. An overview of risk-adjusted charts. J. R. Stat. Soc. 2004, 167, 523–539. [Google Scholar] [CrossRef]
- Nelson, J.C.; Cook, A.; Yu, O. Evaluation of signal detection methods for use in prospective post licensure medical product safety surveillance. Available online: ;D=FDA-2009-N-0192–0002 (accessed on 14 August 2012).
- Spiegelhalter, D.; Grigg, O.; Kinsman, R.; Treasure, T. Risk-adjusted sequential probability ratio tests: applications to Bristol, Shipman and adult cardiac surgery. Int. J. Qual. Health Care 2003, 15, 7–13. [Google Scholar] [CrossRef]
- Norén, G.N.; Bate, A.; Hopstadius, J.; Star, K.; Edwards, I.R. Temporal pattern discovery for trends and transient effects: Its application to patient records. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; Association for Computing Machinery: Las Vegas, NV, USA, 2008; pp. 963–971. [Google Scholar]
- Hauben, M.; Norén, G.N. A decade of data mining and still counting. Drug Saf. 2010, 33, 527–534. [Google Scholar] [CrossRef]
- Madigan, D.; Ryan, P. Commentary: What can we really learn from observational studies? The need for empirical assessment of methodology for active drug safety surveillance and comparative effectiveness research. Epidemiology 2011, 22, 629. [Google Scholar] [CrossRef]
- Ray, W.A. Improving automated database studies. Epidemiology 2011, 22, 302–304. [Google Scholar] [CrossRef]
- Walker, A.M. Signal detection for vaccine side effects that have not been specified in advance. Pharmacoepidemiol. Drug Saf. 2010, 19, 311–317. [Google Scholar] [CrossRef]
- Reisinger, S.J.; Ryan, P.B.; O’Hara, D.J.; Powell, G.E.; Painter, J.L.; Pattishall, E.N.; Morris, J.A. Development and evaluation of a common data model enabling active drug safety surveillance using disparate healthcare databases. J. Am. Med. Inform. Assoc. 2010, 17, 652–662. [Google Scholar] [CrossRef]
- Avillach, P.; Joubert, M.; Thiessard, F.; Trifirò, G.; Dufour, J.C.; Pariente, A.; Mougin, F.; Polimeni, G.; Catania, M.A.; Giaquinto, C. Design and evaluation of a semantic approach for the homogeneous identification of events in eight patient databases: A contribution to the European EU-ADR project. Stud. Health. Technol. Inform. 2010, 160, 1085. [Google Scholar]