Objectives To explore agreement among healthcare professionals assessing eligibility for work disability benefits.
Design Systematic review and narrative synthesis of reproducibility studies.
Data sources Medline, Embase, and PsycINFO searched up to 16 March 2016, without language restrictions, and review of bibliographies of included studies.
Eligibility criteria Observational studies investigating reproducibility among healthcare professionals performing disability evaluations using a global rating of working capacity and reporting inter-rater reliability by a statistical measure or descriptively. Studies could be conducted in insurance settings, where decisions on ability to work include normative judgments based on legal considerations, or in research settings, where decisions on ability to work disregard normative considerations.Teams of paired reviewers identified eligible studies, appraised their methodological quality and generalisability, and abstracted results with pretested forms. As heterogeneity of research designs and findings impeded a quantitative analysis, a descriptive synthesis stratified by setting (insurance or research) was performed.
Results From 4562 references, 101 full text articles were reviewed. Of these, 16 studies conducted in an insurance setting and seven in a research setting, performed in 12 countries, met the inclusion criteria. Studies in the insurance setting were conducted with medical experts assessing claimants who were actual disability claimants or played by actors, hypothetical cases, or short written scenarios. Conditions were mental (n=6, 38%), musculoskeletal (n=4, 25%), or mixed (n=6, 38%). Applicability of findings from studies conducted in an insurance setting to real life evaluations ranged from generalisable (n=7, 44%) and probably generalisable (n=3, 19%) to probably not generalisable (n=6, 37%). Median inter-rater reliability among experts was 0.45 (range intraclass correlation coefficient 0.86 to κ−0.10). Inter-rater reliability was poor in six studies (37%) and excellent in only two (13%). This contrasts with studies conducted in the research setting, where the median inter-rater reliability was 0.76 (range 0.91-0.53), and 71% (5/7) studies achieved excellent inter-rater reliability. Reliability between assessing professionals was higher when the evaluation was guided by a standardised instrument (23 studies, P=0.006). No such association was detected for subjective or chronic health conditions or the studies' generalisability to real world evaluation of disability (P=0.46, 0.45, and 0.65, respectively).
Conclusions Despite their common use and far reaching consequences for workers claiming disabling injury or illness, research on the reliability of medical evaluations of disability for work is limited and indicates high variation in judgments among assessing professionals. Standardising the evaluation process could improve reliability. Development and testing of instruments and structured approaches to improve reliability in evaluation of disability are urgently needed.
Many workers seek wage replacement benefits on the basis of disabling illness or injury, and over the past decade most countries of the Organisation for Economic Co-operation and Development (OECD) have experienced escalating rates of affected workers.12 Current estimates range from four to eight individuals per thousand per year,2 corresponding to 16 000 newly affected workers/year for smaller countries like Switzerland and 1 700 000/year for countries like the US.
Equality before the law requires that claimants with similar health impairments and exposed to similar work demands should receive similar judgments of medical restrictions and limitations. Concerns have been raised, however, regarding low quality evaluations678 and poor reliability between medical experts.91011121314 Evaluation of disability is a complex process that is affected by the skillset, attitudes, and beliefs of the expert, and few countries enforce standards of practice,35 which presents considerable challenges to reliability (box 1).1516 We conducted a systematic review of reproducibility studies to summarise empirical evidence regarding the inter-rater reliability of global judgments on work disability and examined the hypothesis that studies using standardised assessments would show higher reliability.
We followed the standards set by the Guidelines for Reporting Reliability and Agreement Studies (GRRAS)17 and Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA)18 for the reporting of our study.
We included reproducibility studies conducted in an insurance setting (evaluation of claimants) or in a research setting (evaluation of patients for work disability outside of actual assessments) in which two or more health professionals evaluated the work capacity of individuals claiming disability and reported inter-rater reliability on a global rating of work disability. Studies that reported only the inter-rater reliability of experts' evaluation of specific physical or mental activities (such as lifting, conflict management) were excluded. All types of "subjects" qualified: real claimants, records of claimants, videotaped actors, vignettes, short case summaries.
We searched Medline, Embase, and PsycInfo from inception to 16 March 2016, without language restrictions. An experienced medical librarian (RC) developed database specific search strategies combining the following subject terms: reproducibility of results (MeSH, including reliability) and reliability statistics, disability or work capacity evaluation, and sick leave (see appendix 1 for the detailed search strategy). We screened the bibliographies of all included studies for additional relevant articles.
Three teams of paired reviewers (WdB, JWB, JH, SK, JS, RK) with expertise in medical evaluations and training in research methodology independently screened titles, abstracts, and full texts for eligibility, assessed generalisability, and collected data from each eligible study using standardised pilot tested forms with detailed instructions. Reviewers resolved disagreement through discussion or, if required, adjudication by a third reviewer (RK or WdB).
We gave more weight to studies with a broader spectrum and a larger number of experts to reflect the wide variation among medical experts in actual disability assessment, which tends to contribute substantially to the measurement error.
Five reviewers (JB, RK, WdB, JS, JanHo), blinded to the study results, assessed generalisability of each study, independently and in duplicate. Given the lack of empirical evidence about the relative importance of each item we used a sequential approach from medical decision making21 to make the weighting of each item explicit (see appendix 2 for detailed description). This approach facilitated judgments regarding overall generalisability (that is, "generalisable," "probably generalisable," "probably not generalisable," and "not generalisable"). We calculated the reviewers' concordance in generalisability ranking using Kendall's W (coefficient of concordance), which generates values between zero (no agreement) and one (perfect agreement).
We limited assessment of generalisability to studies performed in an insurance setting because studies conducted in a research setting, by definition ("normative or legal considerations not part of the judgment", see data analysis), lack generalisability to real life assessments of disability.
Outcomes—global rating of work disability (for example, work capacity, sick leave, readiness for return to work, reduction in working hours); decisions on suitability for a specific job; occupational functioning); measure of reliability or agreement (intraclass correlation coefficient (ICC), κ statistic, or percentage agreement), including measure of precision, or descriptive measure (for example, frequency of judgments).
For three studies that reported on reduction of working hours,222324 we calculated the κ statistics2223 and intraclass correlation,24 based on the raw data provided by authors.
We distinguished between studies conducted in an insurance setting or a research setting. In an insurance setting, health professionals make judgments on disability for work based on functional limitations that includes normative judgments from a societal perspective. An insurance setting does not imply any specific format of the claimant's presentation in the study, which can range from a real patient to a written case (see also "generalisability, verisimilitude"). Researchers in a research setting who develop and/or validate instruments tend to standardise their research environment when judging occupational functioning. Normative (legal) considerations or a societal perspective are not part of their judgments.
We used studies conducted in a research setting to investigate the association between level of standardisation in the evaluation process and inter-rater reproducibility. Level of standardisation was considered as "not standardised" when medical experts in the insurance setting used only their professional expertise to elicit information and rate findings from the claimant; as "semi-standardised" when they used a structured instrument as one component of the evaluation; and as "fully standardised" when occupational functioning was primarily evaluated with a structured instrument.
Lack of information on variation associated with reproducibility statistics and heterogeneity of statistical measures and outcomes precluded pooling of the data across studies. Using a two tailed Fisher's exact test, we explored whether objective (versus subjective) and acute (versus chronic) health conditions as well as higher levels of generalisability and/or higher levels of standardisation in the evaluation process were associated with a higher inter-rater reproducibility. We defined mental disorders as "subjective complaints" and somatic disorders as objective complaints, though we acknowledge the crude nature of this classification, and acute conditions shorter than six months and chronic conditions longer than six months. We excluded from our analysis three studies that did not specify the chronicity. Fisher's exact test does not provide a test statistic, only whether the difference is significant or not.
No patients were involved in setting the research question, in developing plans for design, interpretation, reporting or implementation of the study. We plan to disseminate the results of this study to organisations supporting patients with disabilities.
Of 4562 potentially relevant citations identified, 101 reports proved potentially eligible after we had screened titles and abstracts. On full text screening, 23 studies,911222324293031323334353637383940414243444546 including four non-English studies,9394041 proved eligible for analysis (fig 1⇓). All studies were published from 1992 onwards and enrolled disability claimants from 12 countries in Europe, North America, Australia, the Middle East, and northeast Asia. Seven studies (30%) were conducted in the Netherlands. Seventy percent of the studies (16/23) were conducted in an insurance setting, with the remainder in a research setting. Investigators used a broad spectrum of designs, ranging from real life disability evaluations, videotapes with actors, and records of claimants to 10 line case vignettes, to perform reliability studies. Study size varied considerably with number of raters from two to 103 and number of patients from one and 3562 per study (tables 1 and 2⇓).
Assessment of methodological quality included blinding of raters to each other's findings, presence of order effects, and appropriateness of the statistical analyses (table 3⇓; appendix 2). The studies on the reproducibility between medical experts conducted in an insurance setting met 80% (31/39) of these items, 15% (6/39) remained unclear, and 5% (2/39) were not applicable. The methodological quality items did not fit the design of the studies that looked at the reproducibility between medical experts and health professionals. Studies conducted in a research setting met 52% (11/21) of the quality items; 33% (7/21) remained unclear and 14% (3/21) were not met (table 3⇓).
With regards to generalisability of the findings to real life disability evaluation, 44% (7/16) of studies in the insurance setting were rated as "generalisable," 19% (3/16) as "probably generalisable," and 37% (6/16) as "probably not generalisable" (table 4⇓). Kendall's W for reviewers' concordance in ranking generalisability was 0.93, with a rank correlation of 0.89, confirming high agreement among the raters' rankings.
In the insurance setting, 13 studies including 463 patients and 367 raters explored agreement between medical experts (two or more experts assessing the same patient) (table 1⇑; appendix 4).9112223243234373943444546 Three studies including 3729 patients (with 3562 patients from a single centre33) and eight raters (information was lacking from one study33) explored agreement between medical experts and claimant's treating physicians33 or independent rehabilitation or occupational health teams with a mandate to care.3842 The median number of patients per study was 13.5 (range 1-3562), and the median number of raters per study was 12 (2-103, excluding one study that did not report the number of raters33). All but three studies24342 used a fully crossed design (that is, all raters evaluated all patients), with a median of 11 patients (range 1-180) per rater and a median of 11.5 raters (2-103) per patient.
Table 5⇓ summarises claimants' characteristics. Studies focused on mental health (n=6), musculoskeletal disease (n=4), and mixed disorders (n=6). They enrolled patients with chronic diseases (n=11), chronic injuries (n=2), or mixed, acute, and chronic conditions (n=3). Most referred to a long term time frame before the evaluation for judging health status and work disability and predicted a long term perspective exceeding six months. Most studies used professional expertise only to generate a global rating of work ability (n=10). Six administered one or more specific rating instruments; five were referenced (appendix 3), and none was reported as validated.
Work disability outcomes varied considerably between studies and included a broad spectrum of domains, definitions, and measurement approaches, ranging from work ability to the employee's readiness and ability to return to work, the degree of disability or handicap, or reduction in working hours. Measurement approaches included scales, scores, and categories (table 6⇓).
Studies conducted in a research setting included 371 patients and 32 raters (table 2⇑; appendix 4). Four studies reported on instrument development,29353641 and three studies validated existing instruments.303140 The median number of patients per study was 39 (range 20-180), and the median number of raters per study was three (2-18). All but two studies2940 used a fully crossed design, with a median of 21 patients (11-42) per rater and a median of two raters (2-4) per patient.
All studies were conducted with actual patients and focused on acute and chronic mental health conditions. Most used a short term time frame before the evaluation for judging occupational functioning, two provided a short term prognostic judgement on occupational functioning, and this information was missing in five studies (table 2⇑). All seven studies used instruments of varying complexity to elicit or to report capacities or limitations to determine a global rating for occupational functioning (appendix 4). All studies generated global ratings on a range of outcomes for occupational functioning, such as "occupational functioning" or "remunerative employment" (table 7⇓).
Overall, across conditions and outcomes, the median inter-rater reliability was 0.76, ranging from an ICC of 0.91 (anxiety and mood disorders; occupational functioning35) to κ of 0.53 (mixed mental disorders; occupational functioning31).
Testing the relation between inter-rater reliability and subjective (versus objective) and chronic (versus acute) health conditions as well as the studies' overall generalisability did not show any association (subjectivity, 23 studies, P=0.46; chronicity, 20 studies, P=0.45; generalisability, 16 studies, P=0.65). Testing the relation between the level of standardisation and inter-rater reliability in all 23 studies showed a highly significant association (P=0.006).
Current evidence regarding reliability of disability evaluation is limited and shows highly variable agreement between medical experts. Higher agreement seems to be associated with the use of a standardised approach to guide judgment and studies in a research (manufactured) setting.
Strengths of our study include broad inclusion criteria to define eligibility and inclusion of publications in any language, which increases confidence that we captured all studies eligible for our review. Our outcome—global rating of disability for work—is highly relevant to the practice of medical experts, disability insurers, and employers, which increases the practical implications of our findings. Further, we evaluated the generalisability of evidence by following international guidance for evaluating reliability studies171920 and by using an explicit approach in eliciting reviewers' judgments on the relative weights of the generalisability items. While the high agreement we found among reviewers strengthens the credibility of the results, this approach requires further validation. Some cut offs of the generalisability criteria (such as number of raters) are context specific and might not be applicable to settings other than assessment of disability. Furthermore, variability of study designs, measures of agreement, and outcomes precluded statistical pooling across studies.
Disability evaluation is a poorly understood process141516 that lacks any reference standard to confirm the validity of the findings. Health professionals who perform this task assess medical restrictions and limitations of claimants and are often asked to infer consequences on the ability to work. This, however, requires expertise in vocational rehabilitation, as medical restrictions do not correlate well with function and the ability to work.5 In such situations, reliability studies evaluate the measurement properties of observers.47 At each step of disability evaluation, multiple sources of variation come into play (box 1),1516 including experts' personal attitudes, beliefs, and values towards disability, all of which affect the global judgment of work disability. Left unmanaged, these sources of variation can lead to low inter-rater reliability.
We found higher agreement when disability evaluation was guided by a standardised instrument. Instruments that standardise the collection, interpretation, and reporting of information are one promising approach to reduce variation.15 Five of the seven Dutch studies that used instruments to guide assessment of work disability all achieved fair to good reliability. As all Dutch insurance physicians undergo four years of specialty training in insurance medicine,48 however, we cannot disentangle whether higher agreement is a result of use of a formal instrument or calibration by training, or both.
We did not detect any association between inter-rater reliability and subjectivity or chronicity of the health conditions, or overall generalisability to real world disability evaluation. The low number of studies in the analyses, however, precludes any premature conclusions that such associations do not exist.
Not all sources of variations are easily accessible to change. Other sources, in particular attitudes, beliefs, and value judgments, will require other approaches.49 Implicit in the use of evaluations of disability by a third party is the concern that treating clinicians could have difficulty providing impartial assessments of their patients. Indeed, our findings suggest that medical experts (versus treating physicians) are more likely to conclude that claimants are capable of working. Claimant lawyers and patients' organisations have raised concerns that experts who are paid to assess claimants for insurers might feel pressure to render opinions that favour the referral source.
Our review suggests that use of standardised instruments could improve reliability in expert judgments on work disability. Appropriate instruments should therefore be considered in routine practice of disability evaluations (see table 8⇑ and appendix 3 for examples). To ensure appropriate administration and interpretation of the findings, experts will need appropriate training and calibration on the use of such instruments. As most instruments reported in this review are available only in Dutch, other countries would need to develop their own instruments or translate instruments and accompanying manuals in national languages.
As few countries have standards to guide assessments, standardised instruments that improve reliability could become a target for change and parties ordering assessments should demand their use.
Given the widespread use of evaluation of disability for work to determine claimants' eligibility for work replacement benefits, our findings suggest that further research to improve reliability is urgently needed. Promising targets include formal training in evaluation of capacity to work,50 use of standardised instruments to guide disability evaluations,50 and addressing the conflict of interest that arises when insurers (or lawyers) select their own experts. Further, there might be greater need for strategies to improve agreement when patients present with subjective complaints. Ikezawa and colleagues found that different medical experts were able to agree on claimant's ability to return to work in 97% of claims involving a fracture and 94% of claims involving a dislocation, but only 56% of claims because of chronic low back pain.37 Our review further suggests that interventions should be validated in real insurance settings, as experimental settings could artificially inflate agreement.
Improved knowledge of individual factors that contribute to variability in evaluation of capacity to work is also needed. Promising targets could provide a starting point to develop and test focused strategies to reduce variability (for example, appropriate assessment tools, guidelines, standard cases). Guidance is also required to inform the required level of inter-rater reliability to ensure equal treatment of claimants. Any decision on what constitutes an appropriate threshold, which might be similar to thresholds for clinical medical tests,2728 will require societal discussion on what constitutes acceptable differences in the treatment of claimants or align to standards set by professional organisations of psychology or education. To make evaluations on work disability fair and meaningful and thereby qualify for decisions on claimants' disability benefits, however, we suggest a minimum intraclass correlation coefficient of 0.6 (the cut off between fair and good inter-rater reliability), with a sufficiently narrow 95% confidence interval (0.5 to 0.7) to exclude poor reliability.
Despite their widespread use, medical evaluations of work disability show high variability and often low reliability. Use of standardised and validated instruments to guide the process could improve reliability. There is an urgent need for high quality research, conducted in actual insurance settings, to explore promising strategies to improve agreement in evaluation of capacity to work.
We thank Gordon Guyatt, McMaster University, Hamilton, Canada, for his input in conceptualising the review; Nozomi Takeshima, Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine, Japan, and Rosella Saulle and Laura Amato, Department of Epidemiology, Lazio Regional Health Service, Rome (Italy), for extracting the data from the Japanese and Italian studies; and Sacha Röschard for administrative support.
Contributors: RK and WdB developed the idea. RK, JB, WdB, JWB, KF, and GG contributed substantially to to the conception and design. RK, JB, WdB, JWB, RC, SK, JS, DvA, and KF contributed to acquisition, analysis, or interpretation of the data. RK, JB, JWB, JH, KF, SK, RC, DvA, and JS drafted or revised the manuscript critically for important intellectual content, approved the final version to be published. RK and JB are guarantors.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that JWB acts as a consultant to Prisma Health Canada, a private incorporated company funded by employers and insurers that consults on and manages long term disability claims. The Evidence-based Insurance Medicine Unit at the University Hospital in Basel is funded in part by donations from public insurance companies and a consortium of private insurance companies (RK). After the manuscript was finalised, RK took a part time position at the Swiss National Accident Insurance Fund, Suva. RK, JB, WdB, JWB, and JH were initiators of Cochrane Insurance Medicine.
Transparency: The lead author affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
International Social Security Association. Country Profiles. https://www.issa.int/country-profiles.
Sickness, Disability and Work: Breaking the Barriers. A Synthesis of Findings across OECD Countries. OECD, 2010 Contract No. ISBN 978-92-64-08884-9.
Anner J, Kunz R, Boer Wd. Reporting about disability evaluation in European countries. Disabil Rehabil2014;356:848-54. doi:10.3109/09638288.2013.821180 pmid:23919642.
Anner J, Schwegler U, Kunz R, Trezzini B, de Boer W. Evaluation of work disability and the international classification of functioning, disability and health: what to expect and what not. BMC Public Health2012;356:470. doi:10.1186/1471-2458-12-470 pmid:22720978.
Busse JW, Bruun-Meyer SE, Ebrahim S, Kunz R. A 45-year-old woman referred for an independent medical evaluation by her insurer. CMAJ2014;356:E627-30. doi:10.1503/cmaj.130863 pmid:25200753.
Pizala H. Evaluation von psychiatrischen Gutachten [Reports of disability evaluations in claimants with mental disorders. A quality assessment.] [Dissertation]. University of Basel; 2010.
Manchikanti L. Impairment evaluation in pain management: physician, or attorney in white coat?Pain Physician2000;356:201-17.pmid:16906199.
Peterson KW, Babitsky S, Beller TA, et al. The American Board of Independent Medical Examiners. J Occup Environ Med1997;356:509-14. doi:10.1097/00043764-199706000-00004 pmid:9211207.
Dickmann JR, Broocks A. [Psychiatric expert opinion in case of early retirement--how reliable?]. Fortschr Neurol Psychiatr2007;356:397-401. doi:10.1055/s-2006-944303 pmid:17031778.
Clark WL, Haldeman S, Johnson P, et al. Back impairment and disability determination. Another attempt at objective, reliable rating. Spine (Phila Pa 1976)1988;356:332-41. doi:10.1097/00007632-198803000-00019 pmid:3388120.
Ingravallo F, Vignatelli L, Brini M, et al. Medico-legal assessment of disability in narcolepsy: an interobserver reliability study. J Sleep Res2008;356:111-9. doi:10.1111/j.1365-2869.2008.00630.x pmid:18275561.
Alexiou G. Disabled people are trapped in assessment 'nightmare' by benefits regime, says Dr. Stephen Duckworth. The Independent 2014 22.4.Sect. Home News, 2014.
Kleinfield NR. Exams of injured workers fuel mutual mistrust. New York Times 2009. http://www.nytimes.com/2009/04/01/nyregion/01comp.html.
Hesse B, Gebauer E. Sozialmedizinische Begutachtung im Rentenverfahren: Stellenwert, Forschungsbedarf und Chancen. Rehabilitation (Stuttg)2011;356:17-24. doi:10.1055/s-0030-1270432 pmid:21321820.
Spanjer J, Krol B, Brouwer S, Groothoff JW. Sources of variation in work disability assessment. Work2010;356:405-11.pmid:21099015.
Kobak KA, Brown B, Sharp I, et al. Sources of unreliability in depression ratings. J Clin Psychopharmacol2009;356:82-5. doi:10.1097/JCP.0b013e318192e4d7 pmid:19142114.
Kottner J, Audigé L, Brorson S, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol2011;356:96-106. doi:10.1016/j.jclinepi.2010.03.002 pmid:21130355.
Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ2009;356:b2700. doi:10.1136/bmj.b2700 pmid:19622552.
Lucas NP, Macaskill P, Irwig L, Bogduk N. The development of a quality appraisal tool for studies of diagnostic reliability (QAREL). J Clin Epidemiol2010;356:854-61. doi:10.1016/j.jclinepi.2009.10.002 pmid:20056381.
Karanicolas PJ, Bhandari M, Kreder H, et al. Collaboration for Outcome Assessment in Surgical Trials (COAST) Musculoskeletal Group. Evaluating agreement: conducting a reliability study. J Bone Joint Surg Am2009;356(Suppl 3):99-106. doi:10.2106/JBJS.H.01624 pmid:19411507.
Edwards W. The theory of decision making. Psychol Bull1954;356:380-417. doi:10.1037/h0053870 pmid:13177802.
Spanjer J, Krol B, Popping R, Groothoff JW, Brouwer S. Disability assessment interview: the role of detailed information on functioning in addition to medical history-taking. J Rehabil Med2009;356:267-72. doi:10.2340/16501977-0323 pmid:19247547.
Spanjer J, Krol B, Brouwer S, Groothoff JW. Inter-rater reliability in disability assessment based on a semi-structured interview report. Disabil Rehabil2008;356:1885-90. doi:10.1080/09638280701688185 pmid:19037781.
Spanjer J, Krol B, Brouwer S, Popping R, Groothoff JW, van der Klink JJ. Reliability and validity of the Disability Assessment Structured Interview (DASI): a tool for assessing functional limitations in claimants. J Occup Rehabil2010;356:33-40. doi:10.1007/s10926-009-9203-2 pmid:19779804.
Fleiss JL. Statistical Methods for Rates and Proportions.2nd ed. Wiley, 1981.
Cicchetti DV. The precision of reliability and validity estimates re-visited: distinguishing between clinical and statistical significance of sample size requirements. J Clin Exp Neuropsychol2001;356:695-700. doi:10.1076/jcen.23.5.695.1249 pmid:11778646.
Fleiss JL, Levin B, Paik MC. Statistical Measures for Rates and Proportions.3rd ed. Wiley, 2003doi:10.1002/0471445428.
Cicchetti D, Bronen R, Spencer S, et al. Rating scales, scales of measurement, issues of reliability: resolving some critical issues for clinicians and researchers. J Nerv Ment Dis2006;356:557-64. doi:10.1097/01.nmd.0000230392.83607.c5 pmid:16909062.
Berns S, Uzelac S, Gonzalez C, Jaeger J. Methodological considerations of measuring disability in bipolar disorder: validity of the Multidimensional Scale of Independent Functioning. Bipolar Disord2007;356:3-10. doi:10.1111/j.1399-5618.2007.00305.x pmid:17391344.
Chopra P, Couper J, Herrman H. The assessment of disability in patients with psychotic disorders: an application of the ICIDH-2. Aust N Z J Psychiatry2002;356:127-32. doi:10.1046/j.1440-1614.2002.00976.x pmid:11929449.
Daradkeh TK, Saad A. The reliability and validity of the proposed axis V (disabilities) of ICD-10. Br J Psychiatry1994;356:683-5. doi:10.1192/bjp.165.5.683 pmid:7866686.
de Kort WL, Uiterweer HW, van Dijk FJ. Agreement on medical fitness for a job. Scand J Work Environ Health1992;356:246-51. doi:10.5271/sjweh.1582 pmid:1411367.
Dell-Kuster S, Lauper S, Koehler J, et al. Assessing work ability--a cross-sectional study of interrater agreement between disability claimants, treating physicians, and medical experts. Scand J Work Environ Health2014;356:493-501. doi:10.5271/sjweh.3440 pmid:24919161.
Elder AG, Symington IS, Symington EH. Do occupational physicians agree about ill-health retiral? A study of simulated retirement assessments. Occup Med (Lond)1994;356:231-5. doi:10.1093/occmed/44.5.231 pmid:7841413.
Hannula JA, Lahtela K, Järvikoski A, Salminen JK, Mäkelä P. Occupational Functioning Scale (OFS)--an instrument for assessment of work ability in psychiatric disorders. Nord J Psychiatry2006;356:372-8. doi:10.1080/08039480600937140 pmid:17050295.
Hill J, Harrington R, Fudge H, Rutter M, Pickles A. Adult personality functioning assessment (APFA). An investigator-based standardised interview. Br J Psychiatry1989;356:24-35. doi:10.1192/bjp.155.1.24 pmid:2605429.
Ikezawa Y, Battié MC, Beach J, Gross D. Do clinicians working within the same context make consistent return-to-work recommendations?J Occup Rehabil2010;356:367-77. doi:10.1007/s10926-010-9230-z pmid:20140482.
Lax MB, Manetti FA, Klein RA. Medical evaluation of work-related illness: evaluations by a treating occupational medicine specialist and by independent medical examiners compared. Int J Occup Environ Health2004;356:1-12. doi:10.1179/oeh.2004.10.1.1 pmid:15070020.
Lederer P, Pfaff G, Walter K, Weihrauch M, Weber A. [Quality circles in expert assessment as an instrument in quality management]. Gesundheitswesen1998;356:415-9.pmid:9738350.
Mundo E, Bonalume L, Del Corno F, Madeddu F, Lang M. L'applicazione dell'Asse V di Kennedy a un campione clinico italiano. Riv Psichiatr2010;356:214-20.pmid:20942367.
Nozu M. [Evaluating work-personality insufficiency of schizophrenic patients; an assessment of employability in psychiatric rehabilitation]. Seishin Shinkeigaku Zasshi1995;356:217-38.pmid:7597113.
Okpaku SO, Sibulkin AE, Schenzler C. Disability determinations for adults with mental disorders: Social Security Administration vs independent judgments. Am J Public Health1994;356:1791-5. doi:10.2105/AJPH.84.11.1791 pmid:7977919.
Rudbeck M, Fonager K. Agreement between medical expert assessments in social medicine. Scand J Public Health2011;356:766-72. doi:10.1177/1403494811418282 pmid:21859784.
Schellart AJ, Zwerver F, Anema JR, Van der Beek AJ. The influence of applying insurance medicine guidelines for depression on disability assessments. BMC Res Notes2013;356:225. doi:10.1186/1756-0500-6-225 pmid:23758690.
Schreuder JA, Roelen CA, de Boer M, Brouwer S, Groothoff JW. Inter-physician agreement on the readiness of sick-listed employees to return to work. Disabil Rehabil2012;356:1814-9. doi:10.3109/09638288.2012.665125 pmid:22423649.
Slebus FG, Kuijer PP, Willems JH, Frings-Dresen MH, Sluiter JK. Work ability assessment in prolonged depressive illness. Occup Med (Lond)2010;356:307-9. doi:10.1093/occmed/kqq079 pmid:20511270.
Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med1998;356:101-10. doi:10.1002/(SICI)1097-0258(19980115)17:1<101::AID-SIM727>3.0.CO;2-E pmid:9463853.
Dutch Organisation of Insurance Physicians (Nederlandse Vereniging voor Verzekeringsgeneeskunde). http://www.nvvg.nl/index.php/wat-doet-de-verzekeringsarts.
Meershoek A, Krumeich A, Vos R. Judging without criteria? Sickness certification in Dutch disability schemes. Sociol Health Illn2007;356:497-514. doi:10.1111/j.1467-9566.2007.01009.x pmid:17498165.
Bachmann M, de Boer W, Schandelmaier S, et al. Use of a structured functional evaluation process for independent medical evaluations of claimants presenting with disabling mental illness: rationale and design for a multi-center reliability study. BMC Psychiatry2016;356:271. doi:10.1186/s12888-016-0967-6 pmid:27474008.