Implicit review by peers (structured or not) is generally considered the community standard for final quality decisions [3, 4]. However, given that practice norms vary widely, can peers agree on the bounds of appropriate care? A recent review [9] of the literature found a paucity of empiric information about the reliability of peer judgments. Little information has been reported on the reliability of peers' judgments about most aspects of care for general medicine inpatients (for example, specific quality problems and appropriateness of resource use). We evaluated implicit review for measuring quality of care and appropriateness of resource use on general medicine wards. We determined how often poor care was identified, the types of reported quality problems, and the level of agreement between different physician reviewers (inter-rater reliability).

Methods

Top Methods Results Discussion Author & Article Info References

Patients

All patients were in one of four general medicine services at a large university teaching hospital. Each ward team consisted of an attending physician, a resident, and two or three interns, who rotated on the service for 1 month. The patient mix was heterogeneous; no single diagnostic-related group contributed more than 5% of patient admissions. A review of a random sample of admissions, occurring from January 1988 to June 1990, was supplemented with oversampling of patient groups of particular interest; deaths, those having readmission within 28 days of discharge, and patients who had an increase in their APACHE-L score (a severity-of-illness measure composed of the laboratory section of the original APACHE score) [10] during hospitalization. We also sampled patients for whom hospital satisfaction data were available (a survey of consecutive patients discharged alive during a 6-month period). Overall, charts from 675 patient admissions were reviewed (5% of charts selected for review could not be obtained for review): 425 patient admissions were selected randomly with replacement and 250 patient admissions were selected randomly to oversample deaths, early readmissions, in-hospital increases in APACHE-L scores, and the patient satisfaction survey. This sampling strategy was designed to allow for detailed analysis of patient groups of special interest for an evaluation of quality screens and of the impact of a clinical management intervention.

Approximately 20% of charts from patient admissions (n = 171) were randomly selected for multiple independent reviews to allow for reliability testing. Although reviewers knew that the completeness and reliability of their reviews were being tested, they did not know which patients had their charts selected for multiple reviews. The 12 reviewers worked varying numbers of hours on the study and are not equally represented in the reliability analyses. However, all but one reviewer reviewed between 25 to 45 of the 171 patient admission charts selected for multiple reviews. The remaining reviewer dropped out of the study early on and reviewed only nine of the reliability charts. Excluding this reviewer from the analyses would not affect our results. The 171 patient admissions selected for inter-rater reliability testing of the reviewers had a mean age of 52 years (SD, 20 years), 20% of patients died in the hospital (as mentioned above, deaths were oversampled), and they were from 39 different diagnostic-related groups (no more than 12 patient admissions were in any one diagnostic-related group). The median number of reviewers per case was three.

Structured Implicit Review Instrument

Using the knowledge gained from structured implicit review for specific medical diseases, we developed a structured, implicit review instrument to evaluate quality of care on a general medicine ward. This instrument included 10 questions about specific aspects of the quality of inpatient care, 5 questions about the appropriateness of resource use, and 3 questions about outpatient care before admission. For patients who died in the hospital, reviewers were asked if their deaths were preventable by better quality of care. The overall evaluation of quality of medical care was rated on a 6-point scale (1 = superior, 2 = excellent, 3 = good, 4 = adequate, 5 = substandard, 6 = poor). Most of the other items were measured on 5-point scales (for example, 1 = definitely adequate, 2 = probably adequate, 3 = unsure, 4 = probably not adequate, 5 = definitely not adequate). For all measures, more appropriate care was represented by a lower scale score. When reviewers were "unsure," they were asked to explain the reason for their uncertainty (borderline medical care was provided, handwriting illegible, inadequate documentation in the chart, or other).

Reviews were done by 12 board-certified internists (7 fellows and 5 faculty) who had reputations for being excellent clinicians at the study hospital. All reviewers had recent or current extensive experience in general medicine inpatient care and had trained in diverse settings; only four had trained at the study hospital. All reviewers had 15 to 20 hours of training before beginning chart review. In a 90-minute initial instruction session, we reviewed the abstracting form and written instructions for completion of the implicit review process. Reviewers were given examples of what we considered substantial and unimportant deviations from standard of care for each item on the review form. We discussed the possible effect of poor outcomes on a reviewer's judgment [12]. Reviewers were instructed not to second-guess reasonable judgments using hindsight and to concentrate on the quality of the process of care. They were asked to evaluate whether they believed the care received was appropriate, regardless of whether the patient had a good or bad outcome. We were only interested in "clinically important quality problems that put the patient at substantial risk of a poor outcome." Each reviewer was required to give a written description of any alleged quality problem and to record the specific risk for the patient. Reviewers were not to consider inefficient care or unnecessary use of hospital resources in their evaluation of overall quality, unless it resulted in undue patient risk (for example, an unnecessary angiogram).

After the initial training session, all reviewers were assigned to review the same 15 to 20 pilot charts. Small group meetings, with three to four reviewers, were then held to discuss their ratings of these patient admission charts for each category of the chart-review instrument. The instructions given in the original meeting were reiterated in detail. Then preliminary assessments of the reviewer's reliability and thoroughness were evaluated, and 4 of the 12 reviewer trainees were assigned more pilot charts for additional training. Once chart review had begun in earnest, reviewers could still direct questions about the review form to the chart review supervisor (RAH). This supervisor examined reviewers' written comments and abstract forms to check for inconsistencies or incomplete data and contacted reviewers when necessary. Every 2 months the supervisor (RAH) contacted all chart reviewers. He asked the reviewers if they were having any problems with the reviews and reinforced training (by reviewing the definitions of the review instrument subcategories and reiterating the training instructions). Copies of the review instrument and the instructions to reviewers are available on request.

Statistical Analysis

Weighted statistics were calculated to quantify the inter-rater reliabilities of reviewers' implicit judgments. Kappas were calculated on 171 charts that had multiple reviews producing 368 separate comparisons. In evaluating inter-rater reliabilities, it is important to distinguish between reliabilities necessary for assessment of each patient and those needed for assessment of an aggregate of patients. For most aggregate comparisons using means or proportions (such as comparing mean scores on two hospital wards), moderate reliabilities of 0.5 or 0.6 are usually acceptable [11]. However, reliabilities must often be high to make a confident assessment of each patient ( 0.80) [11].

However, the statistic is only a summary of the amount of agreement and does not directly relay the predictiveness of a judgment. Therefore, we calculated the sensitivity and specificity of a single review using the mean review of the other reviewers as the standard (using a method described by Rubin and colleagues [8]). This analysis estimates the likelihood that a single review would correctly classify the patient compared with multiple other reviews (cutoff point for substandard care was a mean score of 4.5). These analyses were conducted on 65 patients who had 3 or more reviewers (median = 4 reviewers). In addition, the actual patterns of agreement are shown in the Appendix for those items with kappas of 0.4 or greater. To determine whether the tendency for reviewers to rate a patient's care more harshly if a patient had a poor outcome artificially elevated our agreements [12], we repeated analyses excluding deaths. This did not substantially affect the statistics.

Sampling weights corrected the estimates for oversampling of specific patient subgroups; thus, the results reflect estimates for the overall general medicine ward populations. Analysis of variance evaluated the amount of variance in quality assessments that was attributable to individual reviewers (a reviewer-specific effect). Finally, we were interested in determining which focused quality assessments (such as readiness for discharge and timeliness of diagnostic evaluation) were most strongly associated with the reviewers' overall assessments of quality. We constructed a logistic regression model with the overall quality rating as the dependent variable (0 = adequate care or better and 1 = substandard or poor care) and the serial quality ratings about specific aspects of care as the independent variables.

Results

Top Methods Results Discussion Author & Article Info References

Measurements of Overall Quality and Specific Quality Problems

The mean quality rating of the process of care was 3.1 (1 = superior, 3 = good, and 6 = poor) (Table 1). Overall, 11% of patient admissions were rated as having received substandard or poor care (ratings of 5 or 6, respectively), and 9% of patient deaths were rated as probably or definitely preventable by better quality of care. The inter-rater reliabilities were moderate for these two summary measures ( = 0.5). Adding a second reviewer (for the 40 patients who had four or more reviewers) did not substantially improve the inter-rater reliability ( < 0.6). Another way of representing the inter-rater reliability is using the predictive value of a single review compared with the mean rating of other independent reviewers. The sensitivity of a single review for poor quality was 60%, and the specificity was 87%. Therefore, if the overall rate of substandard care (the prior probability) was 11%, the positive predictive value of a poor quality assessment (the probability that the review would indicate inadequate care if a large number of implicit reviews were done) would be only 36%. However, if targeting patients with a high probability of substandard care (that is, by using validated screening criteria) could increase the prior probability to 50%, then a single review indicating substandard care would increase the likelihood to 82%; but if the review indicated adequate care or better, the likelihood of a review indicating substandard care would decrease to 32%.

Reviewers did not agree on whether errors in following doctor's orders, dispensing medications, or notifying physicians had occurred ( = 0.1). Although physician reviewers frequently believed that patients were not ready for discharge (16% of patients), their agreement on this aspect of care was also poor (see Table 1). Poor agreement existed between physicians for their implicit judgments of the timeliness and appropriateness of the diagnostic evaluation and therapeutic interventions [ = 0.2], and attempts at combining these items into scales did not substantially improve reliability. Inappropriate care for patients' initial problems at presentation was reported more commonly than problems with the timeliness of such care. Overall, reviewers rated 21% of patient admissions as receiving inadequate or inappropriate responses from their medical team to new signs, symptoms, or test results ( = 0.3). In 19% of patient admissions, reviewers believed that documentation in the medical record was substandard.

Measurements of Resource Use Appropriateness and Need for Admission

Table 2 shows reviewers' implicit judgments about resource use. Reviewers rated 9% of patient admissions as probably or definitely inappropriate, but reliability was marginal ( = 0.4). Although chart reviewers reported substantial amounts of overuse and underuse of tests and pharmaceutical agents, inter-rater agreement was poor ( = 0.2 or less). For 22% of patient admissions, reviewers believed that patient length of stay was excessive; for 5% of patient admissions, reviewers believed that length of stay was too short, but agreement was poor ( = 0.2).

Reviewers were also asked whether better outpatient care could have prevented the patient's admission (Table 3). For 17% of patient admissions, reviewers believed that they had inadequate records to evaluate this aspect of preadmission care. Of the remaining patient admissions, reviewers believed that 22% had one or more factors that could have prevented the need for hospital admission: better outpatient care by provider (12%); patient seeking care sooner (11%); and better patient compliance (10%). Kappas for these measures were 0.3, 0.4, and 0.5, respectively.

Reviewer-specific Bias, Reviewer Uncertainty, and Adequacy of Documentation

Poor agreement between reviewers could have been due to biased estimates that were secondary to a reviewer-specific effect (some reviewers were systematically more harsh or lenient than others) or could have been independent of the specific reviewers. We evaluated this issue using analysis of variance for each implicit measure. Only 2% of the variance in the measurement of overall quality was attributable to a reviewer-specific effect (see Table 1). Many measures that had poor inter-rater reliability had a much higher reviewer-specific effect, but this was not universal (see Tables 1, 2, and 3). Measures of resource use, appropriateness of discharge, and errors in following doctor's orders had the highest reviewer-specific effects. Serially removing each reviewer from the analyses showed that the degree of reviewer-specific effects was not influenced by a single outlier reviewer.

A principal limitation of chart review is that the reviewer only has the information available in the medical record. We were, therefore, interested in the frequency of, and reasons for, reviewers reporting that they were unable to decide whether the care was probably or definitely good or bad (a response of "unsure"). For the judgments listed in Table 1, reviewers marked "unsure" for 1% to 6% of patient admissions (detailed results available from authors). In addition, when the reviewers were uncertain, they usually reported that their uncertainty was due to clinically borderline care, not to inadequate documentation.

Reasons for Ratings of Substandard Care

Thus far we have dealt primarily with agreement between different reviewers. We now evaluate the aspects of care that most strongly influence an individual reviewer's decision to rate overall care as substandard. We did logistic regression analysis using the seven focused quality measures as independent variables (Table 4). In bivariate analyses, all measures were significantly associated (P < 0.001) with a reviewer's overall assessment of quality, except for the measure of the appropriateness of following physicians' orders. In the multivariate model containing all seven focused quality ratings, the adequacy and appropriateness of physician response to new signs, symptoms, or test results was the measure most strongly associated with a reviewer's overall assessment of quality (odds ratio = 5.3 [95% CI, 3.2 to 8.7]). The timeliness and appropriateness of the diagnostic evaluation, as well as the appropriateness of treatment and physicians' orders, were also significantly associated with ratings of overall quality.

Discussion

Top Methods Results Discussion Author & Article Info References

Reliability levels for overall quality and preventability of deaths reported in this study are similar to those reported by others [3, 7-9, 13] for evaluation of the care of specific conditions (for example, stroke and myocardial infarction) and care of general medicine inpatients. The level of reliability for these measures ( = 0.5) is usually considered adequate for aggregate comparisons (for example, comparing mean quality ratings at two hospitals); however, agreements for these measures were not adequate for confident judgments about each patient using one or two reviewers [11]. As others have noted [8], using about five independent reviewers could achieve more than 90% accuracy in classification if reliability levels are similar to our results for overall quality. This is too expensive and burdensome for routine review, but certain patients may sometimes justify such an expense. Furthermore, our results were obtained under seemingly ideal conditions: extensive supervision and reviewer reinforcement of training; development and implementation in a teaching hospital; and homogeneity of reviewers (all reviewers were younger than 40 years). The screening of reviewers, training, structured process, and supervision were more extensive than most administrative peer-review processes.

For most specific aspects of care (for example, appropriateness of resource use and readiness for discharge), physicians could not agree. For some measures, a substantial amount of the variation in judgment was due to specific physicians who were more harsh or more lenient than others in their reviews [14]; however, other measures with poor inter-rater reliability had only a small reviewer-specific effect. Although our reviewers believed that medical documentation was often of poor quality, reviewers rarely believed that deficiencies in medical documentation prevented them from making a judgment on the quality of care.

Poor agreement between physicians is not unique to implicit review. The inter-observer reliability for clinician's detection of objective clinical signs ranges from kappas of 0.7 for presence of spider angiomata to 0.1 or less for the presence of whispered pectoriloquy and tactile fremitus [15, 16]. Still, if future studies verify our results, poor reliability would make implicit review for resource-use appropriateness and many specific quality-of-care judgments (for example, readiness for discharge) untenable. Rubenstein and colleagues [3] found better inter-rater reliability levels for some focused implicit quality measures when evaluating care for specific medical conditions (for example, acute myocardial infarction and cerebrovascular accident). Perhaps the reliability and validity of judgments about general medicine inpatient care could be improved by using specialist peer reviewers. If implicit review by a generalist reviewer suggests a quality problem, a second opinion could be obtained by a specialist reviewer. However, the more peer reviews needed to achieve a reliable judgment, the higher the cost of the review. Our study investigated only one implicit review instrument. Reordering, rewording, or restructuring the subcategories of our review instrument could produce higher degrees of reliability. Also, our results were from a single institution and represent the agreements of 12 reviewers. As others have previously suggested [17], the selection and training of the reviewers may be the critical step in the reliability and validity of an implicit review process [17].

We cannot determine the reasons for disagreement in this study. Reviews were independent. Therefore, we only know why care was judged substandard, not why it was judged adequate. Our informal impression, based on group debriefing sessions during reviewer training, is that inter-rater discordance may often be due to factors other than a true difference of clinical opinion. Certainly legitimate differences in opinion occurred during training sessions, such as whether the nature of a patient's chest pain on day 10 of her hospitalization required a workup for pulmonary embolus or a disagreement about whether a liver biopsy was unnecessary. Such disagreements were only occasionally resolved by discussion. However, some disagreements were easily resolved because they were due solely to oversight of a key piece of information. During training, one of our reviewers had not noticed that an elderly woman with severe baseline renal insufficiency had been given more than 4 L of normal saline intravenously during a 24-hour period to treat her dehydration and had not received close monitoring. The reviewer readily agreed that this was poor care. For another patient, a reviewer believed that an inappropriate delay occurred in initiation of antibiotics in an elderly febrile patient but recanted when another reviewer noted that antibiotics were given in the emergency room (this had not been recorded in the appropriate area of the chart). Future research should evaluate this topic more formally. When a quality problem is cited by a reviewer, the effect of calling subsequent reviewers' attention to the specific aspect of care being questioned should be evaluated. Of course, directing the reviewer to a specific, alleged quality problem could bias the review but not doing so could compromise the validity of the review if the second reviewer gives a good rating merely due to oversight. The former approach also better simulates actual quality-assurance efforts.

Whatever refinements are made in implicit review, it is likely to remain an expensive and burdensome process for definitively identifying specific quality problems in single patients. Perhaps too much energy and money are spent on identifying situations in which single patients receive poor treatment. How often does it help a payer or hospital to have a high level of confidence that substandard care occurred for a single patient? It does not show that the physician is incompetent, nor does it necessarily tell whether the quality problem is important enough to devote resources toward preventing it in the future. The goal of quality assurance should be to improve the quality of future care by identifying either substandard providers (and intervening by re-education, rehabilitation, or sanction) or patterns of poor care across multiple providers that could be dealt with by education or administrative changes. In particular, we should concentrate on substandard care that is related to poor outcomes (which is nearly impossible to determine if we restrict ourselves to looking at single patients in isolation from one another). Despite its limitations, implicit review may be useful in evaluations of the overall quality of care supplied by a provider (by evaluating a sample of a physician's or hospital's patients and using implicit judgments as an aggregate measure) and may aid investigations of patterns of questionable or poor care associated with adverse outcomes [13, 18].

We believe that a more cooperative and constructive approach to peer review should occur [19, 20]. Our results should not be interpreted as evidence that "bureaucrats" should leave clinicians alone to practice medicine as they see fit. Instead, this study and other recent work [7, 8, 13, 18, 21, 22] highlight the fact that quality problems and unnecessary care are not rare. Peer review is not the enemy. Indeed, the profession must take an active role in aiding and supporting efforts to improve the process and focus of quality and utilization review. Despite the substantial resources spent on peer review (more than $300 million annually by the federal government alone), there is a paucity of well-designed studies evaluating methods of peer review and their outcomes. We believe [4, 8, 9, 17-19] more attention should be devoted to this critical area.

Presented, in part, at the Society of General Internal Medicine, 15th Annual Meeting, Washington, DC, 1 May 1992.

Appendix. Two-Way Cross-tabulation Tables for Inter-rater Agreements

The tables below show inter-rater agreements for those patients whose charts were analyzed by multiple reviewers. Cross-tabulation tables are shown for only those measures with moderate agreement ( 0.4). The weighted statistics reported in Tables 1, 2, and 3 were generated from these tables. The tables show all possible two-way comparisons between reviewers.