The objective of this study was to explore the ability to evaluate ICU performance using risk-adjusted in-hospital mortality rates and length of ICU stay. In this report, we focus on the amount of variation that can be accounted for after adjusting for patient characteristics present at admission. We describe the nature and relative importance of these factors and the extent of the remaining variation in outcome performance after such adjustment.

Methods

Top Methods Results Discussion Author & Article Info References

Hospital and Intensive Care Unit Selection

We used a stratified random process based on geographic region, size, and teaching status (defined by the presence of resident housestaff or of one or more accredited graduate medical training programs) to select 26 hospitals from 1691 nonfederal U.S. hospitals with more than 200 beds. Twenty-three of the 26 hospitals agreed to participate. The reasons for nonparticipation of the three hospitals were the sale of the hospital, a severe nursing shortage (making data collection assistance unlikely), and a poor fiscal condition (making closure imminent). We chose three alternate hospitals from the same strata using an identical process. Fourteen other hospitals, primarily large tertiary care or university teaching hospitals with an interest in the project, also volunteered to participate in the study, giving a total of 40 hospitals. In hospitals with more than one ICU, data collection took place in the unit with the highest annual admission rate. Data were also collected in two ICUs at two volunteer institutions, giving a total of 42 ICUs for analysis. We excluded burn, pediatric, neonatal, and coronary care units from consideration. Thus, data collection took place in adult general medical, general surgical, and combined medical-surgical units.

Patient Selection and Data Collection

Data collection began in May 1988 and was completed in February 1990; the study period at each ICU averaged 9.7 months (range, 3 to 17 months). In most ICUs, data were collected concurrently for consecutive ICU admissions. A systematic scheme (for example, every second or third patient) was used in 20% of the units when patient volume precluded data collection on consecutive admissions. Patients were excluded from the study if they were admitted for chest pain, rule-out myocardial infarction, coronary artery bypass surgery, or burn injury; were younger than 16 years; or had a length of ICU stay of less than 4 hours.

Information collected for each patient included age, an extensive listing of coexisting illnesses, location before ICU admission (emergency, recovery, hospital, or operating room; ICU readmission; or transfer from another ICU or hospital), and surgery status (elective or emergency, which was defined as surgery for an immediately life-threatening condition). We also recorded the primary reason for ICU admission using 78 mutually exclusive disease categories [12]. During the first ICU day, each patient's clinical record was reviewed for APACHE III and Therapeutic Intervention Scoring System scoring [12, 13]. The APACHE III score consists of an acute physiology score obtained by applying weights to 17 potential physiologic variables, a weight applied to age, and additional weight to one of seven comorbid conditions that influence the risk for short-term death by reducing immune response. A higher APACHE III score (0 to 299) is associated with a higher risk for in-hospital death. The Therapeutic Intervention Scoring System is also a weighted (1 to 4) scoring system derived from 80 interventions and specific nursing tasks representing the intensity of care provided.

During the subsequent 6 ICU days, we recorded all changes in the APACHE III acute physiology score and, using the Therapeutic Intervention Scoring System, in the type and amount of monitoring and treatment received. We also recorded length of stay in the ICU and in the hospital and followed all patients for survival at ICU and hospital discharge and at 30 days after discharge for all Medicare patients and for a 15% random sample of all other patients. Patient data were entered into on-site microcomputers by trained data collectors using specially designed software, and data underwent continuous quality monitoring and review. A formal interobserver reliability study was conducted at 11 of the hospitals [14]. Further details on data collection procedures are available elsewhere [12, 15].

Equations for Predicting In-Hospital Death and Length of ICU Stay

For each patient, we estimated the probability of in-hospital death using a multiple logistic regression equation. The variables used in this analysis were preselected based on previous research [15]. The primary determinants of short-term outcome were defined as acute physiologic abnormalities within 24 hours of ICU admission (APACHE III acute physiologic score); the patient's physiologic reserve as measured by age and the presence of specific comorbid conditions (as represented in the APACHE III score); the underlying disease prompting ICU admission; the location and duration of treatment immediately before ICU admission; and one institutional characteristic: the nature of hospital discharge practices as measured by the excess mean duration of hospital stay for survivors. This variable was determined for each unit based on a statistical analysis of length of hospital stay for all hospital survivors compared with average stay for all ICUs based on disease and APACHE III score (see footnote in Appendix Table 1). This variable was used to account for variations in triage pressure or practice style, both of which affect hospital discharge patterns: Hospitals that discharge patients later are likely to report more deaths because of the longer time during which their patients could die in the hospital [16].

We developed a separate multiple regression equation to estimate length of ICU stay based on the same patient and institutional characteristics as described above, with the following exceptions. The excess mean adjusted length of hospital stay and the patient's length of hospital stay before ICU admission were deleted, and traditional hospital descriptors, such as geographic region, bed size, and teaching status, were added. This analysis excluded patients discharged to another ICU for which there was incomplete data on total length of ICU stay but included the fact that a patient was readmitted to the unit. In cases in which length of ICU stay exceeded 40 days, such stays were rounded down to 40 days and then included in the analysis. To allow for nonlinearities in the relation between continuous variables and length of ICU stay, the method of restricted cubic splines was used [18]. This technique permits the weight attributable to a variable to vary continuously throughout its possible range. The model for length of ICU stay was also cross-validated using the same grouped jackknife approach as has been described for mortality.

To measure how much of the variation in mortality and length of stay were accounted for by the equations, we used the area under a receiver operating characteristic curve [19] for the dichotomous outcome, mortality, and the R² for the continuous variable, length of stay. Except for the comparisons of the full equation with the cross-validated predictions, all these results report associations with the cross-validated predicted risks.

Risk-adjusted (Standardized) Ratios for In-Hospital Mortality and Length of ICU Stay

To calculate a risk-adjusted mortality rate for each ICU, we added individual patient predictions using the cross-validated equations and then calculated a standardized mortality ratio by dividing actual by mean predicted group death rate at hospital discharge. The units were then ranked by relative performance according to their standardized mortality ratio. We used a chi-square test to determine the significance of differences between actual and predicted survival rates for each unit, and a P value 0.05 at the unit level was considered to be significant. To determine the amount of variation across ICUs that was accounted for by predictions, we estimated a univariate least-squares regression equation across all 42 ICUs, using the observed death rate as the dependent variable and the mean predicted risk as the independent variable for each hospital.

We examined survival 30 days after hospital discharge; the standardized mortality ratio was recalculated for each hospital to reflect all deaths observed within 30 days of hospital discharge. These revised ratios were anticipated to have an average greater than 1.0 because the cumulative mortality rate after hospital discharge would be greater than the predicted in-hospital mortality rate. The relative performance across the units was not expected to change, unless an ICU had substantially more deaths immediately after hospital discharge.

To determine each unit's performance regarding length of ICU stay, we added individual predictions for each unit and then calculated a ratio by dividing the mean actual by the mean predicted length of ICU stay. We used a paired t-test to determine the significance of differences between actual and predicted lengths of stay for each ICU, and a P value of less than 0.05 at the unit level was considered to be significant. To measure the amount of variation accounted for by this procedure across units, we estimated a univariate least-squares regression equation across the 42 units. The mean observed length of ICU stay was the dependent variable, and the mean predicted length of stay was the independent variable.

Results

Top Methods Results Discussion Author & Article Info References

Characteristics of Hospitals and Intensive Care Units

Of the 40 hospitals studied, 35 (87.5%) were nonprofit, 3 (7.5%) were for profit, and 2 (5%) were operated by state or local governments. Geographically, 17.5% were in the Northeast, 32.5% in the South, 30% in the Midwest, and 20% in the West. The mean number of hospital beds was 474 (range, 200 to 1315 beds), and the mean hospital occupancy rate was 71.1% (range, 43.4% to 87.6%). Twenty-five hospitals (63%) met our definition of teaching hospital, and 53% were affiliated with a medical school.

The mean number of adult ICU beds at each hospital was 22 (range, 6 to 76 beds). Twenty-five hospitals had one or more intermediate care units. Of the 42 ICUs studied, 4 were medical, 8 were surgical, and 30 were combined medical-surgical. The average number of ICU beds in study units was 13, and their average occupancy rate was 77% (range, 61% to 100%).

Characteristics of Patients

There were a total of 17 440 admissions. Eight hundred eighteen (5%) were readmissions, leaving 16 622 patients for the mortality analysis. Three hundred thirty-five admitted patients were discharged to another ICU, and the total duration of ICU stay was not known, leaving 17 105 admissions for the length-of-stay analysis. The average number of admissions at each ICU was 415 (range, 299 to 449 admissions). The mean age of patients was 59 years; 48% of patients were 65 years or older. The mean total number of different diagnoses given as the primary reasons for admission to each ICU was 60 (range, 44 to 71 diagnoses). These characteristics are summarized in Table 1.

For the 9479 patients who did not have surgery, the most frequent reasons for ICU admission were congestive heart failure (8.8%), upper gastrointestinal bleeding due to ulcer or laceration (6.4%), drug overdose (6.8%), and bacterial pneumonia (4.4%). Among 7143 patients who had surgery, 7.3% had peripheral artery bypass grafts, 7.3% had elective abdominal aneurysm repair, 6.9% had laparotomy for gastrointestinal neoplasm, 5.9% had craniotomy for neoplasm, and 5.8% had carotid endarterectomy.

The unadjusted in-hospital mortality rate for the 16 622 patients was 16.5% (range, 6.4% to 40%). The unadjusted mean length of ICU stay was 4.7 days (range, 3.3 to 7.3 days). The mean first-day APACHE III score was 49.2 (range, 39.6 to 76.1).

Equations for Mortality and Length of Stay

The major patient variables that influenced in-hospital mortality and length of ICU stay were the acute physiologic score, chronologic age, seven comorbid conditions as represented in the APACHE III score [12], the primary reason for ICU admission, surgery status (emergency or elective), the patient's location before ICU admission, and length of hospital stay before ICU admission. Appendix Table 1 lists all factors and their univariate and multivariate relations with outcome. The acute physiology score component of the APACHE III score, age, previous length of hospital stay, and excess mean length of hospital stay were used as continuous variables in both regression equations, although they are summarized as categorical variables in Appendix Table 1. The remaining variables were used as categorical variables. The predictive equation for in-hospital mortality has a cross-validated receiver operating characteristic area of 0.90 and an R² of 0.39 across individual patients.

The observed risk for in-hospital death in groups of approximately 500 patients is shown in Appendix Table 2). These results confirm that predicted mortality rates from the cross-validated models are close to the observed rates throughout the range of predicted risks. All equations and appropriate documentation are available from the authors on request (see Disclosure at the end of text).

The same major variables that were significant in the mortality equation were also important in predicting length of ICU stay. In addition, geographic region, number of beds, and teaching status were also influential. The relative importance and magnitude of these institutional variables in the analysis of length of ICU stay was considerably smaller, however, in the multivariate analysis than in the univariate analysis (see Appendix Table 1). Except for the East Coast, the use of these hospital-level descriptors in the multivariate analysis did not substantially influence the determination of expected length of ICU stay. The equation for length of ICU stay using the cross-validated prediction yields an R² of 0.15 across patients. If length of ICU stay is truncated at 15 days (three times the mean length of stay) rather than at 40 days, the R² is 0.23.

The relative importance of each of the factors in the equations for mortality and length of ICU stay is summarized in Figure 1. Acute physiologic disturbances and disease classification are overwhelmingly the two most important factors in both equations. Predicted and observed risks for in-hospital mortality and length of ICU stay at various levels of initial APACHE III score are shown in Figure 2, which reports the predicted risks from cross-validated models. A close relation throughout the range of risks is shown for both mortality and length of ICU stay.

View larger version (35K): [in this window] [in a new window]   Figure 1. Top. Relative contribution of each factor to in-hospital mortality. The "Other" category includes length of stay before admission to the intensive care unit (ICU), 1.6%; mean duration of hospital stay for survivors, 1.4%; location before ICU admission, 0.1%; and emergency surgery, 0.01%. Bottom. Relative contribution of each factor to length of ICU stay. The "Other" category includes location before ICU admission, 6.7%; region, 3.2%; ICU readmission, 1.1%; bed size of the hospital, 0.8%; emergency surgery, 0.7%; and teaching status, 0.2%. The relative contributions were calculated as the percentage of chi-square uniquely associated with each variable. Asterisks indicate percentages as represented in the APACHE III score. The "Disease" category included 78 mutually exclusive indications for ICU admission.

View larger version (37K): [in this window] [in a new window]   Figure 2. Top. Distribution of patients and the association between first-day APACHE III score and in-hospital mortality rate. The mortality analysis included 16 662 intensive care unit (ICU) patients. Bottom. Distribution of patients and the association between first-day APACHE III score and length of ICU stay. The length of stay analysis included 17 105 ICU admissions.

The standardized mortality ratio for the 42 units ranged from 0.67 to 1.25. Figure 3 (top panel) presents the relative performance of the 42 ICUs by contrasting actual and predicted death rates at hospital discharge. All units are relatively close to the 45-degree line, indicating close agreement between observed and predicted outcomes across the entire range of in-hospital mortality rates (R² = 0.90). Risk-adjusted mortality performance was significantly better (P 0.05) for five ICUs and significantly worse (P 0.05) for five ICUs. These represent a statistically significantly larger number of outlying hospitals than would be expected due to chance alone (2 hospitals) when testing 42 different hospitals (chi-square = 6.2, P = 0.01). No significant difference in risk-adjusted mortality was observed between the 16 volunteer and 26 randomly selected ICUs, and no significant change was seen in relative performance ranking using mortality rates 30 days after hospital discharge.

View larger version (21K): [in this window] [in a new window]   Figure 3. Top. Relative performance of 42 intensive care units according to the actual and predicted death rate at hospital discharge. A linear regression of observed death rate on mean predicted death rate across the 42 units yielded an intercept of –0.006 and a regression coefficient of 1.036 (SE, 0.055). This indicates that the equation is well calibrated across all levels of risk. Bottom. Relative performance of 42 intensive care units according to the actual and predicted length of ICU stay. A linear regression of observed mean length of ICU stay on mean predicted length of stay across these 42 units yielded an intercept of 0.062 and a regression coefficient of 0.989 (SE, 0.083), indicating that the equation is well calibrated across all levels of risk. Units with statistically significant (P < 0.05) variations are denoted by the box or star symbol.

Discussion

Top Methods Results Discussion Author & Article Info References

Our study suggests that most of the substantial variation in observed death rates across ICUs can be accounted for by routinely available clinical data. The 42 ICUs in our study had observed death rates that varied almost sevenfold, from 6.4% to 40%. Most (R² = 0.90) of this variation across institutions was due to measurable differences in patient diagnosis, physiologic and demographic characteristics at admission, and hospital discharge practices.

Using these adjustments, most of the ICUs in our study had few unexplained deaths, an average variation of 1 to 2 deaths per 100 patients treated (Figure 3, top). When viewed from a national perspective, these results provide important reassurance that the number of ICUs with excessive death rates among randomly chosen hospitals with 200 or more beds is not substantial. At the same time, this degree of variation in mortality suggests that the best-performing ICU in our study (standardized mortality rate, 0.67) would, on the basis of its projected annual volume of ICU admissions, have 10 to 12 fewer deaths per 100 patients treated than the worst-performing unit. Although this represents an extreme estimate, this degree of unexplained variation in mortality performance is clinically important and should be a high priority for further investigation.

This variation may be due to yet unmeasured patient factors, specific variations in quality of care, or both. Attempts to explain mortality differences in hospital Medicare discharges [20-22, 24] identified fewer hospitals with significant variations than the 24% we identified (Figure 3, top). We believe this results from our more powerful ability to adjust for patient risk factors, and because the study ICUs had widely variable admitting practices [25]. Our results are consistent with studies such as those on coronary artery bypass graft or trauma using prospectively collected clinical data [26-28]. The percentage of hospitals performing outside statistical limits was 25% in a study of 28 hospitals performing coronary artery bypass graft procedures in New York [26].

As Figure 1 shows, severity of illness, as measured by physiologic abnormalities, is the single most important determinant of variations in mortality for patients admitted to ICUs [29]. We used a physiologic scoring approach with empiric weighting obtained from searching a randomly split half of this multidiagnostic database to determine the contribution of each physiologic variable [12]. In future studies, determining relative weights for physiologic measures in different ways [18] or altering physiologic weighting for specific diseases [5] may improve our ability to predict outcome of these severely ill patients even more accurately.

We also confirmed the result previously reported by Escarce and Kelley [30] that patients admitted to ICUs from hospital wards have a higher mortality rate, after controlling for severity and disease, than do ICU patients admitted directly from emergency or operating rooms. This is an example of lead-time bias (that is, bias related to when in the course of illness the patient receives intensive care). We expanded on this measurement by incorporating a new patient characteristic, the exact length of hospital stay before ICU admission.

The only hospital characteristic used in the mortality analysis was mean excess length of stay for survivors. This variable was used because hospitals with a longer average length of stay for similar patients are likely to report more deaths because of the longer time during which their patients could die in hospital [16]. Movement to a specified time after admission or after discharge (for example, 30 days), might eliminate the need for this adjustment [21], but most hospitals do not routinely follow patients for post-hospital survival times.

There are several possible limitations to our study. First, some of the variations in risk-adjusted mortality rate may be accounted for by inadequately measured individual patient characteristics. The incremental improvement in explanatory power by using APACHE III rather than APACHE II shows how such improvement is technically possible [12] but also means that variation in unmeasured patient characteristics is smaller than in previous comparisons [7, 20-25]. Second, although it is possible that interinstitutional variation in measurement of predictive variables might have confounded the analysis, formal reliability testing of the measures used failed to find any systematic measurement bias across the hospitals [14]. To minimize the possibility that differences in measurement of laboratory tests could bias the results against hospitals that test less frequently, we used the worst laboratory test result during the first day in the ICU.

We acknowledge, however, that selection for ICU care in a 200-bed rural nonteaching hospital differs from that in a 900-bed urban teaching hospital. Some of these selection-bias differences across the 42 units may have influenced their standardized mortality ratios and relative ranking. There may also be subtle distinctions in physician referral (for example, in transferring patients with poor prognoses to the ICU) [31, 32] or variations in patient preferences for hospitals according to perceived quality [33]. Analysis of the potential confounding effect of hospital size, region, and teaching status yielded no significant changes in overall performance ranking within the entire sample; however, future studies might limit comparisons to institutions with similar referral or practice characteristics. For example, we are currently comparing the performance of hospitals that have well-defined teaching functions with the performance of hospitals that have no teaching functions.

Finally, ICUs with a large number of low-risk admissions are likely to have fewer deaths and perhaps a lower standardized mortality ratio than do units admitting high-risk patients. Although we detected no statistically significant associations in our sample Figure 3, larger samples of institutions with low-risk patients are required before definitive conclusions can be drawn.

Length of ICU and hospital stay is commonly used as a measure of cost, even though intensity of care may lead to discrepancies for individuals or groups of patients. Several detailed studies of interpatient and interhospital variation in length of hospital stay among Medicare beneficiaries have been done [34-37]. Several studies have focused on length of ICU stay for a specific diagnostic group or particular therapy [38-42].

In our analysis of length of stay, acute physiologic derangements were again the most important predictor (Figure 1, bottom panel). Disease is relatively more important for length of stay than for mortality. Hospital bed size, regional location, and teaching status were considered as potential reasons for discrepancies between predicted and observed mean length of stay. However, the relatively small variation attributable to these institutional characteristics in the multivariate analysis is reassuring (see Appendix Table 1) because such adjustment is likely to be controversial [4].

Although the accuracy of the equation for length of ICU stay has substantially lower explanatory power than the mortality equation (R² = 0.15 compared with 0.39 for hospital mortality), it is adequate for assessing unit efficiency. Potential reasons for a smaller explanatory power include greater random variations in whether a patient is discharged on a specific day [42]; measurement of length of stay in discrete days rather than hours; and a complex relation between higher levels of severity and length of stay. As can be seen in Figure 2 (bottom panel), patients with the high APACHE III scores (>120) have short stays. This is because many of these patients die quickly. For high-severity survivors, however, stays can be quite long. Future analyses of risk-adjusted length of stay need to focus on the possible interactions between specific diseases and severity. The relatively strong performance of the equation for length of ICU stay in predicting differences across units (R² = 0.78), however, suggests that such analyses can still provide important comparative data to guide potential improvements in ICU efficiency.

The fundamental limitation of our study was the derivation of the mortality ratios from relatively small numbers of patients studied during a relatively brief time period at a limited number of institutions [5, 43]. The need to limit our study to 42 ICUs and to 400 cases per ICU reduced our ability to evaluate fully the effect of potentially important variables (for example, ICU specialization or ICU triage pressures) [42]. This limitation can be overcome by routinely collecting standardized clinical data and monitoring risk-adjusted outcomes over time in all ICUs [44]. This could also help focus quality review of outcomes within diagnosis or by service and facilitate the detection of the reasons for variations in mortality and length of stay [45, 46]. The substantial amount of patient variation accounted for by these methods indicates that such efforts should be undertaken.