We agree with Drs. Kutner and Rao that validation of a model done by using bootstrapping or split-sample methods is essential for obtaining an unbiased estimate of its goodness of fit. Because our database was large, and because the primary focus of our paper was the comparison of models of several different types, we did not include validation results in our paper. While evaluating hospital performance for large coalitions, we have found that c-statistics remain remarkably stable when models developed on 1 year of data are applied to a subsequent year. The data on acute myocardial infarction presented in our paper illustrate this: From the first year of data to the second, the c-statistic changes from 0.92 to 0.90 for the unrestricted administrative model, changes from 0.76 to 0.75 for the restricted administrative model, changes from 0.86 to 0.87 for the laboratory model, and changes from 0.88 to 0.87 for the clinical model.

As we discussed in our paper, our data were abstracted from medical records by carefully trained abstracters. Most missing data resulted from physicians not obtaining or not documenting specific clinical values in patients for whom this information was not considered medically important. Therefore, our imputation of "normal" for dichotomous variables probably was correct in most cases. Because the data are not missing randomly, analyzing only cases in which all values are present would systematically eliminate patients with less severe disease. We are continuing to examine the possible effect of various imputation schemes on our results.

Weighting the summary c-statistics by prevalence of the disease is certainly important when deriving a composite measure for a system that includes many disease categories. However, we were comparing the performance of different models for a few selected conditions. Therefore, all models contributed equally important information, regardless of the prevalence of the underlying disease in the population.

Our restricted administrative model was based only on diagnoses that most physicians would agree are almost never complications of acute hospital care (for example, cancer, chronic obstructive airways disease, diabetes, and old myocardial infarction). Diagnoses that are possible or likely consequences of acute hospital care (such as shock, acute renal failure, and diabetic ketoacidosis) were eligible for inclusion only in the unrestricted administrative models.