Databases in the Next Millennium

Data and Quality: What Are the Questions?

The principal question that concerns us here is whether databases will have a role in determining the quality of health care-and their costs. There are, of course, several points of view on quality, particularly on the question of whose perspective is most important in deciding what quality is (my own view is that the patient's perspective matters most) [2]. But the definition of quality and its importance in the future of the health care system, like the nature of the system itself, go well beyond questions of perspective; the very concept of quality is inextricably intertwined with our most basic cultural traditions, our social values, our political philosophies, our economy, and even our spiritual life. This makes the health care system incredibly complex and hard to understand and, correspondingly, hard to change.

Lying just behind this principal question are two deeper questions. The first is “Can data serve as the agent for meaningful improvement?” The authors who have contributed to this issue clearly feel that the answer is “yes.” That is, they assume that if you can only get your hands on the right data (accurate, risk-adjusted data) and enough of it, people and systems will change, evolve, improve. The second question is the converse, namely, “Is meaningful improvement possible without data?” The authors give us less to work with here, but many who take medical quality improvement seriously have little doubt that the answer is “no.” For example, Kathleen Goonan, in The Juran Prescription: Clinical Quality Management, says it this way: “You can't manage what you can't measure” [3]. Unfortunately, each of these questions forces us to think narrowly and dichotomously, oversimplifying a complex situation. From what we know about the way in which innovations are adopted [4], either question is probably the wrong one, much as the question of whether women are more important than men (or vice versa) in creating children is the wrong one. For our purposes, the better question appears to be, “What elements are necessary and sufficient for improving the quality of medical care?”

Many examples of change in the explosively developing “quality improvement” era have involved the use of data [5]. Even these examples, however, do not prove that the data themselves were the principal lever for change; other factors may have been at least as important, perhaps more so. In this essay, I suggest, first, that although data may often be necessary, by themselves they are usually not sufficient to bring about change. Second, I suggest that medicine, even at its cognitive, scientific, data-driven best, is always a social and emotional act. To the extent that this is true, new and better ways to practice are adopted only when data are tightly linked to the appropriate emotional and social forces. Third, I suggest that a respectable understanding of the role that social and emotional forces play in the diffusion of innovations has developed; indeed, it has become an entire discipline within the social sciences [4]. As in other areas, however, medicine has unfortunately had difficulty recognizing and accepting the lessons that this discipline has to teach, particularly those involving the crucial importance of attitude and values [6]. Finally, therefore, I suggest that a major task, if not the major task, for medicine in the next millennium with regard to the use of databases is to develop a deeper understanding of these social and emotional forces. Only then, I believe, will it be possible to put these forces to work in applying database information to problems of quality.

The Use of Data as an Innovation: A Historical Glimpse

Before turning to the future, it may be helpful to look to the past. If we can look forward to databases serving in the future as a major engine of change and progress, we should be able to find at least some convincing evidence that they have served in this capacity over a respectable period of time, that they have a historical “track record.” When did people start to use large databases (as distinguished from knowledge, such as that found in libraries) as the basis for improved management of complex systems? I am not a historian, particularly not a historian of management systems, so I cannot pretend to provide anything approaching a definitive answer. But I recently did come across a striking example of an early application of database information as an instrument of large system reform in Simon Schama's extraordinary text on the history of human concepts of nature, Landscape and Memory [7].

In the 1660s in France, Schama tells us, one Jean-Baptiste Colbert was appointed by Louis XIV to be the administrator of all French forests. Soon thereafter, Colbert warned his king that: “La France perira, faute de bois” (France will perish for lack of wood). Colbert was talking about the almost insatiable need at the time for wood for building ships of the line, as well as for heating and for fueling foundries and manufacturing of all sorts. Wood was, in fact, the principal source of energy (and some of the construction material) for the French economy and, hence, for French imperial power. Lack of wood was, in a sense, the fuel crisis of its time.

Colbert also knew that a serious quality problem underlay this crisis. That is, he knew that the management of the French forests, royal and otherwise, was in chaos. He intended to set things right and, in some fashion still not completely understood by historians, he decided that to bring about the sweeping management reforms needed to avert a major fuel crisis, he needed data-big data-and he acted accordingly.

In the French forests of the 1660s, the data collection scene was very different from that of today's health care system. There were no data abstractors in record rooms, no minicomputers humming quietly in corners, and certainly no practitioners entering data on the fly into handheld computers. Rather, “Carriage loads of men in long wigs and long coats, carrying surveying rods and spools of horsehair twine descended on the forests of Normandy, Lower Burgundy, and the Ile de France.”

But data collection and analysis there were. Colbert's minions worked at the task for years, and (here comes the clincher): “By the end of the 1660s, Colbert had the data he needed to act” (emphasis added). “The object,” says Schama, “as always in Cartesian France, was to bring order out of chaos”; an object lesson, perhaps, for our less-than-Cartesian times.

The result of Colbert's data-driven process was the great Ordinance of 1669: some 500 regulations, in more than 100 printed pages, that served as the Bible of French forestry for more than 100 years, well after the French revolution. Colbert's strategy was, by all measures, a bureaucrat's dream, because for many years it effectively did improve the use of French forest resources.

Data and the People Who Use Them

In Colbert's case, a large database seems to have been necessary to bring about change. This point should be of particular interest to us because without those data, it seems, even the absolute power of the French monarchy was not enough to bring French forest resources under control. But it is also arguable that the data alone would have had little impact if they had not been linked to the monarchy's awesome force. In sum, neither the social, administrative, and military power of the monarchy nor the data alone were enough to bring about the needed reforms. Although each was necessary, neither, by itself, was sufficient; improvement required both.

Indeed, 30 years of recent intensive study has made it clear that both knowledge and emotional engagement are universally required for the diffusion of innovations, no matter what kind of new idea is involved, from the sterilization of contaminated drinking water by boiling, to the adoption of new fertilizers and seed in farming, to the use of fax machines and the Internet [5]. As Rogers conceives of it, the diffusion process for innovations always involves five stages: knowledge, persuasion, decision, implementation, and confirmation [8]. Turning to medicine, the Rogers diffusion model finds echoes in continuing medical education, which, until recently, was the major way in which the profession approached quality improvement. The model most widely accepted in medical education sees clinical work as consisting of four distinct elements (Table 1): knowledge, skills, performance, and outcomes. According to this model, the practice of medicine, particularly high-quality medicine, requires four related activities: understanding a clinical practice, knowing how to carry it out, doing it, and doing it well. Less well recognized, but equally essential, however, is the associated social and emotional infrastructure of clinical learning, without which physicians have difficulty applying these elements of practice [9].

Thus, before clinicians-students, residents, and experienced practitioners alike-are willing to acquire knowledge about a subject, they must be interested in it, at least to some degree. To acquire the competence needed to apply that knowledge in practice, they have to believe in it; that is, they need to find it credible enough to be willing to develop those competencies. To actually use acquired skills in practice, they need to be confident of their ability to do so. And to continue using those skills on a day-to-day basis, consistently and as effectively as possible, they need to be able to take satisfaction over time in the outcomes of those practices. In sum, at every turn, at least one social and emotional influence-interest, belief, confidence, or satisfaction-is involved in effective clinical learning, as well as change and improvement (Table 1).

In fact, the best available evidence tells us that knowledge transfer-that is, access to the information in the enormous database known as medical knowledge-by itself is virtually ineffective in modifying physician practice or performance [10, 11]. In contrast, continuing education can and frequently does lead to the adoption of improvements (that is, innovations) when these changes are first practiced and then promoted locally by “education influentials”: respected leaders in a medical community [12]. Similarly, academic detailing, in which representatives from academic departments personally review the evidence for improved practices in face-to-face, one-on-one encounters with clinicians, has been shown to change medical practice [13]. (Industry has recognized this principle for years, which is why medical equipment and pharmaceutical detailing has continued to flourish.) And clinicians are much more likely to adhere to clinical practice guidelines if they or their colleagues have been actively involved in developing those guidelines or adapting them for local use [14].

It is possible, of course, that the social and emotional dimensions are important in continuing medical education because traditional continuing medical education has focused so narrowly on the product-knowledge (a cognitive product at that)-rather than on the process of medical care and the system in which medicine is practiced [5, 11]. But the evidence suggests that, if anything, the social and emotional dimensions are even more important in dynamic, “real-time research,” the newer approaches to quality improvement that focus on process and systems [3, 5]. Thus, the report of a recent multi-institution study designed to improve outcomes in coronary artery bypass graft surgery describes at length the methods used for intensive, direct observation of the processes of care but is also careful to note the “set of facilitating circumstances that made the study possible.” These include “the willingness to be observed” (termed “courage” by one of the authors of the report), “the ability of the members of the group and its leadership to communicate the value of the study to others in their institutions,” and “the implicit or explicit belief on the part of senior management in each institution that an increased emphasis on QI [quality improvement] is important” [15]. Contrast this description with the recent report noting that large employers in the United States have remained skeptical about the utility of hospital mortality and morbidity data because they believe that “the data are not reliable or timely” and because they doubt “whether it was used or understood by employees” in choosing among health plans [16].

The process by which innovations diffuse is rich and complex, encompassing such important phenomena as re-invention (local adaptation and rediscovery of an innovation), discontinuance (rejection or replacement of an innovation after it has been adopted), and communication channels (either interpersonal or mass media) [8]. The social dimension of the adoption of innovations is particularly fascinating, involving a spectrum of players from innovators through early adopters, an early majority, a late majority, and “laggards” as well as an array of social phenomena, including critical mass effects and thresholds, diffusion networks, opinion leaders, and change agents [17]. Innovations can diffuse spontaneously from the bottom up, or their use can be forced from the top down-with greater or lesser success, depending on the innovation itself, the users, and the nature of the social system in which these events take place.

Some Cautions and Concerns

The history of innovations also reminds us that although “the new” has an almost irresistible attraction (this is sometimes referred to as “pro-innovation bias”), new is not always better [18]. (Indeed, Philip M. Nowlen [11] has commented on “the traditional American love of new gadgetry and dread of being caught up in a fad past its prime” as powerful driving forces for professionals to “keep up to date”; this, in turn, helps to cast continuing professional education overwhelmingly in the “update” mode.) It is not difficult to find examples of innovations that seemed to represent true progress when viewed narrowly but that from a broader perspective were seen to have had damaging, retrogressive, or even perverse effects. The environmental costs of the internal combustion engine, the introduction of bottle-feeding for infants in developing countries, and the unemployment among bracero field workers created by the introduction of a breed of tomato that could be picked by machine are a few of the better-known examples.

It may be useful, therefore, to consider some potential risks of the use of large databases for medical quality improvement in the next millennium. Large databases are relatively new, are technologically sophisticated, and are almost endlessly complex; as such, they are objects of almost endless fascination. The first concern, therefore, is that developers of large medical databases may become so caught up in the intellectual and technical attractions of designing these databases that they will lose sight of why they are creating them in the first place. This is not unlike current concerns about World Wide Web sites; it has been said that by the year 2000, 90% of organizations will have Web sites but only 5% will know why they have them. A related and perhaps more serious concern is that the content, architecture, and functional characteristics of large databases will begin to define and drive quality improvement efforts, rather than the other way around.

A second concern is that even if large databases themselves evolve into powerful, flexible, and responsive instruments of medical quality improvement and are well used for that purpose, their effect on the larger system may be negative. It could be, for example, that these positive characteristics are only achieved by spending very large amounts of money and staff resources that might be better spent in other ways. Stated differently, we might discover that we had purchased a Rolls Royce when all we could really afford was a Chevrolet.

A third concern is more subtle and has to do with our willingness to accept the authority of “black boxes.” That willingness seems paradoxical, given the general point being made here about resistance to the diffusion of innovations, but life is full of such paradoxes. But despite that general resistance, people are often willing to accept complex and mysterious systems on faith, particularly when those systems are “high tech.” People apparently assume in such situations that someone, somewhere has checked out the accuracy and the performance characteristics of the system, whether that has been done or not-the “It must be right because a computer said so” approach.

Finally, in addition to their uses in purchasing decisions, resource planning, and system changes that are applicable to populations, large databases can be and are being used increasingly to predict clinical states in individual patients, as in the APACHE (Acute Physiology and Chronic Health Evaluation) predictive instrument [19]. These data-based “clinical” instruments may have their uses in practice, and current technology even opens up the exciting possibility that the data accumulated during everyday use could lead to continuous improvement in predictive performance. At the same time, the creation of data-based predictive instruments is fraught with pitfalls and untested assumptions [20]. Many of these limitations are not recognized by clinicians, who may assume that because a system is “databased” it must be right, an assumption that can lead to inappropriate uses and interpretations.

Conclusions

Large databases offer the promise of a powerful new way to improve medical quality. However, if we are really interested in improving the chances that large databases will have a major, positive effect on the quality and cost of medical care in the coming millennium, we would do well to confront at least two important but generally underappreciated realities.

First, we need to recognize that large database technology is a true innovation that carries with it all of the social and emotional problems associated with the diffusion of innovations in general. As we continue to refine the cognitive side of database work-the technical and methodologic rigor of the discipline-we need to begin working seriously to define the social and emotional forces that oppose or facilitate the adoption of large databases in quality improvement and to put those forces to good use. Skeptics may write off this approach as “soft stuff” or, worse, just the “marketing”-academic detailing, if you will-of serious intellectual effort. But as experience with many other innovations has shown, simply building a better medical quality improvement “mousetrap” does not mean that the world will beat a path to your door.

Second, we should confront as early and as honestly as possible the limitations and potentially negative consequences of using large databases in quality improvement. Among these consequences are the possibility that people may lose sight of databases as a means to improved clinical practice and see them increasingly as an end in themselves; that the use of databases may achieve local improvements at the expense of negative system-wide effects; and that databases may be misused because of failure to appreciate their limitations and the difficulty of applying them in clinical practice.

Dealing with these challenges will probably turn out to be at least as important to the implementation and full effectiveness of databases as the technical side of the effort. But, although rummaging around in the social and emotional context will increase the difficulty of the work, it should also make the entire enterprise much more interesting, exciting, and ultimately, more satisfying. And why, after all, should others have all the fun?