Building Measurement and Data Collection into Medical Practice

A Measurement Story

Good measurement can help physicians improve the care they provide. The following case describes the experience of busy clinicians who used measurement for improvement.

An internist in a multispecialty group practice with several locations learned about a colleague in another community who had used a telephone protocol to streamline care for women with recurrent urinary tract infection. The internist was curious and a little skeptical but decided to try the protocol out for herself. After consulting with a partner who had gathered several protocols, she brought together her clinical team to organize a small test of “protocol care” and measure the results.

The internist and her team targeted a specific population (women 18 years of age or older who telephoned the office with symptoms of dysuria), established a broad aim (improvement of clinical outcomes and patient satisfaction and reduction of costs of care), and selected a balanced set of outcome measures to evaluate the protocol (clinical outcomes, including symptom resolution, side effects, and complications; costs, including those of urine cultures, office visits, and first-line antibiotics; and patient satisfaction).

When they analyzed and discussed their existing care process, members of the team learned that different physicians handled similar patients in very different ways. For example, they varied in methods of risk assessment, use of diagnostic tests, choice of antibiotics, and approach to patient follow-up. The protocol that the team adopted was based on a combination of their experience, their colleague's work, and the scientific literature [2, 3]. The protocol divided women into high-risk and low-risk groups; low-risk patients received telephone treatment by a nurse-administered algorithm and a follow-up telephone call at 7 days to assess results. Before embarking upon full-scale testing, the protocol was tested by a single nurse on 10 patients and was revised on the basis of that experience.

When the team studied their results for the first 130 consecutive patients with urinary tract infections (mean age, 55 years; high-risk patients, 52%), they found that they had used the protocol with 9 patients per month, that 21% of patients were given same-day office visits, that 44% of patients had received a urine culture (which had been universal procedure before), and that 60% of patients were treated with the first-line antibiotic suggested by the protocol. Telephone follow-up was achieved for 100 of the 130 patients. Of these patients, 87% had symptom resolution in 7 days, 11% had side effects of medication, and 1% had a clinical complication. All of the patients whom nurses managed by telephone with the protocol reported high satisfaction. Many patients volunteered that they were delighted to receive treatment without having to make an office visit.

Measurement and Improvement

Measurement and improvement are inextricably intertwined. The preceding case shows how measurement can support clinical improvement in local settings [4].

The urinary tract infection team began with curiosity about a novel clinical approach [5]. They wrote a broad aim statement that called for a balanced set of outcome measures and built a structured “observation point” into the patient follow-up routine to gather information on clinical outcomes, costs, and patient satisfaction. After promptly running a small pilot test, they began to use the new protocol and collected data as the change was taking place. They analyzed both qualitative and quantitative results to assess the impact of their innovation and to determine whether the new approach should be adopted, modified, or abandoned.

Measurement and improvement are two sides of the same coin. The connections are evident in the simple model for improvement that was presented in the introductory paper in this series [6]. The model comprises three questions.

Aim: What are we trying to accomplish?

Measures: How will we know that a change is an improvement?

Changes: What changes can we make that we think will lead to an improvement?

The model also incorporates the Plan-Do-Study-Act cycle (plan the change, do the change, study the results, and act on the results on the basis of what has been learned).

The second question in the model specifically calls for measurement, but data collection is also integral to all of the steps in the Plan-Do-Study-Act cycle. Measurement methods are described in the Plan step; data are gathered in the Do step; information is analyzed in the Study step; and key measures are monitored in the Act step.

Principles of Measurement

For measurement to be helpful in the improvement effort, a few simple principles can act as guides.

Principle 1

Seek usefulness, not perfection, in the measurement. The urinary tract infection team focused on key clinical results and patient feedback, even though they could have chosen to cover more territory. They skipped baseline data collection and opted for a prompt feasibility test on 10 patients. This reflects an emphasis on practicality rather than comprehensiveness.

It helps to begin with a small, useful data set that fits your work environment, time limitations, and cost constraints. The utility of data is directly related to the timeliness of feedback and the appropriateness of its level of detail for the persons who use it. The choice of measures should be strongly influenced by considering who will use the data and what they will use it for. It may be helpful to gather baseline data; however, gathering data over time is often sufficient to spot effects in time series analyses. The goal is continuous improvement with concurrent, ongoing measurement of impact.

Principle 2

Use a balanced set of process, outcome, and cost measures. The urinary tract infection team wanted to do more than just cut costs. They sought a better way to treat infections that would yield better clinical outcomes, fewer side effects, higher patient satisfaction, and lower treatment costs. They wanted to measure clinical value: that is, outcomes in relation to costs [7].

Medical care systems comprise subprocesses that interact, flow into and out of one another, and contain feedback loops. They produce a fluid family of results that include clinical outcomes, functional status, risk level, patient satisfaction, and costs. This complexity has important implications for tracking attempts to make improvements; most important, it requires a mix or balance of measures to do it justice. Balanced measures may cover “upstream” processes and “downstream” outcomes to link causes with effects; anticipated positive outcomes and potential adverse outcomes; results of interest to different stakeholders (such as patient, family, employer, community, payor, and clinician) because participants have differing viewpoints on the relative importance of the many manifestations of care; and cumulative results related to the overall aim as well as specific outcomes for a particular change cycle [8]. More detailed explorations of balanced measures of quality and value have been published elsewhere [9, 10].

Principle 3

Keep measurement simple; think big, but start small. The urinary tract infection team's broad aim was to improve outcomes and lower costs, but they selected a sparse set of outcome measures.

Principles 1 and 2 operate in different directions, creating the need for principle 3. Anyone who wants to improve a system in the real world must balance a fast start and lean measurement with a broader understanding of the complex web of causation [11]. We recommend that you recognize and discuss the true complexity of data collection, but when you are ready to make the data collection plan, strive for simplicity amidst the clutter and focus on a limited, manageable, meaningful set of starter measures.

Principle 4

Use qualitative and quantitative data. The urinary tract infection team used quantitative data to create tension for change and to measure impact on clinical behavior. They used qualitative data to learn how the physicians, nurses, and patients felt about the new system.

Data and measures are meant to reflect reality, but they are not reality itself. Reality has an objective and subjective face, and both are important. Quantitative measures are better at capturing the objective world, whereas qualitative measures are better at reflecting subjective issues [12].

Principle 5

Write down the operational definitions of the measures. The urinary tract infection team wrote operational definitions for clinical outcomes and medical costs. For example, to measure symptom resolution, nurses telephoned patients 7 days after their index date and asked, “Are you still bothered by your urinary tract symptoms? Please answer ‘yes’ or ‘no’.”

The clarity of the signal sent by measures depends on how well everyone doing the measurement understands operational definitions and on how consistently they are used [13]. An operational definition provides a clear method for scoring or measuring a variable in a reproducible manner. The better the operational definition, the better the data elements; the better the data elements, the more reliable and valid the aggregate measures.

Principle 6

Measure small, representative samples. The urinary tract infection team began with a 10-patient pilot group and proceeded to collect data on a consecutive series of patients that presented with the problem.

The clarity of the signal received from the measures also depends on how representative the samples are. One way to get a representative sample is to measure everyone; another is to measure all of the time. This universal approach is sometimes best; in fact, it may already be a routine procedure (such as recording vital signs, sex, or age). However, it is sometimes necessary to gather prospectively data that are not usually collected. Under these circumstances, it is generally best to use a sampling strategy that avoids the costs and trouble of collecting data on everyone or all of the time [14]. Researchers often do not appreciate how powerful and illuminating a representative sampling strategy can be for understanding variation. Again, the emphasis is on usefulness, not perfection.

Principle 7

Build measurement into daily work. The urinary tract infection team built data collection into the nurses' regular routine of telephone follow-up. The nurses used a self-coding data collection sheet (a form for recording data that has operational definitions and response choices built directly into it), making scoring and measurement easy.

Every process constantly throws off data that can reveal how that process is performing. If the data are not available from some other source and if they are essential, it helps to find ways to capture the data as the process is taking place by using the participants to code data as the work is being done. Common options include the use of self-coding data sheets and modification of the information system to help the right person capture the right information at the right time. Examples of self-coding data sheets are structured visit encounter forms, laboratory order sheets, and some patient satisfaction questionnaires.

Principle 8

Develop a measurement team. The case of the urinary tract infection team illustrates the value of developing a measurement team. Although exceptions exist, it is generally best to avoid having one person try to create a measurement system alone. Success in measurement requires time and technical expertise. Team up to lighten the workload, add knowledge, and boost morale. When things are not going well, having the support of others can provide a needed psychological lift.

The Physician's Role in Measurement

Assume that physicians decide to make improvements to the functioning of their practice and want to know if their changes have worked. How will they know whether a change is an improvement?

The short answer is simple: Collect some data on baseline performance, plot the data over time (preferably on a process control chart), start a test of change, and see whether the control chart shows a substantial improvement after the start point for the change [15] (Figure 1 and Figure 2).

This approach does not prove unequivocally that a change has produced improvement, but it begins to build evidence one case at a time. In fact, some students of change believe that the nontrivial replication of results is the strongest proof of an hypothesis [15]. In addition, common methods of data analysis are available for using time-ordered data to decide whether a change is occurring (such as a trend, a shift, or a substantial increase or decrease) on the basis of standard statistical concepts [15]. These rules are based on a fundamental understanding of the nature of variation-the ability to distinguish regularly occurring random fluctuations (common cause variation) from special cause variation [17].

The power of this approach is illustrated by the experience of Dr. Larry Staker, a general internist in private practice who asked a visiting medical student to do a chart review on a sample of his diabetic patients. The results were disappointing: The patients' glycosylated hemoglobin values averaged 10.8%, which was higher than expected. He decided to use a protocol to discuss treatment plans with the diabetic patients, to set a specific blood glucose level goal for each patient, and to teach each patient to plot his or her self-monitored blood glucose levels daily on a control chart that showed the patient's level compared with the goal. For many patients, the results were spectacular. Figure 1 shows the improvement in self-monitored blood glucose level achieved by one of the first patients to be involved in this new approach to care.

Not all patients were equally successful: Variation happens, and the data showed that some patients did well and some did badly. However, Dr. Staker was able to gauge the success of his new approach one case at a time. When he checked on how well he was doing by repeating the focused chart review with his entire panel of diabetic patients, he found that average glycosylated hemoglobin values had decreased from 10.8% to 7.4%. Plotting data over time is often a good way to start using measurement to learn about variation and improvement. The Appendix describes an efficient approach to planning data collection [9, 10, 18].

Although the physician's role in hospitals and other large, complex organizations differs from that in the office, physicians can promote measured improvement by posing questions and working with other members of their health care teams to develop suitable, practical answers. To this end, physicians may consider asking the following types of questions.

Benchmark: “Who gets the best outcomes? Who has the lowest costs?”

Performance: “What is our current performance? What is the best that we can do today?”

Improvement: “What could we do with our next 25 patients to improve our performance?”

Success: “If we are successful, what could we measure in these next 25 patients to build evidence that the change may be working?”

Failure: “If something goes wrong, what can we learn about the process that produced the failure, and how can we measure in the future to get an early warning signal?”

Data displays: “How can we display our data to show, as quickly as possible, whether we are achieving the results we hope to achieve and avoiding the problems we hope to avoid?”

Special Challenges

Practitioners who try to increase their use of measurement to support improvement often meet obstacles and challenges. Using measures in clinical practice may look easier than it really is. For many practitioners, the systematic use of data for improvement is a departure from the status quo. Anticipating common barriers can help increase the odds of success.

Interpreting Results: Making the Data Tell a Story

The design of measurements should begin with clarity about the underlying question that must always be answered: “If we do this differently, will we get better results?” The challenge is to display the data in a manner that answers the critical questions and tells the story of what has happened over time.

A good way to do this is to use graphical data display methods. Some graphical displays are already familiar to physicians in the form of laboratory result reports, medical record flow sheets, and intensive care monitoring screens [19]. Graphical displays, such as run charts, dashboards, instrument panels [20], or clinical value compasses (which provide a balanced set of quality and value measures, such as clinical outcomes, functional status, patient satisfaction, and costs) [18], can answer key questions and tell the story as it unfolds. Of course, some measures that are needed most to reveal key outcomes (such as disease progression, functional status, patient satisfaction, and costs of care) are fraught with reliability, validity, and sensitivity issues that merit careful attention. It may be useful to get focused, expert statistical advice on these issues.

Plotting the results in time order and updating them frequently is a good method for graphically telling the story. Time trend charts can display variation in results by patient type (for example, by stratification) and under different conditions (such as time of day, day of week, or provider site). An example of a clinical team's instrument panel for improvements in inpatient pneumonia care is shown in Figure 2. The team used control charts to show results over time for key measures, including the number of days that intravenous antibiotics were administered, the amount of time required to administer antibiotics, and the length of hospital stay. Other methods of summarizing and analyzing data used in standard medical statistics can help augment a graphical display and provide further information [21].

Is This Science? Traditional Medical Research and Continuous Improvement Approaches

When physicians are challenged to make improvements, they sometimes respond with suggestions that are, in effect, roadblocks. They may demand complex analyses and long baselines (“Let's first get someone to analyze the trends for the past few years by patient type, disease severity, and site of care”), unnecessary randomization and large samples (“That would be great, but we won't learn anything if we don't randomize several hundred patients”), or investment in large databases and information systems (“I'm willing to try it, but only if we can get someone to pay for the information system to get us the data we need and to do our analysis”).

Each of these suggestions has merit in some settings and is common practice when doing funded academic research. However, these approaches may be inappropriate for the fast-moving, front-line world of patient care, in which the time and resources used for funded research are often lacking. A small-scale, rapid-cycle, iterative approach to the use of empirical methods is preferable in many clinical settings. It is usually best to start with before-and-after data or data collected over time, gradually increasing the scale and complexity of the interventions themselves, moving from small changes to major innovations [22]. Standard research designs (experimental, quasi-experimental, and observational) [23] and methods of data management, analysis, and display (tables and graphs) are routinely adapted to meet the needs of the situation. Methods of analysis and reporting can become more sophisticated as the improvement team becomes more experienced.

Data Collection and Measurement Can Be Complex, Technical, and Costly

Although we have already pointed out many challenges and problems, we have ignored other thorny analytical considerations, such as sampling theory, clustering effects, reliability, validity, sensitivity of measures to change, and the difference between administrative data and clinical data. Space constraints do not permit full coverage of all of these topics. We do not believe that the Plan-Do-Study-Act approach is a “dumbing down” of the scientific method; rather, it fits the tools of science to the pragmatic needs of real physicians attempting to get better results in real-world settings. Both real physicians and real-world settings have tangible constraints that must be respected in the short run.

Other real obstacles that challenge physicians are the time required for and the financial costs of data collection and analysis. The physicians who did the diabetes case study capitalized on “helping hands”: Students helped to gather and analyze essential data. The urinary tract infection and pneumonia teams added some modest costs by gathering new data on selected measures. Certainly, measurement may consume valuable resources. This is one important reason to start small on important problems, build new data collection into the care delivery process, and use appropriate methods for the analysis of data and display of results. Nevertheless, improvement requires measurement, and some investments must be made today to produce better results tomorrow.

Conclusions

Any physician can make changes to his or her practice. It is done every day-trying this medication and that therapy, one patient at a time, with little measurement and scant reflection on results. However, it is feasible to make changes that lead to measurable improvements in quality and value for defined groups of patients and to learn over time what works, what fails, and why.

We have presented real clinical cases to illustrate how practicing physicians can measure aspects of their practice and use the resulting data to better understand and improve the care they deliver. Measurement can lead to improvement, but sustaining improvement is almost impossible without measurement. Both qualitative and quantitative data can help reveal how the system worked before, how it works now, and what will happen after a change is made. Physicians may wish to take the following steps when using data to improve their clinical practices. First, begin with curiosity about outcomes or a need to improve results. Second, try to avoid knee-jerk, obstructive criticism of proposed measurements. Instead, propose forward-directed solutions that are practical, goal-oriented, and good enough to start with. Third, gather baseline data on a small sample and check the findings. Fourth, try to change and improve the delivery process while gathering data. Fifth, plot results over time and analyze them by using a control chart or other graphical method. Sixth, refine your understanding of variation in processes and outcomes by dividing patients into clinically homogeneous subgroups (stratification) and analyzing the results separately for each subgroup. Finally, make further changes while measuring key outcomes over time.

Appendix

A Logical Approach to Planning Data Collection and Measurement

The following regimen can be used to avoid measurement gridlock.

1. Write down the critical questions that must be answered. Are outcomes better? Are costs lower? Is satisfaction improved? Has the new way been adopted, and under what conditions? Do outcomes and costs vary by patient type, location of service, or provider type?

2. Design dummy data displays that will be used to answer critical questions. Draw out your tables or graphical data displays (such as run charts, control charts, box plots, or bar graphs) and fill them in with made-up numbers, as if the pilot test had succeeded or failed. Ask yourself, “If I had these results and displayed them in this way, would I have an answer to my critical questions? Would I have good evidence that the pilot test was failing or succeeding?”

3. Make a list of variables that must be collected to fill in the dummy data displays and write down conceptual and operational definitions for each one.

4. Overall, write a simple protocol and follow it.

After making your dummy data displays, you are in a good position to streamline the measurement and analysis path. You know just what you want to measure because it is needed to answer a critical question. You may defer measurement of other things until later if those measurements are not needed to answer a critical question.

Albert Einstein said, “Theory precedes measurement.” Using the process outlined above gives you a simple way to state your theory about cause and effect and variations in outcomes and to specify the measurement that will be used to test your theory.

A Data Collection and Measurement Checklist

Aim, Users, and Uses

1. Ensure that the intended use and analysis of the data are clear.

2. Ensure that the methods of organizing, displaying, and summarizing data will permit the study of factors that have important effects on results.

Definitions

1. Develop clear definitions of how observations will be translated into measurements or evaluations.

2. Ensure that the method of measurement will result in acquisition of the intended information.

3. Ensure that the measurement methods to be used are clear and simple and that they minimize on-the-spot decision making.

Planning

1. Include provisions for recording potentially important auxiliary information (for example, in a diary) in the design of the measurement process.

2. Embed measurement and data collection in the daily activities of the system under study.

3. Ensure that measurement and data analysis are timely.

4. Develop a plan for training persons who will make the measurements and record the data.

5. Perform a small pilot study of definitions, methods of measurement, data collection forms, and training.

6. Determine who will be primarily responsible for the measurement process.

7. Inform all affected persons about the purpose of collecting data. Help persons to be comfortable with the purpose of the measurement and alleviate fear of being measured.

Dr. Berwick (Series Editor): Institute for Healthcare Improvement, 135 Francis Street, Boston, MA 02215.

Dr. Nolan (Series Editor): Associates in Process Improvement, 1110 Bonifant Street, Suite 420, Silver Spring, MD 20910.