Measuring and Reporting Managed Care Performance: Lessons Learned and New Initiatives

Report Card Projects

HEDIS and the Report Card Pilot Project

In January 1994, NCQA, a nonprofit organization that is the leading accreditor of health plans, launched a 1-year pilot project to test the feasibility of implementing a standardized set of performance measures then in its infancy: HEDIS, the Health Plan Employer Data and Information Set. Documentation for HEDIS specifies how a health plan should calculate many performance measures in various areas, such as quality and utilization.

The large databases already maintained by health plans for various administrative purposes were initially seen as major resources for supporting the performance measurement initiative. Subsequent discussion, however, revealed great variations across plans in the types and extent of the collected data. For many practical, political, and financial reasons, it became clear that any development activity would have to allow for such variations and ultimately accept measures gleaned from a mixture of sources and methods.

Version 1.0 of HEDIS was introduced in September 1991 and was developed through a cooperative effort involving three major employers-Digital, GTE, and Xerox-along with representatives from the health care industry. Soon after its introduction, HEDIS was turned over to NCQA for further development. In October 1993, NCQA released a revised version, HEDIS 2.0, which contained about 60 performance measures [3]. Updated specifications were released during the next several years [4], followed by a comprehensive revision and expansion to approximately 100 measures. The latter version was released in early 1997 as HEDIS 3.0 [5].

In contrast to the previous versions, HEDIS 3.0 1) focuses on outcome as well as process to measure quality; 2) includes measures for Medicare and Medicaid populations in addition to commercially insured groups, thereby bringing private and public measurement efforts together; 3) covers a fuller spectrum of health care, notably illness as well as preventive care; 4) standardizes the member satisfaction survey process; and 5) addresses implementation issues, notably the requirements for management information systems.

An acknowledged deficiency of HEDIS 3.0 remains its inability to adjust for differences in population health risk. Readers who want more information on NCQA and HEDIS can visit the NCQA site on the World Wide Web (http://www.ncqa.org).

The developers of HEDIS and potential users of the data have been keenly interested in assessing how difficult implementing such a system would be. As an initial response, the RCPP was undertaken in 1994. Twenty-one health plans chose to participate and fund this groundbreaking effort. The goal of diversity was kept in mind as the plans were chosen. Chosen plans were large and small, urban and rural, and of different types: group-, staff-, and independent-model health maintenance organizations, along with a single preferred-provider organization.

In January 1994, an RCPP steering committee met for the first time. The committee consisted not only of plan representatives but also of persons from major employers, consumer and labor groups, and health policy organizations. The NCQA staff provided leadership, staff support, and technical assistance. This group of 43 persons met monthly for the next year and accomplished the following.

1. Selected and refined the measures. Between January and May 1994, the Steering Committee and a group of commissioned task forces worked on selecting a subset of HEDIS 2.0 measures based on value to a broad audience, ability of the health plans to supply requisite data, and appropriateness for across-plan comparison. Twenty-eight measures were selected from the broad HEDIS 2.0 categories of quality, access and patient satisfaction, membership and utilization, finance, and health plan management and activities. These measures were then refined with respect to such issues as methodology, technical specification, age and sex stratification, and reporting.

2. Conducted a member satisfaction survey. After some debate, the steering committee decided that the member satisfaction ratings chosen for the RCPP would be obtained from a standard survey instrument and would be administered by an external vendor to ensure objectivity and comparability. The Group Health Association of America's Consumer Satisfaction Survey [6] was chosen as the instrument because it is well established and has been thoroughly tested. A university-based vendor was picked to administer the survey.

3. Generated the performance data. In June 1994, NCQA disseminated the final technical specifications for the RCPP. Using these specifications, the 21 participating health plans began collecting the requisite data. Three basic methods were used to collect data: compilation of administrative data, review of medical records, and a newly developed hybrid approach combining these two methods.

Although some plans expected to be able to use administrative data exclusively, this proved unfeasible. Because of problems with quality and completeness, HEDIS results from these data sets alone were discouraging. These disappointing results led to the hybrid strategy, in which medical record review augments the administrative data. Although considered successful in generating more accurate results, the medical record reviews added cost, time, and intrusiveness for the participating plans.

4. Conducted an external audit. The steering committee clearly recognized that assuring the integrity of the data was key to accepted and trusted results. In this spirit, an external audit was made a linchpin of the project. A third-party vendor was chosen to conduct a major portion of the audit. An expanded discussion of the auditing function follows this section.

5. Developed a communications strategy. Identifying the information needs of the diverse audiences to be served by the RCPP was indeed a challenge. An equally daunting task was deciding how to format the data in order to streamline presentation without sacrificing scientific soundness. Feedback from the final product indicates that the RCPP successfully addressed these two critical components.

Figure 1 is a page from a sample plan profile issued by the RCPP showing groups of measures in the general categories of member satisfaction and quality of care. With allowance for variation due to sampling error, the plan described in this profile is about average with respect to the eight dimensions of member satisfaction. Quality measures, however, varied considerably. For example, the plan's performance is roughly average for cholesterol screening, below average for mammography, and above average for Papanicolaou smears. The implications of developing and using such performance profiles in the context of quality improvement in managed care are profound.

Data Auditing

Auditing health plan data, such as HEDIS, is generally done to assure the integrity of the measures and to instill confidence in the users of the data that the data are accurate. This is critical in the case of HEDIS because the data are being produced by the health plans themselves rather than by an outside organization. Auditing is also done to identify problem areas in health plan systems for the capture of data so that plans can continuously improve the abilities of their internal data systems.

It was extremely valuable to evaluate the participating plans' ability to completely and accurately collect and report the data necessary for the HEDIS 2.0 measures in the RCPP. The two major tasks were reviewing the process plans followed in accessing the data and validating the measures themselves.

The goal of the process audit was to evaluate the processes that act on data as they flow from the provider's encounter with the patient until they reach one or more centralized databases in the final form necessary for calculating the HEDIS 2.0 measures. During the process audit, plans performed an internal baseline assessment with a document created by NCQA. The document requested information on data acquisition capabilities, data management and reporting activities, and physician compensation structure. Site visits were conducted to validate the plans' baseline assessment and to document best practices. A particularly productive aspect of the process audit was the verification of the specifications and source code used by the plans.

The objective of the measures audit was to evaluate a subset of end point data (the output of the process) and compare this subset to “ground truth.” In this case, the medical record was considered the ground truth. For administratively derived measures, medical records were abstracted to determine the level of agreement. When the measure was obtained by using medical records or the hybrid approach, the external auditor reabstracted the medical records and compared the findings.

Complete results of the RCPP auditing feature can be found in the project's final technical report [7]. In general, the audit found that plans frequently departed from the underlying HEDIS 2.0 specifications. The plans were allowed to take corrective action so that by the end of the RCPP compliance had largely been attained.

Local Report Card Projects

After conclusion of the RCPP, NCQA began to organize several local report card projects in order to further test and refine HEDIS and the audit strategy and to evaluate the use of HEDIS in local market settings around the United States. In addition, local report cards built on HEDIS began to develop from existing business coalitions and coalitions set up specifically to gather HEDIS data. For example, the Pacific Business Group on Health, working with all health maintenance organizations in California, launched the California Cooperative HEDIS Reporting Initiative; the New England HEDIS Coalition produced its own HEDIS report card; and the North Central Texas HEDIS Coalition developed health plan performance profiles. A commercial magazine, Health Pages, has continued to publish HEDIS-based plan profiles in markets around the United States [8].

Lessons Learned

The lessons learned, as reported in the technical report of the Report Card Pilot Project, fall into four broad categories.

Enhancement of Performance Measures

1. Broaden HEDIS. The NCQA staff acknowledged that HEDIS 2.0 was, by nature, restricted in focus. The HEDIS 3.0 developmental process addressed this issue directly, expanding the domains of measurement to the eight measures listed previously (effectiveness of care, access and availability, satisfaction with the experience of care, stability of the health plan, informed health care choices, use of services, and health plan descriptive information). The number of measures was increased commensurately.

2. Research the potential of integrating risk adjustment. Comparing plans on measures that are affected by characteristics of the plan membership can be problematic unless some type of adjustment is done [9]. However, the costs and complexities of doing this on an industry-wide basis are formidable. Thus far, HEDIS 3.0 has been able only to indicate that developmental work is needed.

3. Perform field tests of future measures. The RCPP was the field test of HEDIS 2.0. It is preferable to perform field tests of the measures before they are finalized. HEDIS 3.0 moves in this direction by specifying a “testing set” of additional measures that will be refined within 2 years.

Health Plan Information Systems

1. Improve standardization. Differences in health plan data systems and reimbursement and compensation structures were evident in the RCPP audit. Auditing was critical because resulting data sets varied with regard to content, completeness, coding, and reliability. Greater standardization would enhance comparability and integrity of data and reduce the audit burden.

2. Enhance investment. In the past, health plans tended to focus their attention on administrative and financial information systems. Greater attention to and resources for clinical information systems will improve quality measurement and data comparability. Ultimately, such an investment should also improve plan performance.

External Auditing

1. Perform and refine external audits. In the RCPP audit, many of the errors and inconsistencies identified were a function of the newness of HEDIS. As plans become more familiar with HEDIS, many of the problems should abate, allowing the audit process to be streamlined. To remain viable, the process followed in the RCPP must be made less cumbersome and costly. Whether this will be feasible remains to be seen, given the expansion of HEDIS and the increased emphasis on standardization.

2. Remove barriers to implementation of an external audit. It is important that audit teams be uniform in their activities, and source codes must be reviewed in a standardized fashion. These are but two of numerous barriers that need to be overcome before auditing of HEDIS data can be fully operational. Because of these costs and problems, NCQA does not require that HEDIS 3.0 data be audited by an external party, and the credibility of the plans' reports will no doubt suffer.

Communication of Performance Information

1. Begin communications planning early. Early and frequent communication with stakeholders is essential in a collaborative project of this kind.

2. Research consumer needs. With the many interested constituencies, it is difficult to get input from all parties. More study is needed to determine consumer needs for information.

New Initiatives and Directions

Physician Profiling

Concurrent with increasing the flow of aggregate performance measurements to employers and other purchasing groups, health plans are beginning to cull similar information at the physician level from their large administrative databases. The objectives of profiling are twofold. First, some plans consider their contracted physicians to be simply another audience and believe that reporting to them should be a natural and essential part of the relationship. Second, the transmittal of physician-level performance measures is becoming an important component of programs designed to improve the efficiency and effectiveness of the delivery system itself.

One type of physician-based performance measurement is hardly new. For years, many managed care organizations have been compiling statistics on the aggregate costs incurred in rendering services to the membership panel of each primary care physician and, in turn, basing some component of compensation on the results. To help primary care physicians better understand the relation between utilization and cost, the data may be broken down into various service categories, such as radiology and pharmacy. Rates of specialist referral and inpatient admission, generally presumed to be key drivers of cost, may accompany these statistics. Inevitably, however, this emphasis on cost has neither endeared the plans to their physicians nor assuaged the public's concerns about the quality of care rendered under the accompanying incentive structure.

Predictably, the determination of quality-oriented measures of individual physician performance under managed care is being strongly influenced by the external requirements for aggregate plan reporting. Some of the same data structures and software can be used, and a plan's desire to improve its externally reported performance leads naturally to profiling at the individual level. Thus, for example, profiling immunization and screening rates among the appropriate demographic groupings of a primary care physician's membership panel is now fairly routine. Further, given the perceived importance of the physician–patient relationship in membership retention and overall satisfaction ratings, as well as recognizing the clinical linkage to promoting compliance with appropriate regimens of care, member satisfaction surveys with scores calculated for individual primary care physicians and, occasionally, specialists are becoming commonplace. Finally, many plans are beginning to compare physicians with regard to their practice patterns and appropriateness of care for particular diagnoses and conditions; case-mix adjustments are then performed to compile summary level profiles.

Even though health plans are finding value in expanding their physician profiles beyond simple statistics on cost, their recognition of potential pit-falls is also expanding. The risk adjustment issue, already a concern for plan-to-plan comparisons, is intensified at the physician level by the much smaller number of cases available. This is particularly true for physicians who treat patients from many different plans. Further, although the limited clinical information that can be obtained from administrative data can be supplemented by reliable indicators gleaned from medical record sampling at reasonable cost for plan-wide measurement, this process is prohibitively costly when profiling across a network of individual physicians is done.

Figure 2 shows one effort by a health plan to compile a brief profile of clinical activity among a primary care physician's (in this case, an internist's) panel of patients. As is typical, the profile presents similar statistics for the subject physician and for a peer group of participating physicians in comparable circumstances. On several indicators, a judgment is rendered on whether the comparison exposes an outlier pattern-a significant difference between the performance of the primary care physician and peer performance.

If such a profile does show an outlier situation, the physician should expect the plan to follow up with a more detailed “drill-down” of data to identify the reasons for this. An apparent outlier is often quickly disregarded as resulting from only a few cases; these few cases would rarely be questioned, given the imprecision of the administrative data underlying the report. In other instances, the difference can be traced to particularly imprecise or inaccurate coding of diagnosis and procedure, which the plan may seek to remedy through discussion with the responsible physician or office staff. When a more substantive pattern of care deviating from peer standards is ascertained, practical interventions could be directed not only to the manner in which the primary care physician renders services but also to the overall management of care and the choice of specialists for referral.

The appeal of profiling and feedback has been recognized by physicians themselves when they are placed at financial risk for the utilization of services. A survey of medical group practices that contract with a health maintenance organization under capitation found that 79% of these groups profile individual utilization rates and 58% report these results to their physicians [10]. Whether done by the plan or by the group, however, the effects of profiling have received relatively little rigorous investigation. A recent meta-analysis of randomized clinical trials found a total of 12 such studies; only 3 of these explored an ambulatory-based, managed-care environment [11]. This analysis, although it was limited, did document a statistically significant but modest effect of profiling and peer-comparison feedback on the use of clinical procedures.

A controversial aspect of physician profiling is whether the results are to be shared with a plan's employer clients and membership. A few plans do release some performance-based information, especially satisfaction scores, to help members choose a physician, but most do not. Plans that do release information no doubt wish to create an extra incentive for physicians to improve in the reported aspects of care. These plans presumably perceive a marketing advantage as well. Plans that do not release information may question the marketing aspect and their own image as a true “manager” of care. Should improvement in quality be visibly dependent on members making the right choices, or should the emphasis be placed on internal management to bring all contracted providers up to the level of the “top” performers? Plans dealing with these issues must also address a heightened concern about methodologic issues and the costs of responding to the inevitable queries from employers, members, and physicians.

Outcomes

Current efforts to measure and report on managed care performance focus on the structure of the organization, the process of care as described by utilization and cost statistics, and the satisfaction derived by plan members. Will health care outcomes, generally acknowledged as the ultimate indicators of quality, receive commensurate attention?

Few observers argue that health plans should ignore the measurement and tracking of outcomes. A managed care organization that takes responsibility for the provision of a full range of services is in a unique position to assemble the requisite data and create viable interventions that may lead to demonstrable improvements in clinical quality. Indeed, various public and private interests (especially pharmaceutical manufacturers embarking on an orientation to the total management of disease) are rapidly converging on managed care as the preferred locus of such activities. However, the external reporting of outcomes is much debated.

Outcomes reporting is handicapped by most of the difficulties that affect the external release of other types of performance measures, only more so. Well-recognized impediments are low volumes of cases in a single health plan together with the imprecision precision of available diagnostic information and the resulting inadequacy of case-mix adjustment. The plan may not even be able to observe some of the risks affecting outcomes [12]. Member turnover is another issue that affects the validity of all but short-term outcomes as measures of the quality of care received in the plan itself. Although focused research, special studies, data pooling, and external grants may surmount these problems, routine reporting is another matter entirely.

The managed care industry's caution toward outcomes reporting and concern for the attendant heightened costs are indicated by the relatively small number of such measures developed by NCQA for HEDIS 3.0. (The notable exception is a broad-based, longitudinal assessment of change in the self-reported functional health status of members enrolled through Medicare risk contracts.) However, influential employers and public agencies continue to express their desire for outcomes-based statistics. Certainly, some experience must be gained before the merits of the exercise are clearly worth the costs.

Conclusions

The use of report cards and performance measurement will no doubt continue, and plan and provider performance profiles are becoming a staple commodity. However, improvements along the lines of those we described are needed. The underlying data must become more detailed and accurate, their retrieval must become more efficient, individual measures must be tied more convincingly to the well-being of plan members, analytical methods for comparing plans to one another must be strengthened, and the integrity of the resulting reports must be assured.

The attractiveness of performance measurement in managed care rests on the belief that, for the first time, organizational accountability for the care of defined populations is being made broadly available to the U.S. public. As performance data become more widely available and of higher quality, the potential for using them to improve the care rendered to managed care memberships is obvious. This potential will be realized only with the consultation, cooperation, and good will of the participating physician.