Studies based on record reviews are limited by the completeness of records, whereas those based on physician or patient self-report, have limitations in accuracy of recall [3-10]. In addition, variations in patient presentation and case mix make it difficult to compare physicians accurately. "Standardized" patients are lay persons trained to present a fixed patient scenario [11-13]. They provide an ideal method to assess physician performance while controlling for variability in patient characteristics, especially if physicians are unaware of the specific evaluation encounter. Although this tool has been used extensively in teaching and evaluation of clinical and communication skills in medical student and residency training programs [14-24], its use in clinical research is limited. Unannounced standardized patients can enter practices undetected [12, 19, 21] and can provide accurate observations consistently [22, 23].

We used standardized patients to describe the physical examination procedures done by primary care providers in the context of the periodic health examination. The data were collected as part of a study to determine whether education or an office system would improve cancer control services provided by primary care providers as assessed by standardized patients. By virtue of their voluntary participation in the Cancer Prevention in Community Practice (CPCP) Project, the providers in this study were more likely to be oriented toward cancer prevention and control than were nonvolunteer providers. The actresses, who posed as new, asymptomatic patients overdue for most indicated services and at risk for lung, breast, and colorectal cancer, requested a "checkup". Our report describes physician responses to this request from these unannounced standardized patients and also describes the feasibility of this approach.

Methods

Top Methods Results Discussion Conclusion Author & Article Info References

The CPCP Project was a randomized, controlled trial in community practice that tested two interventions designed to improve physician compliance with National Cancer Institute recommendations about early detection of breast, cervical, and colorectal cancer [25]; about counseling on smoking cessation [26]; and about nutrition [27]. The interventions consisted of an office system and physician education. Results from surveys of study patients and record reviews are reported elsewhere [28] and indicate that physician performance of most National Cancer Institute recommended services increased (P < 0.05) when an office system intervention was provided. This prevention intervention used medical record flow sheets and required an increased role for office staff in providing the indicated preventive services [29]. Implementation of the office system was supported by a nonphysician consultation from the Project using the facilitator model developed in the United Kingdom by Fullard and colleagues [30, 31].

The physician education intervention increased only 1 of 10 services [29]. This intervention consisted of an 8-hour meeting where National Cancer Institute recommendations, their rationale, barriers to providing indicated services, and necessary skills were reviewed. In the format used, physicians met in small interactive workshops with content experts on each target area [32].

The standardized patient instrument was designed to complement record review and patient survey results, which were the primary methods used to evaluate the study's main effects. Whereas these other evaluation instruments assessed services provided to established patients of the practice during a 12-month period, regardless of whether they had received a periodic health examination, using standardized patients allowed for evaluation of examinations provided to new patients.

Study Participants

Of the 98 physicians participating in the CPCP Project, those who were accepting new patients provided the sample for this study. Physicians were excluded if they had practiced at their current office for fewer than 2 years, provided ongoing care to few adults, devoted most of their clinical time to a subspecialty, or were based in a major residency training site.

Design

At least 9 months before the standardized patient visit, physicians were randomly assigned to receive the office system intervention, the physician education intervention, both, or to serve as controls. A standardized patient telephoned each practice and explicitly requested an appointment for a "checkup" to establish ongoing care. Physicians consented to and were aware that they would be visited during the study period, but they did not know when the visit would take place, who the feigning patient would be, or what the standardized patient clinical presentation would be. The consent process was approved by the institutional review board, and all physicians and actresses signed the consent form. Willingness to allow a visit by a standardized patient was a condition for participation in the CPCP Project.

Development of the Standardized Patient Scenario

The standardized patient "scenario" (standardized dialogue) had to be credible to physicians and their staffs, to be delivered consistently and convincingly by nonprofessional actresses, and to invoke predictable physician responses for which we could develop standard patient replies. Department clinical faculty and persons previously trained to be standardized patients assisted in the development of the standardized patient scenario. The scenario was pilot tested in nonparticipating practices by experienced standardized patients and was then modified to increase credibility, consistency, and ease of delivery.

Study Variables

An evaluation checklist was developed to assess components of the physical examination and provision of National Cancer Institute recommendations [25-27]. It was extensively pilot tested to ensure that each item was readily observable by a standardized patient. Five of these are recommended by the U.S. Preventive Services Task Force [33] for 50-year-old women and include blood pressure and weight measurement, clinical breast examination, heart auscultation, neck palpation, and oral cavity inspection. For exploratory purposes, to discover what other components physicians deemed important, we also assessed performance of lung and abdominal auscultation, abdominal palpation, visual screening, funduscopy, otoscopy, and assessment of deep tendon reflexes and vibratory sense. All 14 physical examination items were found to be observable. We were unable to develop criteria that allowed the actresses to determine consistently and confidently that a skin examination was provided. Eight cancer early detection and prevention recommendations were found to be observable, including mammography, smoking cessation counseling, fecal occult blood testing, sigmoidoscopy, nutritional recommendations (increase in fiber, decrease in fat), breast self-examination, and future Papanicolaou tests. The standardized patient scored each of the above items as "yes" or "no" on the score sheet depending on physician actions.

High-fidelity microphones with voice-activated audio recorders were used to record the physician-"patient" interactions. These were concealed in purses specially adapted for this purpose. After an additional review and approval by the institutional review board, participating physicians were asked to sign a specific consent explicitly agreeing to use of these devices.

Standardized Patient Training

Actresses were recruited from the women's auxiliary of local hospitals. During two 4-hour didactic sessions, actresses learned the detailed scenario and the skills necessary to present it. Training addressed how to determine whether various aspects of the history, physical examination, preventive counseling, and cancer early detection recommendations were provided. At the conclusion of didactic training, all the actresses watched a videotape together of a periodic health examination, completed the checklist based on what they saw, and then compared their observations. A complete determination of what actually occurred was then reviewed.

During further training, each actress visited three different nonparticipating clinicians, two in a setting where videotape equipment was available. These clinicians had no specific knowledge of the study and were instructed to respond as they would to a request for a "checkup" from a new patient. Immediately after each videotape session, the actress completed the evaluation checklist. The videotapes were then scored by a trained observer using the same checklist, and any discrepancies in scoring and in keeping to the scenario were discussed. Nine of 11 actresses consistently kept to the scenario and accurately assessed physician performance and, therefore, were allowed to continue the study and enter the field.

Agreement between the observer and the actress on elements of the physical examination was scrutinized with special care because this could not be monitored accurately in the field using only the audiotape. Blood pressure and weight were not addressed in these videotapes. The 9 actresses scored the checklist for the videotaped pilot sessions, as well as the remaining 11 physical examination components provided during each of 2 videotaped physical examinations, which represented 198 items for comparison (9 actresses x 2 visits x 11 physical examination components). For two of these examinations, camera placement did not allow accurate observer assessment of most of the components of the physical examination provided. This left 176 items (198 –22) for comparison. The observer confirmed the actresses' scoring on 151 (86%), was unsure on 21 (12%) usually because of an inadequate view, and disagreed on 4 (2%).

The Standardized Patient Scenario

The actresses portrayed the specific role of a 55-year-old woman who had just moved to the area to be nearer relatives. The scenario was constructed so that the standardized patient was overdue for and receptive to all services recommended by the National Cancer Institute for women ages 50 to 59 years with the exception of Papanicolaou tests, pelvic, and rectal examinations. Services indicated included clinical breast and oral cavity examinations and recommendations to have mammography and sigmoidoscopy, quit smoking, do breast self-examination, return fecal occult blood tests, increase dietary fiber, and decrease dietary fat. None of these services was explicitly requested by the standardized patients except the clinical breast examination, which was requested at the end of the encounter if the physician did not spontaneously provide one.

None of the actresses themselves had a personal history of cancer or of having a breast biopsy. If asked, actresses indicated they had smoked a pack a day for about 20 years, had previously quit twice, and were interested in trying to quit again. Any surgical scars were explained by the actress's actual history. All standardized patients were within 20% of their ideal body weight. If asked, all gave a high-fat, low-fiber dietary history and the same history of a mother with breast cancer and a grandfather with colon cancer.

In calling the office to schedule the appointment and in talking with the physician, all standardized patients volunteered that they had had no health care contact for more than 3 years except for receiving a free Papanicolaou test 6 months earlier at a community clinic. Special community clinics had actually been provided in the region during this period [34]. The standardized patient had no specific symptoms or concerns and stated that she wished to establish care with a primary care physician as part of relocating to the area. All patients gave a post office box as an address and stated that they did not yet have a telephone.

The standardized patient declined a Papanicolaou test and pelvic examination if the physician advised them at that visit but agreed to a future appointment for those purposes. If the physician advised a rectal examination, the standardized patient also declined stating that she recently had a "flare-up" of hemorrhoids that had just improved and that she did not want to risk irritating them. Standardized patients also refused serum cholesterol and other laboratory tests as well as immunizations if offered, explaining that they anticipated getting health insurance shortly and wanted to defer these expenses for now. The exception was urinalysis if it was part of the physician's routine orders before seeing the patient. At the conclusion of the visit, standardized patients paid for services in cash. Receipts for payment were given to the CPCP Project.

Evaluation

Criteria of feasibility included whether standardized patients could visit the practices undetected, maintain the scenario, and accurately record observations. Observations on the periodic health examination were documented on the evaluation checklist that the standardized patient completed immediately after the visit on returning to her car. These audiotapes were then reviewed by a research assistant, and any incorrect responses on the evaluation checklist were tallied then corrected before data entry and analysis. Standardized patients also received immediate feedback on maintaining the scenario from research assistants.

Standardized patients recorded the amount of time the physician and clinical office staff spent with them as well as the cost of the visit. At the conclusion of the study, physicians were asked whether they had been visited by a standardized patient and were asked to describe the patient or provide the patient's name. Then all physicians who had been visited by a standardized patient were told the name of the actress and the date of the visit so they could inactivate the chart.

Statistical Analysis

Exploratory data were assessed using descriptive statistics. Evaluative data on components of the examination, recommendations, charges, and time with the physician were analyzed using the chi-square test, the t-test, and the Pearson product moment correlation coefficient. An level of 0.05 was considered statistically significant. Interobserver reliability was determined using coefficients.

Results

Top Methods Results Discussion Conclusion Author & Article Info References

Sixty-five of the 98 CPCP Project participants were open to new patients. Fifty-nine physicians received a visit from a standardized patient. An appointment with the remaining six physicians could not be scheduled because these physicians needed to change the appointment time on short notice but did not have a telephone number for the patient. The standardized patient subsequently came to the office at the scheduled appointment time, could not be accommodated, and could not be rescheduled soon enough for the Project timetable.

Of the 59 visits made, 2 physicians (3%) identified the standardized patient as being part of the CPCP Project, 1 during the course of the visit and 1 when physicians were queried at the project conclusion (2 different actresses). These were excluded, leaving 57 visits for analysis. Forty (70%) visits were successfully audiotaped. One mechanical failure occurred, and 17 physicians refused audiotaping. The physicians who refused audiotaping were not statistically different from those who consented in terms of age, sex, specialty, or nature of their practice (solo compared with group practice).

As expected, physicians who saw standardized patients were somewhat younger and had been in practice for fewer years than physicians whose practices were closed to new patients. Table 1 outlines the characteristics of study participants and eligible nonparticipants. The mean physician age was 44.2 years (range, 32 to 67 years). General internists constituted one third of the sample; approximately one half practiced alone. The three participating women physicians constituted 5% of the physician sample. Characteristics of these volunteer study physicians differed from nonparticipating but eligible physicians in the region in that nonparticipants were slightly older, were somewhat more likely to be women, were more likely to be general internists, and were more likely to practice alone.

Nine actresses each saw between 3 and 12 physicians. Each actress was audiotaped at least twice. Standardized patients were blind to study group assignment of the physician. In every audiotaped visit, the actress conformed to the standard scenario with no significant deviations from it. Based on the 40 interactions audiotaped, patients completed the evaluation checklist for cancer control recommendations correctly 96% of the time, ranging from 95% for the recommendation about future Papanicolaou tests to 98% for the recommendations about sigmoidoscopy and dietary fiber. The coefficient was 0.91 overall (range, 0.87 to 0.95) for each of the eight recommendations. No statistical differences occurred in average physician charge or components of the examination provided among the nine actresses. No physician acknowledged identifying a genuine patient as being part of the study.

Physician Performance of the Physical Examination

Performance of components of the physical examination is summarized in Figure 1. Those components recommended by the U.S. Preventive Services Task Force for 50-year-old women included blood pressure measurement (which was done for 98% of the standardized patients), weight measurement for 86%, heart auscultation for 83%, neck palpation for 70%, and oral cavity inspection for 61% during the 57 visits. No physician did an oral cavity palpation. Of the examination components recommended by the National Cancer Institute, physicians spontaneously provided a clinical breast examination to 65% of standardized patients and another 20% of physicians did so when specifically asked. The remaining physicians who did not comply with the patient's request indicated that they preferred to examine the woman's breasts when she had her next Papanicolaou test. No statistical differences in physical examination components were associated with study group assignment.

View larger version (22K): [in this window] [in a new window]   Figure 1. Percentage of physicians (n = 57) providing various components of the physical examination.

Provision of Cancer Early Detection and Prevention Recommendations

Provision of specific indicated recommendations ranged from 16% to 89% of the standardized patients, as illustrated in Figure 2. All but four of the standardized patients (93%) were asked if they smoked. Seventy-four percent of these (79% of the 53 who had been asked if they smoked) were advised to quit. Fewer physicians offered assistance with quitting. Of those who determined smoking status, 35% mentioned or prescribed nicotine gum and 23% recommended a smoking cessation class. Whereas 53% were advised to do monthly breast self-examination, 30% were actually instructed in its performance. No statistical differences in these cancer preventive recommendations were associated with study group assignment. Thus, subsequent analyses of intervention variables by time spent and charges were not done.

View larger version (16K): [in this window] [in a new window]   Figure 2. Percentage of physicians (n = 57) providing selected preventive recommendations. Pap = Papanicolaou.

Visit Length and Cost of Visit

The time that physicians spent in the room with the standardized patient averaged 28.7 minutes (SD ± 11.6; range, 5 to 60 minutes). Clinical staff spent an average of 5.9 minutes with the patient (range, 0 to 15 minutes).

Physician charges ranged from $0 to $108. Two physicians provided the consultation at no charge and indicated that this was their usual practice for new patients establishing care. Neither of these physicians identified the standardized patient as an actress. These two physicians spent 30 and 35 minutes, respectively, with the standardized patient. Among the physicians who charged, the average fee was $53.54 (SD ± $20.18; range, $24 to $108). A urinalysis was billed as a separate additional part of the total charge by four physicians. These charges ranged from $5 to $15. The correlation coefficient between charges and the time spent with the physician was 0.33. The correlation coefficient between charges and time spent by office staff was 0.1.

Discussion

Top Methods Results Discussion Conclusion Author & Article Info References

Physicians provided many of the preventive recommendations and components of the physical examination recommended by experts. Blood pressure measurement was almost universally provided, whereas mammography was recommended to almost 90%. Some services were provided less often than the National Cancer Institute recommends. For example, whereas oral cavity inspection and neck palpation were provided to 61% and 69% of these smokers, respectively, no physician palpated the floor of the mouth. In addition, 35% did not examine the breasts unless prompted, more than 20% did not advise smoking cessation, and fewer than one in five gave dietary advice. Various services were provided that experts do not recommend. A few physicians tested visual acuity and vibratory sense, and about one half did funduscopy and otoscopy. Clearly, physicians offer no consistent response to a uniform request for a checkup.

Office System Intervention

Why did the office system intervention have no detectable impact as determined by the standardized patient, although it improved physician services as determined by patient surveys and record reviews? The office system intervention was directed principally at opportunistic cancer early detection and preventive counseling, that is services provided during routine acute and chronic care, not during periodic health examinations such as these in which physicians might be expected to provide needed services without special assistance from an office system. For some items, such as mammography recommendations and smoking cessation advice, a ceiling effect may also have been present due to high rates of performance before interventions, as shown in control practices. A larger sample size may have assisted in providing enough power to detect differences in this area of physician performance.

We were surprised that more physicians, especially those who determined that the standardized patients were smokers, did not advise smoking cessation or offer to assist with cessation. After completion of the study, we asked those physicians to explain. Although they could not recall the specific patient interaction, the most common response about general practices was that a new patient visit already had a full agenda and that if the patient planned to establish ongoing care, they would defer difficult-to-follow advice such as smoking cessation to a later visit when a better relationship had developed.

Did physicians misconstrue the standardized patient's intent for the visit because preventive care or a periodic health examination was not specifically requested? Possibly, but patients using those explicit terms would have been unusual in practice, and use of them would have increased the physician's suspicions. Some physicians may have planned to provide indicated services over time, such as the physicians who did not counsel smokers to quit. However, the patients were well overdue for all services except Papanicolaou tests and pelvic examinations. Whenever the physician did recommend a specific next appointment, it was for the Papanicolaou test at least 6 months in the future. We believe the variability in physician performance of the periodic health examination is genuine and not an artifact of the methodology.

Variations in Cost and Length of Visit

Of particular interest are the variations in cost and in the amount of time that physicians spend in response to the same patient request, as well as the limited correlation between the two. Some variation in time spent was expected. The vicissitudes of office practice include running behind due to fitting extra patients into the schedule, needing more time with the previous patients, and getting back from the hospital late. Having extra time can occur after unexpected cancellations. However, we did not anticipate that the time physicians would spend in response to the same request would vary by a factor of 12 (5 to 60 minutes). The reasons for this degree of variation are beyond the scope of this study to determine, but we speculate that physicians have different agendas in responding to the request for a physical examination. Some physicians appear to use this opportunity to become thoroughly acquainted with the patient, whereas others use it as a more limited introduction.

Variations in cost were also expected. No doubt, some physicians see initial visits as important in attracting new patients and in building their loyalty. The two physicians who provided visits at no cost may be examples of this. However, among physicians who charged, the fee varied by a factor of four. This may derive from the different economic agendas of the physicians. Some view the physical examination as a marketing opportunity and provide care free or with a subsidy, whereas other physicians charge what they feel is the full value for their time and may even use the physical examination to subsidize other aspects of care.

These variations are consistent with variations found in other physician practices. Henke and Epstein [35] found large differences among rheumatologists in the visit length, monitoring procedures, and use of the laboratory for patients with rheumatoid arthritis. Wennberg [2] found variations in the rates of specific surgical procedures across small geographical areas, with rates for appendectomy and other procedures with well-defined indications showing little variation, whereas rates for hysterectomy and procedures with less well-defined indications vary widely. Perhaps the components of the periodic health examination provided in response to a new patient's request for a checkup need to be added to the list of procedures that have wide variation.

This study shows that it is feasible to use standardized patients to assess physicians' responses to the new patient request for a periodic health examination. The scenario proved convincing and training was effective in that only two of these nonprofessional actresses were detected. No false-positive identifications occurred. Standardized patients presented the scenario consistently and obtained accurate observations on what physicians recommended as verified by audiotapes and on the physical examination provided as verified by videotapes during pilot testing.

Limitations

Some limitations of this study should be noted. The study population included physicians who volunteered to participate in a study on cancer early detection and prevention, who probably are more interested in this area, who probably provide more services, and who probably spend more time than physicians who did not volunteer. Therefore, these findings may not be representative of other physicians in the region. Similarly, this study concerns only new patients and does not address the services provided to established asymptomatic patients when they request a checkup or when the physician advises one. Furthermore, day-to-day variations in the response of individual physicians to the same patient request are not addressed. Some physicians acknowledged to the patient that they "had to hurry" with the visit because of the need to attend a hospitalized patient. Finally, only women claiming to be 55 years old served as patients. How physicians respond to men or to women of different ages was not addressed.

Conclusion

Top Methods Results Discussion Conclusion Author & Article Info References

A standardized patient can be a feasible evaluation tool to assess physician performance of the periodic health examination. Our findings suggest that physician responses to an identical request for a "checkup" vary widely with respect to the cost, the services provided, and the time spent with the physician. These variations may be even larger among more representative physicians.

This study should be repeated with a more generalizable physician population to assess the implications for health care costs as well as prevention and early detection care. For the present, we believe that patients need to state their expectations about the specific contents of the checkup they desire or make this determination during a discussion with their physicians. Physicians should reflect on what they plan to accomplish with a new patient periodic health examination in terms of health care for the patient and the economics of their practices. Health care providers should ensure that their concept of a routine checkup includes what is currently recommended by the appropriate agencies and groups formulating guidelines. As with other parts of medical care, patient requests for a routine checkup should not be taken for granted by physicians, their staffs, or by health services researchers.