Experimental Analyses of Implementation Strategies

Of the 743 citations that resulted from the review of the implementation evaluation literature (NIRN: 2005), 20 were identified as experimental studies that employed within subject or randomized group designs and 2 were identified as meta-analyses of experimental studies.

Experimental Research: Ineffective Implementation Strategies

Some of the most rigorous research provides evidence of what does not work.  Perhaps ineffective implementation strategies are evaluated more often because that is what is done more often in practice.  As the field matures and Implementation Teams make use of Active Implementation Stages, Drivers, Improvement Cycles there should be more data on what does work.

As an implementation strategy, access to information alone appears to have little impact on practitioners’ performance. Azocar, Cuffel, Goldman, & McCulloch (2001) and Azocar, Cuffel, Goldman, & McCarter (2003) randomly assigned clinicians in a managed care organization to one of three groups: a general dissemination group (single mass mailing of best-practice guidelines), a targeted dissemination group that received guidelines with a letter targeting a specific patient, and a control group that was not mailed guidelines. This research demonstrated that dissemination of evidence-based treatment guidelines was not effective in influencing the behavior of mental health clinicians, even in the context of a managed behavioral health organization. Four months after mailing the guidelines, only 64% of clinicians reported receiving guidelines and less than half of them reported reading the quick reference sheet or the 8-page reference booklet. In addition, there was no difference in guideline-consistent practices between clinicians who received the general mailing and those who did not receive the guidelines.

A similar result was found by Fine et al., (2003). Physicians in the experimental and control groups each received mailed information regarding an evidence-based guideline for use with patients with pneumonia. The guideline was designed to change practices to reduce the duration of intravenous antibiotic therapy and length of hospital stay. Information alone had no effect on the clinical practice of the control group. Physicians in the experimental group received the information and had the support of specially trained nurses who made patient assessments, informed the physician when the patient met the guideline criteria, placed prompt sheets in the patient’s file, and offered to take an order for antibiotics and arrange for nursing home care. Physicians in the experimental group prescribed antibiotics significantly more often but there was no change in length of hospital stay.

Schectman, Schroth, Verme, & Voss (2003) conducted an assessment of clinician adherence to acute and low back pain guidelines. Clinicians were randomly assigned to one of four groups: no intervention, physician education and feedback on usage, patient education materials, or a group that combined physician education and feedback on usage and patient education materials. No effect was found for the first three groups. A modest effect was found for the group that combined physician education, feedback on guideline usage, and patient education materials (guideline usage increased by 5.4% as opposed to the control group who decreased guideline usage by 2.7%).

Schofield, Edwards, & Pearce (1997) randomly assigned primary and secondary schools to two groups. Group 1 received mailed education materials and information on the SunSmart skin program in Australia. Group 2 received the mailed information and a staff development module for preparing staff and changing school policies to reduce sun exposure and eventual skin cancer. The results indicated that Group 2 schools adopted sun protection policies at a rate twice that of Group 1 schools (i.e., paper implementation as described in Chapter 1). However, there were no differences in the sun protection practices in either group of schools. Ellis et al. (2003) conducted a thorough review of the experimental literature regarding cancer control interventions. They concluded that passive approaches (diffusion) such as mailings and educational presentations were ineffective.

Taken together, these experimental studies indicate that dissemination of information does not result in positive implementation outcomes (changes in practitioner behavior) or intervention outcomes (benefits to consumers).

Experimental Research: Effective Implementation Strategies

A high level of involvement by program developers on a continuing basis is a feature of many successful implementation programs. In their classic study, Fairweather et al., (1974) randomly assigned hospitals who had agreed to develop lodges to one of two groups. Group 1 received printed materials and a manual. Group 2 received printed materials, a manual, and face-to-face consultation. All received telephone consultation and had free access to making calls to consultants any time. There was significantly greater implementation of the lodge model in Group 2. On-site face-to-face time with staff, managers, and directors provided opportunities to help explain the lodge model and to resolve the structural and policy issues associated with the implementation process.

Wells, Sherbourne et al., (2000) matched (on several dimensions) primary care clinics in 6 managed care organizations. They then randomly assigned one of each matched trio to usual care (mailing of practice guidelines) or to 1 of 2 quality improvement (QI) programs that involved institutional commitment to QI, training local experts and nurse specialists to provide clinician and patient education, identification of a pool of potentially depressed patients, and either nurses for medication follow-up (QI-meds) or access to trained psychotherapists (QI-therapy). The managed care organizations did not mandate following the guidelines for treating depression. Over the course of a year, the QI programs resulted in significant improvements in quality of care, mental health outcomes, and retention of employment for depressed patients without any increase in the number of medical visits.

The value of these multilevel approaches to implementation was confirmed in a meta-analysis of cancer control program implementation strategies (Ellis et al., 2003). They found 31 studies of cancer program implementation factors and concluded that active approaches to implementation were more likely to be effective in combination.

While it is encouraging to see some examples of experimental research on implementation strategies, the few examples pale in comparison to the need for clear and effective strategies to move science to service and transform human service systems nationally.