Research on Implementation Stages

Research on the stages on implementation is rare, especially research that evaluates the relative contributions of implementation factors across stages. In one well-designed study, McCormick, Steckler, & McLeroy (1995) randomly assigned school districts to experimental or control conditions. All districts were provided with a choice of middle school tobacco prevention curricula. Health education teachers and administrators in the experimental school districts also received in-depth training on the curriculum. In their analysis of the results, the authors found that smaller school districts (smaller numbers of teachers, less bureaucratic administrations) were more likely to decide to adopt a curriculum at the conclusion of the exploration stage. However, during the initial implementation stage, larger school districts (more resources, greater flexibility) were more likely to implement more of the curriculum. A positive organizational climate (job satisfaction, perceived risk taking, managing conflict, involvement in decision making) was associated with both the adoption decision and with the extent of implementation. However, they found no carry-over effects. That is, events measured during the exploration stage did not affect outcomes during the implementation stage.

Implementation Variables: Complex Interactive Factors

The complexity of implementation variables was captured by Panzano et al. (2004). These researchers conducted an evaluation of 91 behavioral healthcare organizations that adopted or considered adopting one or more evidence-based practices and programs: Cluster-Based Planning (a consumer classification scheme; N=23); Multisystemic Therapy (home-based treatment; N=16); Ohio Medication Algorithms Project (medication management for persons with serious mental illness; N=15); and Integrated Dual Disorder Treatment (for consumers with mental illness and substance abuse problems; N=37). A range of interview, survey, and implementation outcome data were included in a longitudinal design that allowed the authors to relate the data from earlier stages to later stages. The authors tested the usefulness of four conceptual models: Model 1: the adoption decision; Model 2: multi-level model of implementation success; Model 3: cross-phase effects on implementation outcomes; Model 4: effects of implementation variables on outcomes over time.

The results indicated that out of the original 91 organizations, 50 decided to adopt an evidence-based practice or program. During the exploration stage (Model 1), perceived risk discriminated the adopters from the non-adopters. Risk was seen as lower when the organization staff members felt they could manage the risks involved in implementation, when management support was high, and resources were dedicated specifically to implementation. Thus, during the exploration stage, perceived risk discriminated the adopters from the non-adopters.

During the initial implementation stage (Model 2), positive consumer outcomes were positively related to fidelity (conversely, “reinvention” was associated with poorer outcomes) and positively related to assimilation into the agency (making the new program a permanent part of ongoing operations). At the next level of analysis, assimilation was related to quality of the communication between the purveyor and the organization, the extent to which the organization was seen as having a learning culture and a centralized decision making structure, the availability of dedicated resources, and the extent to which implementation was seen as relatively easy and as compatible with the organization’s treatment philosophy. Overall implementation effectiveness was positively related to having a system in place for monitoring implementation progress, access to technical assistance, the perceived ability of the organization to manage risks, and belief in the scientific evidence in support of the program.

Overall implementation effectiveness was negatively related to the extent to which the program had been modified from its prescribed form. Thus, during the initial implementation stage, implementation success was associated with a range of contextual, organizational, and purveyor variables and with fidelity to the evidence-based practice or program as described.

In the next two analyses, Panzano et al. examined the relationships among variables across stages. They found that later assimilation and positive implementation outcomes were higher when, during the exploration stage, the advantages of the program were seen as outweighing the disadvantages, staff had high expectations of the benefits of the program for consumers, the organization staff felt they had a good relationship with the purveyor, and the outcomes of implementation were demonstrable. In addition, objective decision making strategies that involved staff members, good information about the intervention, and organizational leadership support during the exploration stage were positively related to assimilation during the implementation stage. Thus, the methods used to consider adoption appear to have an impact on the later success of implementation (Model 3).

Another interesting analysis indicated that proximal factors exerted greater influence on current outcomes (Model 4), a conclusion similar to the one reached by McCormick et al., (1995). Top management support and access to dedicated resources during the exploration stage were important to the adoption decision but were not related to later implementation outcomes. However, top management support and access to dedicated resources during the initial implementation stage were directly related to implementation outcomes. Similarly, access to technical assistance during the exploration stage was related to 3 of the 7 later implementation outcomes while access to technical assistance (versions of Implementation Teams) during the initial implementation stage was related to all 7 outcomes. Thus, implementation seems to require a sustained effort in order to produce desired outcomes.