NIRN Discussion: Measuring, Assessing and Improving Implementation

Date Published: 
10/18/2012

by Allison Metz


What types of assessments are you and your organization using to assess the strengths and gaps of the implementation infrastructure? There is growing recognition in the health and human services field of the importance of effective implementation and the need for implementation research that can guide selection, development, evaluation, and ongoing improvement of interventions.  Allison Metz from NIRN discusses the Initial Implementation Assessment tool. Your comments and ideas are invited.


The Initial Implementation Assessment developed by the National Implementation Research Network (2012) assesses the strengths and gaps of “Implementation Drivers” which are the core components or building blocks of the implementation infrastructure. These components are needed to support practice, organizational, and systems change.  The Implementation Drivers include: Competency Drivers, Organization Drivers, and Leadership Drivers (see diagram).

The assessment is designed build the capacity of Implementation Teams to develop and install action plans for ongoing improvement of the implementation infrastructure for any intervention.  The Initial Implementation Assessment was administered at baseline and follow-up periods during a five year initiative to develop, implement and improve a post-care service delivery system for children exiting foster care to permanency – the Catawba County Child Wellbeing Project.  The service system consisted of selected interventions ranging from evidence-based programs to evidence-informed practice models that required intensive program development by the public child welfare agency. Implementation Teams served as accountable structures for promoting high-fidelity implementation of all interventions. In some cases the Implementation Teams were accountable for building the infrastructure from scratch, while in other cases the Teams were accountable for filling in the infrastructure gaps of national purveyors of evidence-based models or working with purveyors to shift the locus of responsibility of some drivers from the purveyor to the public agency over time. The assessment was used for all interventions.

Building the capacity of Implementation Teams to shift responsibility of certain drivers from the purveyor to the public agency without compromising program fidelity is important for sustaining evidence-based models.

+ More About the Tool
The Initial Implementation Assessment was administered at baseline and follow-up periods during a five-year initiative to develop, implement and improve a post-care service delivery system for children exiting foster care to permanency (the Catawba County Child Wellbeing Project).  The service system consisted of selected interventions ranging from evidence-based programs to evidence-informed practice models that required intensive program development by the public child welfare agency. Implementation Teams served as accountable structures for promoting high-fidelity implementation of all interventions. In some cases the Implementation Teams were accountable for building the infrastructure from scratch, while in other cases the Teams were accountable for filling in the infrastructure gaps of national purveyors of evidence-based models or working with purveyors to shift the locus of responsibility of some drivers from the purveyor to the public agency over time. The assessment was used for all interventions.
+ Major Findings

Strengthening the implementation infrastructure (i.e., implementation drivers) improves fidelity scores

  • Conducting a stage-based formal assessment of the implementation drivers yields important findings for purposeful action planning by Implementation Teams  whether a purveyor is formally involved or not
  • Building the capacity of Implementation Teams to address infrastructure gaps not installed by purveyors is important for achieving fidelity

 

Community Questions

  • What types of assessments are you and your organization using to assess the strengths and gaps of the implementation infrastructure?
  • How can results from assessments like this Initial Implementation Assessment tool help Implementation Teams?
  • What types of data collection are important to consider before moving to the next stage of implementation?

Comments

Implementation support tools

Allison recently spent an intensive week working with us here at the Parenting Research Centre in Australia, to help us refine our approach to implementation support, and to the assessment of implementation infrastructure, gaps and planning. Along with other tools generously shared by Allison, the Initial Implementation Assessment Tool will help us to improve our work with family support and child welfare agencies by providing us with more accurate assessments of baseline supports, which will subsequently assist in the development of action plans for implementation work.
Thanks again Allison!

Implementation Tools and Measures

The tools you mention and your presentation about them are great. They are a terrific addition to the field and your willingness to share them and engage us in their use - from multiple perspectives, is truly appreciated. I look forward to hearing more about them as they develop and hope to be able to contribute to their refinement as I get the opportunity to use them.

NIRN Assessments

Love the way the measures are tied to the "Implementation Drivers" and scaled as "in place", "partially in place" or "not in place". Thanks for the contribution to the field. Such a big leap from management tools like SWOT analysis and environmental analysis in the past.

Implementation Measure

Allison- So glad to see this up and running. Given the near absence of data on implementation variables, I am anxious to "hear" the conversation you are beginning with the field.