Content Validity Report for Rivet Education’s Scoring and Evidence Guide (SEG)
Author(s): Melanie Livet, Ph.D. Caryn S. Ward, Ph.D. Amelia Krysinski
Publication Date: May 2024
Rivet Education developed a Professional Learning Partner Guide (PLPG), a searchable database of learning providers with expertise in the adoption and implementation of High-Quality Instructional Materials (HQIM), in response to the lack of resources and guidance available to support districts with evaluation of professional learning (PL) services. To be selected as a PL provider (PLP) in the PLPG, PL organizations have to submit an application that is reviewed and scored by Rivet Education using a Scoring and Evidence Guide (SEG). To ensure confidence in the PLPG as a robust database of high-quality PL services, the SEG needs to be evaluated as a reliable and valid assessment instrument. The National Implementation Research Network (NIRN) is conducting psychometric testing of the SEG, with this first phase focusing on establishing content validity.
Overall, the results supported the content validity of the SEG, with high scores on the Content Validity Index (CVI) and Interrater Agreement (IRA) indices for clarity and relevance of gateways, constructs, and indicators (except for the Gateway 2 System Design and Leadership Support clarity CVI).
The SEG was representative of the concepts it was designed to measure, including levels of content and HQIM expertise, quality of professional learning by type (i.e., adoption, launch, ongoing PL for teachers, and Systems Design and Leadership Support), and use of data for planning and improvement. Recommendations for refining the instrument were provided, with modifications being made by Rivet Education to produce Version 3.0 of the SEG. The next phase will be focused on establishing the psychometric properties of the SEG, by assessing inter-rater reliability and structural validity.
File(s):