Inter-rater Reliability Report for Rivet Education’s Professional Learning Partner Guide (PLPG) Rubric

Rivet Education developed a Professional Learning Partner Guide (PLPG), a searchable database of learning providers with expertise in the adoption and implementation of High-Quality Instructional Materials (HQIM), in response to the lack of resources and guidance available to support districts with evaluation of professional learning (PL) services. To be selected as a PL provider (PLP) in the PLPG, PL organizations have to submit an application that is reviewed and scored by Rivet Education using a PLPG Rubric (formerly known as the Scoring and Evidence Guide or SEG).

To ensure confidence in the PLPG as a robust database of high quality PL services, the PLPG Rubric needs to be evaluated as a reliable and valid assessment instrument. The National Implementation Research Network (NIRN) is conducting psychometric testing of the PLPG Rubric. PLPG Rubric’s content validity was established in Phase I (Spring 2024). This report summarizes the research on testing inter-rater reliability or the degree to which reviewers (or raters) are consistent with each other when rating the same application. Inter-rater reliability was assessed across all 3 Gateways of the PLPG, and across subtypes of Gateways 1 and 2, for a total of 7 Gateway categories.

Overall, results for inter-rater reliability varied by the Gateway category of the PLPG Rubric. Recommendations were made to refine the instrument and reviewer training, as well as to retain the use of consensus calls between reviewers. Rubric modifications and training revisions were made by Rivet Education. The next phase will assess structural validity.