Site Index | CDE Offices | Staff Directory

Elevate Colorado (Inter-rater Agreement)

To support fair and consistent evaluations, CDE is developing tools to promote common interpretations of teacher quality and help evaluators provide useful and actionable feedback to educators. One such tool is Elevate Colorado, an online inter-rater agreement training system, being developed in partnership with My Learning Plan. This online system helps evaluators develop a deeper understanding of the professional practices embodied the Colorado State Model Evaluation System's teacher rubric. Using the system, evaluators can view short videos of practicing teachers, rate the videos according to the Colorado State Model Evaluation System rubric and then receive feedback showing how closely they rated the videos relative to ratings from master scorers.

Another tool to support fair and consistent evaluations is The Resource Guide for Deepening the Understanding of Teacher's Professional Practices. The resource guide is meant to support Colorado school district employees, administrators and teachers in developing an understanding of the Colorado State Model Evaluation System Teacher Rubric. As individuals apply the rubric to their own teaching and to that of others, the resource guide can support observers and coaches in accurately identifying evidence for the professional practices. The guide also supports classroom teachers to accurately reflect on their teaching and plan for implementation of specific practices in their instruction. The research, examples provided and the glossary can be used to support the development of a common language for Colorado school district employees to analyze, reflect and plan instruction.

 *This is a living document and because it will be updated periodically and eventually be an interactive online format, we want to encourage your feedback and submission of examples and resources for the continued development of this resource.  Please contact Dawn Paré at pare_d@cde.state.co.us with your feedback, questions or additional resources.

Features of Elevate Colorado

Elevate Colorado was released in August 2013 with 6 videos to allow educators to begin to come to a common understanding of what the Colorado State Model Evaluation system teacher professional practices look like in a variety of classrooms. Throughout the 2013-14 school year, additional videos were added and this summer last year’s videos were updated to reflect the revisions to the 2014-15 teacher rubric.

  • For the 2014-15 school year, 4 training videos for each level (elementary, middle, and high school) are available to give educators the opportunity to quickly gauge their understanding of the professional practices in the model system rubric.  Each video has been aligned to the professional practices by a group of teachers and administrators from around the state who have been trained as master scorers.
  • When an educator completes their evidence in a video, immediate feedback is provided.
  • Additional calibration videos are being developed for school districts and organizations to use as assessment of evaluator learning.

Process: Administrators and teachers from around the state applied to be Master Scorers. The selected Master Scorers were trained on the Elevate Colorado system and best practices in observation. The Master Scorers first watched video clips individually and aligned them to the professional practices. CDE then held a two and a half-day session where the Master Scorers calibrated their thinking by reviewing the collated evidence and coming to agreement on the final evidence and scores based on the quality and explicitness of the practice. 


There may be multiple examples of the same professional practice within a video, but in most cases we limited the alignment to one-two examples of each. For example, students may be engaged and working in groups throughout a lesson segment, but we tried to align it only once or twice during the video clip. In addition, Standard II, Classroom Environment provided a special challenge for Master Scorers. Many of these classroom examples did not have explicit examples of a teacher addressing classroom culture or student behaviors because students were on task and performing as expected. We decided that because the goal was to show explicit examples of the teacher’s practice that we would not align professional practices unless it was evident that the teacher was addressing it in that moment. In actual practice, this would be a conversation with the teacher later about how the teacher established the practices and routines that encourage positive culture and appropriate student behavior.

Uses of Elevate Colorado

For Individual Professional Development

Any Colorado educator can use this system to view video clips of lessons. Select an elementary, middle school or high school video and practice aligning the professional practices and evidence. When finished, click on the Evidence Comparison tab to compare your results to that of the Master Scorers. Other videos are provided at each level for additional practice.

 

For Staff Development

Because this is an online system, staff can view videos and align evidence prior to coming together. Once together, review the results, refer back to evidence in the video by clicking on the film icon, and calibrate based on the master scores. This practice allows for rich discussion that can uncover inconsistencies and/or bias in observation as well as support staff in coming to consensus on what is high quality implementation of the Teacher Quality Standards. 

In addition, these video clips can be used to generate discussion about the type of coaching conversations and support that could be offered to the teacher. 

For Evaluator Training

Training for evaluators can incorporate each of the above recommendations.However, we know that this is the very first step in “norming” or “calibrating,” so during the 2014-15 school year, CDE will begin testing “teaching cases” that will include a variety of 10-15 minute video clips of a teacher over time along with artifacts that capture the broader body of evidence to support the calculation of an overall rating for professional practices.

In addition, CDE plans to designate two videos at each level as “calibration plans.” These can be used by organizations to assess the learning of their evaluators. We know that with more videos and practice, evaluators will refine their skills in aligning evidence to the rubric for a more accurate and fair assessment of teacher performance.

How do you know if you are meeting a level of proficiency in identifying professional practices?

Current research* indicates that a minimum of 75 percent alignment is recommended. As you align evidence at the professional practices level to each video clip, you will be able to compare your response to that of the Master Scorers through the Evidence Comparison tab. Remember that Master Scorers sought to include only explicit, quality evidence for each practice. In other words, try not to infer what is happening, but what is visible in teacher and student actions.

Keep in mind that we are not yet using this system to determine an overall professional practice rating. These 10-15 minute video clips would not provide sufficient evidence because of the ongoing nature of evaluation process over a full school year.

* Measuring and Promoting Inter-rater Agreement of Teacher and Principal Performance Ratings, Center for Educator Compensation Reform. Graham, Milanowski, and Miller, Feb. 2012, p. 9-12.

Ready to Begin?

Visit the Elevate Colorado website to get started. For detailed instructions on how to create an account and use the system, make sure to download this reader-friendly technical user's guide.  

 

Share your skills and expertise with educators across Colorado to support inter-rater agreement

The Educator Effectiveness Unit is partnering with Colorado educators to create a library of professional practice that will be made available in Elevate Colorado. The library will deepen Colorado educators’ understanding of the Teacher Quality Standards and will support coaches and evaluators in identifying quality teaching practices.  To make this a robust system with the goals and values of Colorado educators, CDE would like to film interested teachers’ lessons and reflections, and collect supporting artifacts that will then be aligned to the professional practices by experienced Master Scorers.  If you are interested in being filmed or learning more, please contact Dawn Pare at pare_d@cde.state.co.us

Quick Links

 

Contact Us

Colorado Dept. of Education
201 East Colfax Ave.
Denver, CO 80203
Phone: 303-866-6600
Fax: 303-830-0793
Contact CDE

CDE Hours
Mon - Fri 8 a.m. to 5 p.m.
See also: Licensing Hours

Educator Licensing