A Methodology for Developing and Validating an Assessment Rubric

The techniques in this study are commonly expected in the process of developing a legally defensible certification exam. The presentation, however, focused on exceptionally rigorous application of these techniques for assessment in a context of teaching.

With the goal of improving a rubric's validity and reliability, the process in this study began with development of learning objectives and a review of related professional guidelines and literature before engaging in academic discussions about the constructs in the rubric. Representatives from the academic area and from the profession were engaged in discussions about criteria and weights for the rubric. A process of consensus moderation known as the Delphi technique was used to reach agreement among review groups, thereby reducing variance in scoring.

Among the reported results, the research team reported the observation that there was more agreement in using the rubric with an "A" paper, but below that level, variance becomes greater as teachers consider how to give feedback without discouraging the lower-performing students. The questions after this observation: What is fair? Don't students deserve equitable evaluation? Or do we want to encourage a system in which students "section shop?"

The rubric in this study was a "News Release Evaluation Rubric" had a left column with five major constructs, each with a checklist of elements, and a right column with a % weight for each major construct.

Presenters: Sandra Allen, Columbia College Chicago and John "Ed" Knight, University of Tennessee at Martin.