Relying on the idea that in order to understand what's happening, I need to be part of what's happening, I signed on to do a (paid) assessment gig for NWU.
We're working on assessing one aspect of an assignment for a course that has four assignments in common between all the sections. So, for example, if this were in intro to underwater basketweaving course, we'd look at a basket the student has produced, and assign a score for each feature, basic basket shape, regularity of weave, and size of weave. We'd assign a score of blue for any feature that is better than our expectations (based on criteria for each feature), green for any feature that meets our expectations, yellow for any feature that shows progress towards expectations, but doesn't meet them, and red for any feature that shows no progress (they turned in a rock, for example).
In this hypothetical world, the underwater basketweaving faculty have agreed that all sections of intro to underwater basketweaving will include several common assignments, including a basket starting assignment, a patterning assignment, a basic cup-shaped basket, and a non-cup-shaped basket. We're doing the cup-shaped basket.
The thing is, it looks like while all the sections do assign a cup shaped basket, some of them seem to require that students use at least two different weaving techniques, which makes it really hard to judge the "regularity of weave" section. Other sections seem to all think cup shapes are tall and skinny, or more shallow bowl, so the basket shape is hard. And still others let students use different materials, meaning that the ideal size of the weave changes a lot.
We got together in a small group and did a half day of "norming" using four basket examples. ("Norming," for those unfamiliar with the term, means working together, in our case with rubrics, to come to a basic level of agreement about the scoring of an assignment.) We used a rubric developed by the faculty earlier, and found some pretty basic difficulties in scoring baskets where, for example, a student had made a cup-shaped basket, and seemingly tried to incorporate several different weaving techniques. It was an ambitious basket, but it scored way down on regularity of weave because the person wasn't using anything like a regular patterning, and the sizes were pretty all over the place. So it got a low score. If I'd been grading it, I would have given it a pretty good grade because it took risks and showed real learning.
One of the problems with assessment done this way is that there's no room for a "no, but" answer. There's no room for a "no, the weaving isn't regular, but wow, there's something cool happening here."
That's not to say the rubrics are bad, because the UB faculty clearly values basic shape, regularity of weaving, and weave size (the tightness matters, I suppose). So those are important things to make sure they're teaching. And so that's what they're trying to measure in their assessment.
What they aren't measuring is how a given student is taking a risk, trying something different, reaching out, finding their own path. I don't know that those things can be measured, or how, but I know they're important aspects of student learning, probably more important than the regularity of weaving or the size of the weave.
This is always my struggle with rubrics, and the reason I don't use them (and fortunately I have the freedom to make that decision). We all know that something can be greater than the sum of its parts, and sometimes I'd rather see a student create a big mess because she was trying something way beyond her ability than to see her do a competent, safe job. But I've never yet seen a rubric that could handle that kind of thinking.
ReplyDeleteThat's a good critique of the current process but not an insurmountable problem. Perhaps a "difficulty factor" could be used as a multiplier - for part or all of the project. So a student attempts and only partly succeeds at a more difficult weaving technique or employs particularly challenging reeds? Use a multiplier (from 1.1 up to 1.5, say) for different levels or have a rubric that advises something of this sort "students will be expected to demonstrate, at minimum, mastery of two weaving techniques. Students who attempt more difficult techniques such as A, B and C may be rewarded for their engagement with the more challenging materials."
ReplyDeleteOr even just an additional "difficulty" slot in the rubric (potentially extra credit).
DeleteI just finished running our programmatic assessment. One of the things we did that I was oh, so thankful for, was that in addition to the rubric categories, we also did a "holistic" score, and had a space for notes. It meant more work for me as I went through the notes and had to figure out what was causing holistic scores to be different (either higher or lower) than the individual category scores, but we got *much* more useful data out of it that way.
ReplyDelete