Relying on the idea that in order to understand what's happening, I need to be part of what's happening, I signed on to do a (paid) assessment gig for NWU.
We're working on assessing one aspect of an assignment for a course that has four assignments in common between all the sections. So, for example, if this were in intro to underwater basketweaving course, we'd look at a basket the student has produced, and assign a score for each feature, basic basket shape, regularity of weave, and size of weave. We'd assign a score of blue for any feature that is better than our expectations (based on criteria for each feature), green for any feature that meets our expectations, yellow for any feature that shows progress towards expectations, but doesn't meet them, and red for any feature that shows no progress (they turned in a rock, for example).
In this hypothetical world, the underwater basketweaving faculty have agreed that all sections of intro to underwater basketweaving will include several common assignments, including a basket starting assignment, a patterning assignment, a basic cup-shaped basket, and a non-cup-shaped basket. We're doing the cup-shaped basket.
The thing is, it looks like while all the sections do assign a cup shaped basket, some of them seem to require that students use at least two different weaving techniques, which makes it really hard to judge the "regularity of weave" section. Other sections seem to all think cup shapes are tall and skinny, or more shallow bowl, so the basket shape is hard. And still others let students use different materials, meaning that the ideal size of the weave changes a lot.
We got together in a small group and did a half day of "norming" using four basket examples. ("Norming," for those unfamiliar with the term, means working together, in our case with rubrics, to come to a basic level of agreement about the scoring of an assignment.) We used a rubric developed by the faculty earlier, and found some pretty basic difficulties in scoring baskets where, for example, a student had made a cup-shaped basket, and seemingly tried to incorporate several different weaving techniques. It was an ambitious basket, but it scored way down on regularity of weave because the person wasn't using anything like a regular patterning, and the sizes were pretty all over the place. So it got a low score. If I'd been grading it, I would have given it a pretty good grade because it took risks and showed real learning.
One of the problems with assessment done this way is that there's no room for a "no, but" answer. There's no room for a "no, the weaving isn't regular, but wow, there's something cool happening here."
That's not to say the rubrics are bad, because the UB faculty clearly values basic shape, regularity of weaving, and weave size (the tightness matters, I suppose). So those are important things to make sure they're teaching. And so that's what they're trying to measure in their assessment.
What they aren't measuring is how a given student is taking a risk, trying something different, reaching out, finding their own path. I don't know that those things can be measured, or how, but I know they're important aspects of student learning, probably more important than the regularity of weaving or the size of the weave.