Sunday, April 17, 2016

Assessment - Vent

If you're in education, you know that we've been working in a variety of ways to "assess" student learning for something like two decades now.  Before that, we graded students.  It wasn't perfect, of course, but it was there.  We assumed that a student who got an A on an exam had demonstrated that they'd learned the material being tested (and hoped that exam grades weren't primarily a matter of being good at bubble tests or whatever).

But, assessment folks have told us forever that grades aren't the same as assessment, but that assessment has to be "outcome driven" and such.  And the assessment people told us that what's important is that we all use the information to tell us how we're doing as teachers, and to help us improve.

And we needed to assess programs and not just how students do in individual courses.  So we couldn't assume that a straight A math student was succeeding at learning math.  We had to set goals, and then figure out outcomes that would reflect those goals, and then figure out a product that would demonstrate the outcomes that would reflect the goals.

Let me tell you about some of the history of our Underwater Basketweaving Program's assessment adventure.

In Underwater Basketweaving, they set some program goals.  For example, they want students to understand the structure and context of underwater-woven baskets. 

Some years ago, the idea was that they'd look at the baskets the students wove underwater and the papers they wrote about baskets woven underwater in a sort of portfolio.  But then someone would have to collect the portfolios, and someone would have to read the portfolios, and there was no money to pay someone to do this, since the expensive assessment guru was too busy making forms for departments to fill out to actually do any assessment himself.

So portfolio assessment went away, and the department struggled for several years to meet the ever-moving targets and to do the paperwork the assessment guru set for them.

Finally, the assessment guru said that they had to do a grid for their department goals, and even though they started with 25 goals for their students, they quickly realized that they couldn't fill out paperwork for 25 goals, not without making paperwork the primary job rather than teaching and research and weaving underwater.

So in line with the assessment guru's rules, they had to make an outcome to go with the main goal, which was that all students should be able to describe the structure and context of an underwater-woven basket, since you have to be able to measure understanding in some way other than "student nods knowingly." 

And then they decided that in one of the early courses in the major, students would do an assignment where they'd describe the structure and context of an underwater-woven basket.

So the guru suggested they make a grid and turn in the numbers this way:  for each goal, they'd have three possibilities: the student exceeded the expectations of the outcome, the student met the expectations, or the student didn't meet the expectations.  And then someone would look at the assignments and fill in a separate grid piece for each of the outcomes.  And to make it easy, they'd rotate the outcomes, one a year, so that someone would only have to fill in one of the grids after reading all the papers for these sections.

But again, there was no money to pay people to look at the assignments for assessment.

So the assessment guru said that each faculty member should be responsible for filling out the grid for their section.

And the faculty met, and noticed that they were already grading these assignments, so how about if they found a way to translate grades into the assessment grid.  So they decided that As would be exceeding expectations, Bs and Cs would be meeting expectations, and Ds and Fs would be not meeting expectations.  And when the assignment came in, they dutifully tabulated the grade information into the assessment grid.

Last week, the assessment gurulings in the department gave the department assessment feedback (called "closing the loop" in assessmentese).  And here's what they found:

58% of students were exceeding expectations
36% of students were meeting expectations
6% of students weren't meeting expectations.

You would think this would be pretty good news, right?  That department must be doing something really good to have so many students meeting and exceeding expectations!  Let's give them a raise!

You would be wrong.

No.  The department was chastised.  They obviously had low expectations if so many students were exceeding those expectations (besides, you know, grade inflation?)

They needed to change the way they do assessment so that the numbers will be more in line with what the assessment guru says they should be.

Let's consider: is there a way to tell if the department really is doing a great job preparing students (who are also, perhaps, dedicated, smart, and hard-working)?

In the new competitive world here, the department will be competing with other departments based, in part, on assessment data.  Isn't it reasonable to think that all departments will have, perhaps, inflated numbers?  Or perhaps everyone is doing a really good job (how could you tell?)

In the new competitive world here, we're working hard to retain students, so we put a lot of effort into helping students learn stuff and demonstrate that learning.  Gone are the days of more than half of students in an intro chem course getting Cs or below based on a brutal curve.  Instead, we expect chem teachers and departments to work some amazing magic, and they seem to do it.

In the new, brutally competitive world here, a world where the unions are broken and tenure is more imagined than real, are departments being asked to do a sort of confessional move to help prepare people for firings to come?    (I'm listening to a book on tape of a history of modern China, and it sounds more like some of the brutal movements which led to a lot of people suffering and dying because some cadre or other forced them to confess and then beat or executed them.)

It's not hard to imagine our crazy administrator saying, "look, that department [say, philosophy] I don't respect has self-confessed and shown that their students are only meeting expectations, so let's can the major, keep adjuncts to teach some GE courses, and use canning the major based on bad assessment as a good reason to fire those pesky professors who ask hard questions about ethics and such during meetings."


  1. In our accreditation visit, a lot of people complained to the visitors that our assessment was meaningless and that it all went into a file cabinet to die. They thought maybe the visitors would do something about that. I think the complainers would have liked it if meaningful assessment would have taken the place of meaningless assessment, if we had to do it at all, which, for accreditation, we do. The result is that we are now having to do more assessment on top of the old stuff and it is COMPLETELY UNCLEAR that the new stuff is going to be any more meaningful than the old stuff. Actually, I lied. It's clear that it won't be. It's clear that this is just one more layer of bullshit that no one is ever going to care about or listen to (or read) and that we only do it to say that we've done it so we can stay accredited.

    Meanwhile, we have no administrative assistants. We have no one who can do any of our grunt work. Not even student assistants. We keep getting more and more work, and we don't get any more pay. We get less and less respect and higher ups saying things like, "We don't reward incompetence."

    I said in a meeting of the big committee the other day, "Imagine a world in which the faculty are happy," and EVERYONE LAUGHED.

    Ten more days of instruction, then finals, then I am going to sleep for 36 hours, and wonder what I'm doing with my life.

  2. There is so much wrong here. Going in with the assumption that the department is failing in its job seems like very bad assessment procedure--both for the political reasons you point out, and because you can't KNOW until you do assessment. Changing the experiment/assessment to get a predetermined set of results is practically fraudulent.

    And the whole "meets expectations" is a really problematic way of designing a rubric if you haven't defined those expectations more specifically. (Maybe you did that and I just can't tell from your description? But if faculty are just using the grading scale to determine if students have met expectations, probably not.)

  3. ARGH! Sounds like you've passed us-- we're still at the portfolio stage. UGH UGH UGH.

  4. I don't love assessment, but I am *very* grateful that so far our assessment is allowed to be rooted in faculty practice. And ours is particularly idiosyncratic, so it's really really nice to be allowed to do what makes sense.

    But I send my condolences to the underwater basketweaving faculty.

  5. This is a frightening, and frighteningly plausible, scenario. While I'm all for period reflection on (and, yes, assessment of) what one is trying to do and how well it's working, formal assessment does seem, all too often, to be a meaningless exercise, especially given the time and personnel available to do it. And I say that from the perspective of someone whose institution actually has very good support for assessment.

    Assessment strikes me as one of those things that naturally happens, in an informal but effective way, when people of good will who have reasonable workloads and decent compensation work together on other projects (e.g. creating and updating curriculum). In such circumstances, periodic formal assessment is even likely to arise organically from ongoing conversations (and to take a break again when it has met its purpose).

    The problem we've got now, from my perspective is a combination of overwhelmed faculty and students and outside mandates for assessment that only make the overwhelmedness worse, which leads to more (justified) worry from outside, which leads to more mandates and possible financial penalties, which leads to the hiring of people to manage (from both the practical and the spin point of view) assessment, which exacerbates the administration-heaviness of higher ed institutions and necessitates cost-cutting elsewhere, including in funds spent directly on instruction/instructional faculty, and so on and so on.

    And yes, there's a reason I keep a reasonably satisfactory but far from ideal job teaching core courses at a large state university. The institution itself seems unlikely to fold, and even if the humanities division becomes increasingly service- rather than research-oriented, there's a good chance that my job will survive at least until I need to retire. Even if tenure-track faculty were moved to a higher load, they'd still need some of the full-time NTT faculty to cover the courses, and I'm pretty senior.

    Which is a pretty discouraging way to look at one's career prospects (and would be even more discouraging if I were tenure track).