Assessment = Grade?

 I was in one of those meetings with people where we were supposed to talk about assessment stuffs, and the question came up:  Assume you have three possible outcomes for a specific assessment bit, which is being assessed based on a given assignment, say an essay.

exceeds expectations

meets expectations/

doesn't meet expectations.

One of the people in the room said, basically, that meeting expectations meant the assignment was an A.

I tend to think more, meets expectations demonstrates competency, which is a C to start, though a B definitely meets expectations, too, and maybe an A exceeds expectations?

How do you and your colleagues talk about these?

A Big Assessment Question

Like a lot of people, I'm in a couple of facebook groups, including several aimed at academics.  There are a couple feminist groups, and pandemic response groups and so forth.  They're interesting, especially the one with most members in another country.  Today I was reading, and saw an acronym I didn't recognize, and asked.  And the person responded partially, which sort of helped, and then I asked for the rest, and I think they thought I am especially dense.

It was one of those assessment acronyms, the ones we all use at our different schools, some of which mean the same things even though we use different words, and others. . .  well, they don't.  And it really made me think about how thoroughly the language of assessment has taken over academics.  For me, it started right after grad school; I don't think I ever heard anything about assessment in grad school, but right after, BOOM!

A whole lot of academics (at all levels) have been working very hard on assessment stuff for a LONG time now.  And here's the question:

Have students gotten better educations as a result?

The one program around here that clearly points to a change is Introductory Underwater Basketweaving.  Where, before, we all used a hodgepodge of strategies and techniques to teach students the basics, the field has sort of redefined the basics to focus on a lot of terms to do with weave patterns and such.  And our entrance exams all focus on students recognizing weave patterns.  And, by golly, at the end of our course, when they're assessed, they all recognize and can name various weave patterns better than ever before.

But, you know, the baskets they weave don't look much different than they did 20 or so years ago when I first got here.  Some students learned good weaving strategies in high school, and they still use them with great success.  Some students need help preparing the reeds for weaving, and don't really get it, so they're baskets are a mess, but they can name the weave patterns. 

If the point really is that students can name weave patterns, then we've done a good job.  But if the point is that they can weave baskets underwater better, than we haven't.

That's how assessment feels: where we've defined something and focused on that, yes, students can often do that thing better.  But if what we really want students to learn is hard to measure, then are we actually doing a better job educating students?  Seriously, because every school I know of has poured tons of money and time and effort into doing massive assessment.  But by and large, students earning degrees seem no more or less educated than in previous years.  They learn different things, as they should, but overall, are they better critical thinkers?  Better able to understand the world?  Better able to communicate? 

Further: are they more humane?  better citizens? 

The Not So Obvious Agenda in Assessment

I was at a meeting today about some assessment stuffs.  It was one of those meetings where people look at assessment info and decide if the course gets to keep its qualification for GE.

One course was flagged because every student in the course met the goals.  That's a problem, a colleague said.  They should tell us how and why they're doing that.

It's a small upper-level course, I said (well, not in these words, since it was on line and all, and it seems like it would be a problem if the instructor weren't reaching most students with this rather modest goal.

And the other person wanted more information about the meeting the goal thing, because they suspected it couldn't be so.  It might be that the faculty member wasn't doing more than clicking a number.  Which could be, of course.

So, I understand, but what I want to know is, what's the secret number that will or won't provoke Professor A's suspicion?

There's really no answer.  But 100% for sure, unless there's a good explanation.  Though the form didn't ask for any explanation like that.

It's one of those things where, ideally, everyone would be aiming for 100%, but if you get there, suddenly you're suspect.  Damned if you do, damned if you don't.  Frustrating!

Another Meeting, Another Depressing End to the School Week

This week's meeting wasn't quite as bad as last week's, but was frustrating.

We talked about assessment.  What do you know, most of our majors are doing okay or better.  Wheee.  At one point we were told that we shouldn't have too many of our majors doing better than okay, because then we'd have to change what okay means.  And you could tell some folks obeyed, and so their part of the major had students who were mostly okay, and a few better.  And some folks resisted, and their part of the major had students who were mostly better and a few okay.

And we all acknowledged that this is pretty useless and meaningless, and we're just checking boxes.  But we still have to do it.  And then we talked about whether there's a way to make it at least minimally useful to us without adding a lot of work, and we basically came down to: we need to have time to talk about what we're doing in the classroom, what we're seeing students do, and what we're looking for.  That's probably another meeting, but at least it has the potential to be useful.

It won't be useful to the administration, though, unless we put numbers somewhere.

Some people want to know how students perceive, say, their ability to do research.  In my experience, most of our students think they're brilliant researchers because they know how to search googlebooks, even if they don't read carefully or deeply enough in the 1950s book they found to realize that it's incredibly racist in its assumptions about, say, Othello.

Then there was another meeting at which we talked about colleagues, and that was good, because our colleagues are stellar.  My little part in this was praised, as were other peoples', and rightly so all around.

And then we talked about what search we'd like to do on the off chance that anyone gets to put in a request to do a search, and that was long and frustrating because it's unlikely we'll get to do a search, and we really need two positions at least.  But pretty much every department across the university is in the same boat.  And the dean's criteria (at least he gave us some for this) are basically, "shiny, new, not just replacing people to be able to teach your curriculum."  So if you lost a historian of antiquity, you can't say, well, we really need someone to teach ancient history because it's important in our curriculum.  Instead, you need to find some shiny, new thing to ask for, and hope the eventual hire can also teach ancient history as well as digital histories or whatever.  Because shiny!  new!

At least my colleagues are generally good people to work with.  We disagree about some things, but we were able to do so productively and decently, and that counts for a lot.

By the time we finished, it was 5pm, and I had dinner plans at 6, so I didn't bother to go home, but just graded, recorded grades and stuff, and now I have only one thing to grade for the weekend, and it shouldn't be onerous.

And I get to read and prep Gawain, and that makes life good.

In way better news, I had a really good violin lesson this week.  I'm working on the first Seitz concerto in Suzuki Book 4, and it's quite hard at my level.  I think Suzuki does a really good job of putting pieces in that make me work hard, but not impossibly hard to play them.  Part of this piece has a section of slurs that cross strings with fairly fast fingerings.  And practicing them, in order to play them all on one bow, I move the bow fairly slowly.  And on the violin, if you move the bow too slowly, you sound scratchy.  So all week practicing, working on the slurs, I've been sounding scratchy.  But I knew why, and so knew I just had to work to the point where I could play the slurs all on one bow more quickly.  But I wasn't there in practice.  I also realized that I could move the bow a little faster, and thus less scratchier, if I consciously used the whole bow.

When my lesson started, I pulled out my music and said I'd really been focusing my practice on these two difficult areas.  And I played one, and Strings helped me with it.  And then I played the slurred parts, and amazingly, and for the first time EVER, I sounded better in my lesson than I had in practice!  I shocked myself!  So Strings was able to give me some more help, and then we worked on some of the things I hadn't done so much work with, and Strings gave me some help on those, too.

And I got this new scale thing, almost more of a key thing than a scale thing.  It's called a one position scale.  The idea is that working within the notes of a scale, you play every note that you can within one position.  So, for example, if I start with the C scale on the violin.  I can start with G, then A, B, C, and so on, all the way up.  So unlike a regular scale, you don't start and end on C, but start and end where you can in that position, with any note in the scale.  It's weird, and sort of hard, but I'm starting on C.

And, Exhale

The term is over!  Well, mostly.


I turned in grades, but still have to go fill out assessment forms.


Why didn't I fill out the forms when I did the grades?  Good question.  On Sunday, I got a call from my niece that she was doing a special concert on Thursday, and inviting me.  And a friend was moving, and ended up needing a whole lot more packing help than I'd figured on.  So, I helped my friend, did my grading, turned in my grades, raced across the Midwest to the concert, and then came back.  My niece did really well at her concert.  She's really an adult now, and a good, kind, thoughtful, smart, caring one.


When I got back, there was news from our one year search in interdisciplinary basketweaving.  We'd passed five names along to the interdepartmental director, and they started calling people.  One of the five politely declined the director's interview because they had another job.  The job's been offered, and we're waiting to hear from the person now, I gather.


If they don't accept, then what happens?  I don't know, but I'm guessing it will be my responsibility somehow.


I offered to chair a search in my own department next fall, and the offer was accepted.  The ad has to be done on our end within the month, so that it can be approved on up the line and be ready to go out at the very beginning of the search season.


I need to make a list and check it twice.


So, of course, I've gotten messages from my mom, asking if I'd turned in my grades yet, and if I were taking time to hang out with my niece and sister-in-law across the Midwest.  Apparently, in her mind, I'm an irresponsible person who barely keeps my job and doesn't do anything much all summer.


I dread the stupid assessment forms.  I guess I'd better look up and see what I have to fill out and for what authority (departmental goes one way, program another, GE another, and if we had accrediting in our field to worry about, that would go another yet); each path, and each form in the path has its own unique set of acronyms.


The other thing I found out when I got back is that two thirds of my colleagues are relatively sane.

Assessment - Vent

If you're in education, you know that we've been working in a variety of ways to "assess" student learning for something like two decades now.  Before that, we graded students.  It wasn't perfect, of course, but it was there.  We assumed that a student who got an A on an exam had demonstrated that they'd learned the material being tested (and hoped that exam grades weren't primarily a matter of being good at bubble tests or whatever).


But, assessment folks have told us forever that grades aren't the same as assessment, but that assessment has to be "outcome driven" and such.  And the assessment people told us that what's important is that we all use the information to tell us how we're doing as teachers, and to help us improve.


And we needed to assess programs and not just how students do in individual courses.  So we couldn't assume that a straight A math student was succeeding at learning math.  We had to set goals, and then figure out outcomes that would reflect those goals, and then figure out a product that would demonstrate the outcomes that would reflect the goals.


Let me tell you about some of the history of our Underwater Basketweaving Program's assessment adventure.


In Underwater Basketweaving, they set some program goals.  For example, they want students to understand the structure and context of underwater-woven baskets. 


Some years ago, the idea was that they'd look at the baskets the students wove underwater and the papers they wrote about baskets woven underwater in a sort of portfolio.  But then someone would have to collect the portfolios, and someone would have to read the portfolios, and there was no money to pay someone to do this, since the expensive assessment guru was too busy making forms for departments to fill out to actually do any assessment himself.


So portfolio assessment went away, and the department struggled for several years to meet the ever-moving targets and to do the paperwork the assessment guru set for them.


Finally, the assessment guru said that they had to do a grid for their department goals, and even though they started with 25 goals for their students, they quickly realized that they couldn't fill out paperwork for 25 goals, not without making paperwork the primary job rather than teaching and research and weaving underwater.


So in line with the assessment guru's rules, they had to make an outcome to go with the main goal, which was that all students should be able to describe the structure and context of an underwater-woven basket, since you have to be able to measure understanding in some way other than "student nods knowingly." 


And then they decided that in one of the early courses in the major, students would do an assignment where they'd describe the structure and context of an underwater-woven basket.


So the guru suggested they make a grid and turn in the numbers this way:  for each goal, they'd have three possibilities: the student exceeded the expectations of the outcome, the student met the expectations, or the student didn't meet the expectations.  And then someone would look at the assignments and fill in a separate grid piece for each of the outcomes.  And to make it easy, they'd rotate the outcomes, one a year, so that someone would only have to fill in one of the grids after reading all the papers for these sections.


But again, there was no money to pay people to look at the assignments for assessment.


So the assessment guru said that each faculty member should be responsible for filling out the grid for their section.


And the faculty met, and noticed that they were already grading these assignments, so how about if they found a way to translate grades into the assessment grid.  So they decided that As would be exceeding expectations, Bs and Cs would be meeting expectations, and Ds and Fs would be not meeting expectations.  And when the assignment came in, they dutifully tabulated the grade information into the assessment grid.


Last week, the assessment gurulings in the department gave the department assessment feedback (called "closing the loop" in assessmentese).  And here's what they found:




58% of students were exceeding expectations
36% of students were meeting expectations
6% of students weren't meeting expectations.


You would think this would be pretty good news, right?  That department must be doing something really good to have so many students meeting and exceeding expectations!  Let's give them a raise!


You would be wrong.


No.  The department was chastised.  They obviously had low expectations if so many students were exceeding those expectations (besides, you know, grade inflation?)


They needed to change the way they do assessment so that the numbers will be more in line with what the assessment guru says they should be.


Let's consider: is there a way to tell if the department really is doing a great job preparing students (who are also, perhaps, dedicated, smart, and hard-working)?


In the new competitive world here, the department will be competing with other departments based, in part, on assessment data.  Isn't it reasonable to think that all departments will have, perhaps, inflated numbers?  Or perhaps everyone is doing a really good job (how could you tell?)


In the new competitive world here, we're working hard to retain students, so we put a lot of effort into helping students learn stuff and demonstrate that learning.  Gone are the days of more than half of students in an intro chem course getting Cs or below based on a brutal curve.  Instead, we expect chem teachers and departments to work some amazing magic, and they seem to do it.


In the new, brutally competitive world here, a world where the unions are broken and tenure is more imagined than real, are departments being asked to do a sort of confessional move to help prepare people for firings to come?    (I'm listening to a book on tape of a history of modern China, and it sounds more like some of the brutal movements which led to a lot of people suffering and dying because some cadre or other forced them to confess and then beat or executed them.)


It's not hard to imagine our crazy administrator saying, "look, that department [say, philosophy] I don't respect has self-confessed and shown that their students are only meeting expectations, so let's can the major, keep adjuncts to teach some GE courses, and use canning the major based on bad assessment as a good reason to fire those pesky professors who ask hard questions about ethics and such during meetings."

The Assessment Dance

This summer, I again said I'd help with an assessment task.  It's basically the same task I blogged about here and here last summer.

Remind me not to say I'll help again next summer, please.  Except the extra money certainly helps with summer.

Some things are better.  The design for entering data is slightly better, maybe two fewer clicks. 

And the way they've organized the work.  Last year, everyone had to have their first 40 pieces done before the organizers reshuffled and we did the next (and last) 40 pieces.  This year, they're pre-shuffled and it's set up so that once the person I'm paired with has clicked that she's done with her part, I can do that same one and then enter the data and be done.

And they've also cut the days they're giving us to do the project so that we have a week to do each set of 40 pieces.  Last year, it took well over a month start to finish, so I was rushed because they hadn't told us about the second half when we signed up.  And, of course, some people will put off the work until near the deadline anyway.

So, today marks a week since we started, and I've completed my first set of 40 and half of the next set, and I'm going to work on the next quarter of the next set (which my partner has already finished) later today.  The first set is due today, so if I get with things, I should be done totally by tomorrow morning.


Since I'm an English professor who teaches comp (about half my load), you can probably guess that I'm reading a lot of essay type stuff to do these assessments.  It's a LOT faster than grading, in part because you don't have to give any feedback or explain stuff to the writer, and in part because you pretty much have no emotional investment in the writing.

But, holy cow, there are certain topics that students seem to love and I just don't get it, and a couple of these topics they just can't seem to wrap their heads around in useful ways.  (If you're someone who teaches writing to first year students, you may want to turn away at this point.  Or maybe you'll want to commiserate.)

Social Media.  Students always want to write about how bad social media is.  But if you ask them, they love using social media.  And they can't seem to get that they need to think about why they love it, because they're getting something from it that's satisfying or whatever, and that means that it's meeting some perceived need, so it's not all bad.  (I think of it like, say, cookie dough.  I know cookie dough is bad for my health, just in sugar alone.  But I also know I like it, and that if I've had a horrible day, eating cookie dough makes it feel less horrible.  So if I were writing about cookie dough, I'd want to write about the benefits of eating cookie dough for my morale.  Cookie dough eating would make more interesting writing projects.  Just saying.)

Bodybuilding.  The students who choose to write about this seem to be already into body building, so they always use their favorite body building websites.  And then they moralize about how bad steroids or other naughty stuff is.

Food.  These seem to be driven by one or two faculty who focus on the evils of eating animals, so the projects are predictably about the evils of meat production, without recognizing the complications of meat production, food pricing, and so on. 

Princess-type Topics.  Whether it's Disney or kiddie pageants, these tend to be really shallow and blah.  One I read sounded like it was an excuse to rewatch every Disney movie the student had loved as a kid.

Favorite Fandom.  Enough said.

Disease of the Week.  These are pretty variable, to be honest.  They tend to use WebMD type sources, and most really aren't scientifically informed enough to read actual science papers, so they never get very deep.  I think they could do okay with flu, but they don't pick flu.


Surprisingly, I've read a couple topics where students seem able to dig in and do well.  I've seen a couple projects on issues about paying college athletes that were well done.

So far, none of the projects are on abortion or legalizing marijuana, and the one on gun control was actually pretty okay.  I haven't had any on tattoos or piercings this time around, thank dog.

On Assessment

Recently, Jeffrey Alan Johnson had an interesting article in Inside Higher Ed, "On Assessing Student Learning, Faculty are not the Enemy."  It's a smart article, and makes good points about how assessment isn't "neutral" and will require change in curricula.  Johnson notes that administrations using assessment to demand changes in curricula is a change in the power structure; rather than faculty controlling the curriculum, administration uses assessment to control it.  (Administrators may say they don't, but Johnson argues against that.)

Part of the problem Johnson notes comes in what gets assessed.  If we assess Pattern Regularity in Deepwater Basketweaving, and not Basket Shaping, then faculty folks are going to be forced to focus on pattern regularity, and the basket shaping parts of the curriculum are going to fall by the wayside. 

And the faculty may, given the mandate to choose to assess something that can be measured and proven, choose to assess pattern regularity, rather than basket shaping, just because it can be measured.  A regular pattern may mean nothing without a well-shaped basket, of course, but we can't assess every single thing we do, because we do a whole lot in any course, so what gets assessed is going to be what can be assessed most easily because it can be measured.

Business folks are all about how you have to be able to measure something in order to manage it.  That's easy if you're measuring how many widgets get produced, but considerably harder if you try to figure out the quality of different foods cooked in a restaurant, or even more, the education of a student.  If you measure food quality by taste, then that's one thing, but if the consumer gets horribly sick from food poisoning four or five hours later, then the food quality was bad no matter what it tasted like.  That won't make it onto the survey at the end of the dinner, though.  Education is even more complicated because many people won't really gain an appreciation of their education until they've been out of college for a couple of years or more.  And some students won't put things together fully until they've had a chance to mature a bit.  Let's face it, Jane Austen in your 20s is good; Jane Austen in your 30s is WAY better.  It's not Austen's quality that changed, it's the reader.

Johnson also talks about the dismissive ways administrators talk about faculty when they talk about assessment, as if faculty are recalcitrant children.

The comments to Johnson's article are also useful and telling.


Over at Confessions of a Community College Dean, the Dean reads Johnson's article and talks about how difficult assessment is to do when there's no obvious capstone (as in most community college programs, especially those intended for transfer).  One thing he talks about is that his college has an assessment committee made up of people whose workloads are adjusted so that the assessment work counts, and those people read assignments and make recommendations to their colleagues.


If you've read here for a while, you've probably gathered that I have a negative attitude towards assessment.  Reading the two articles has made me think about why.  So, here, in no particular order, are some reasons I'm negative about assessment.

1)  It's not actually about student learning, but rather it's always about accreditation, and especially about accreditation on a sort of business model, which sees us as turning out products (we call them "students" but assessment measures products) and asks how much value we've added as the products pass through our system.

When we're told, as I've been told lo these many times, that we have to "assess ourselves or someone else will 'do it' to us" with the "do it" being characterized in somewhat violent ways, that's coercive.  And it doesn't help me be a better teacher.  

2)  What's important can't necessarily be measured.  And what can be measured may not be all that important.  If you want to measure that students can learn how to think critically by taking courses that ask them to think in different ways, to learn about thinking, and to solve problems, how do you measure critical thinking?  You don't do it by telling students that you're teaching them critical thinking and then asking if they know you've been teaching them critical thinking.

I have a colleague who thinks I should write down all the goals for a semester long Shakespeare class so they can be measured.  I could begin, but the list would go on and on, everything from thinking about how racialized thinking in early modern culture contributes to our own racism, to how playhouse structure matters in theatrical experience, to ... well, I don't even want to go on.  How do I measure those things?  Some students will learn more, and be able to integrate what they've learned about race in US history, and something good will happen.  Other students in the very same class will deny that race can even matter because they think we're post-racial somehow.

3)  Administrators communicate that they think faculty have never ever in our lives thought about teaching, and can't even begin to think about teaching until we've been taught the official language of assessment-speak.  It may be that some few faculty never think reflectively about what the heck they're doing standing in front of 30 or 200 students, but I've never met a single faculty member who didn't talk as if they thought reflectively. 

4)  There's an ever-changing target, and it always involves a lot more work for faculty.  I recently wrote about spending hours in department meetings figuring out how we were going to apply for our courses to move from the old general education to the new general education system, even after we've all been told that the new system was specifically designed so that the top 70 courses from across the university would be the same top 70 courses in the new system.  That is, the system is designed to be almost exactly like the old system in everything but names.  We fill out the forms in good faith, spending hours to get the language just as the committee has told us it wants, and then the committee comes back and says that we have to do all those forms differently because what they said they wanted isn't it after all.

5)  There's an ever-changing target because the folks who claim to be assessment experts have little idea about what they're doing, and so keep constantly coming up with new ways of figuring out how to measure students as products because students as products aren't measurable.

I've been through assessment systems based on: portfolios, exit interviews, "embedded" assessment, and on and on. 

6)  It takes an administrator just a few seconds to demand hours of work from faculty and staff in any department, and none of that work seems at all helpful to our teaching.  Time is not flexible, not really.  If 20 people in my department spend an hour in a meeting working on assessment stuff, that's an hour we could be spending another way, and 20 hours altogether, which is half a work week for normal employees.


That's it.  I have to work on some stuff that actually may be of use to my colleagues.

Re-Formations

Like lots of schools, we here at NWU are busy reforming our general education system.  In large part, this is prompted by the corporatization requirements of our accreditor, which demands that we measure output to prove that our students actually learn stuff.  (Our accreditor also accredits one of the more disreputable for-profit "educational" corporations, so I might be a little cynical about it as an accrediting body.)

In part this is prompted by everyone's sudden interest in producing graduating widgets at the fastest rate possible.

At any rate, we're supposed to measure output using criteria we develop ourselves, and then prove to the Committee for Professional Output Optimized Performance and Institutional Excellence that we've measured the output so that they can prove to the accrediting organization that we've measured and are putting out the output we claim we're outputting.

And now all the stuff is phasing in.

Right.  Last year, the Committee for POOPIE requested that all the "bigger" courses (that is, the courses that served the most students overall) that had previously given general education credit under the old system be put through a process to request POOPIE credit under the new system.

The new system has different categories.  One is content, or C.  There's content that's math, and content that's science, and content that's historically oriented, and content that's artistically oriented, and so forth.  That is, we have CM, CS, CH, CA, and so on.

There's a Social category, or S, for getting students to learn about civics and so forth, with three subcategories, organization, action, and international, so SO, SA, and SI.

And then there's the Practice category, or P, which include writing (PW), arts (PA) and so on. 

Each student is required to acquire a certain number of POOPIE stuff in each category.  (A course can count for POOP stuff, but supposedly other, err, stuff, can also count as POOPIE stuff.  This makes the housing folks happy, because they want to count dorm parties as POOPIE stuff.  As they tell us, most POOP happens outside of classrooms.)


The Department of Underwater Basketweaving duly (so my contact reported) spent many hours of department meetings deciding which categories to apply in.

"Intro to Understanding Underwater Basketweaving" folks applied for CH and CA POOPIE categories, because they study both the history of underwater basketweaving and the artistic aspects of underwater basketweaving.

"Social Underwater Basketweaving" folks applied for SA and PA POOPIE categories, because they make baskets and teach about how underwater basketweaving is social action, because the personal is the political in underwater basketweaving.

Intro to Underwater Basketweaving Reed Cultivation" folks applied for CS and SA POOPIE categories, and so forth.

From what I heard from my Underwater Basketweaving friends, many hours of brainstorming went into these discussions.  Over the years, the department has developed lots of general education courses to serve the university, so they had a number of courses to do applications for, and a number of them applied under CH and CA POOPIE categories.  And sent them off for review in April and May, as required by the Committee on POOPIE.

And now, this year, the Committee on POOPIE is meeting regularly to decide which categories of POOPIE each course is going to qualify for based on the applications put forward by the departments.

And now, they've apparently decided that no single course can teach two Content categories (or two Practice categories, and so forth).  So all the courses that had put forward as CH and CA (a fairly common request in the Underwater Basketweaving area, because the courses teach an art form and often historical contexts for the art, and so forth) have all been rejected, while the courses requesting SA and PA POOPIE categories have been accepted.

Why didn't they tell departments across campus last year that no course would be allowed to claim it taught both CH and CA content?  If they had, departments across campus would have decided during all those hours of brainstorming and work which they'd apply for, and not spent time needlessly trying to word things just right in both categories.


And thus ends my tale of woe.  Or, perhaps, tail of woe?

Clicking Through Assessment

I finally finished working on this assessment project.  On one level, it was a good learning experience about how assessment folks think. 

We used a rubric with three categories, read 40 of the same basic assignment across many sections, and then rated each within the three categories.  Then the powers that be shuffled, and we read and rated 40 more that had been done by someone else in the first set.  (But the list included more than just our assigned pieces, so that there were 3 screens of 30 each, and a fourth page that was shorter.)

Everything was supposed to be done on line by unique identifying number.

I printed out my list of numbers and wrote things down so I could do the ratings entries in groups.

And that's when the real frustration began,

To enter the rating:

Pull up the list on one tab, and a rating page on another tab. 

Copy a number from the list, and click over to the rating page.

Click on "add a rating" to open a new dialog box thing.

Paste in the number.

Click on a drop down box and pick a number.  Do it again.

Scroll down so you can see the final category.  Click on the drop down box and pick a number.

Click on the save button.

Rinse and repeat.

*** Why use a drop down box rather than just a set of horizontal buttons?  Why not make it so that the full dialog box is visible without scrolling?  They could save four motions, easily.  That may not seem like much, but if you're doing 40, it gets to be.


The final step is to enter that you've done a specific assignment.

Go to the list again.

Click on a button next to the identifying number.  That opens a separate dialog box.

In that box, put a check in a "done" box.  Hit Save.

The "save" takes you back to the first page of the list.  And then, if you're not on the first page, you have to scroll down to the bottom where they have the button to go to the next page (but not, of course, a choice of going to the third or fourth page directly).

***Why not just a click on that first list to say a given identifying number's assignment is done?


I don't think I'm super picky about on line document design, but when you're entering 40 pieces, you notice stuff that makes it less efficient.

And I do think the lack of efficiency reveals that the people who are designing these assessments don't do much pre-testing.  And that lack of efficiency thus makes me think that these assessment designers don't think through their assessment stuff, the basics, any better.

I've been through three or four big assessment "pushes" here at NWU so far.  It started with portfolios and exit interviews, and then after some years of putting in all this effort to do these, we were told that the assessment folks didn't even look at the data, and we had to do this other thing.

The other thing got done for a couple of years, and then the assessment gurus told us that they didn't look at the data because it's useless, and we had to do this other thing.

And so forth.

There's a pattern of bad planning, no pre-testing of the plans to make sure they'll work, little thinking ahead, and a whole lot of wasted faculty (and in the portfolio days, student) time.


***If I'd designed the document, here's what it would have looked like.

A list with columns for the assessor's name, the unique identifier, the three ratings, and then a box that says "done" and, if you're concerned, a "save" tab that makes it saved, and maybe a final "edit" box so you can make changes just in case.  (If really necessary, a "comment" button that opens a place to add a comment.)

On each screen, top and bottom, page arrows and number choices so you can move from page 1 to 4 with one click.

But yeah, I'm just an English professor.  It's not like I know anything about texts.


And that task done, I turn back to starting my summer anew!  Yesterday, I mowed, harvested a bunch of yellow squash (cut them up and put them in freezer bags for winter crockpot use), and went out with a friend for some biking/paddling, with fresh strawberries for a snack.

Reading for Assessment

Relying on the idea that in order to understand what's happening, I need to be part of what's happening, I signed on to do a (paid) assessment gig for NWU.

We're working on assessing one aspect of an assignment for a course that has four assignments in common between all the sections.  So, for example, if this were in intro to underwater basketweaving course, we'd look at a basket the student has produced, and assign a score for each feature, basic basket shape, regularity of weave, and size of weave.  We'd assign a score of blue for any feature that is better than our expectations (based on criteria for each feature), green for any feature that meets our expectations, yellow for any feature that shows progress towards expectations, but doesn't meet them, and red for any feature that shows no progress (they turned in a rock, for example).

In this hypothetical world, the underwater basketweaving faculty have agreed that all sections of intro to underwater basketweaving will include several common assignments, including a basket starting assignment, a patterning assignment, a basic cup-shaped basket, and a non-cup-shaped basket.  We're doing the cup-shaped basket.

The thing is, it looks like while all the sections do assign a cup shaped basket, some of them seem to require that students use at least two different weaving techniques, which makes it really hard to judge the "regularity of weave" section.  Other sections seem to all think cup shapes are tall and skinny, or more shallow bowl, so the basket shape is hard.  And still others let students use different materials, meaning that the ideal size of the weave changes a lot.

We got together in a small group and did a half day of "norming" using four basket examples.  ("Norming," for those unfamiliar with the term, means working together, in our case with rubrics, to come to a basic level of agreement about the scoring of an assignment.)  We used a rubric developed by the faculty earlier, and found some pretty basic difficulties in scoring baskets where, for example, a student had made a cup-shaped basket, and seemingly tried to incorporate several different weaving techniques.  It was an ambitious basket, but it scored way down on regularity of weave because the person wasn't using anything like a regular patterning, and the sizes were pretty all over the place.  So it got a low score.  If I'd been grading it, I would have given it a pretty good grade because it took risks and showed real learning.

One of the problems with assessment done this way is that there's no room for a "no, but" answer.  There's no room for a "no, the weaving isn't regular, but wow, there's something cool happening here."

That's not to say the rubrics are bad, because the UB faculty clearly values basic shape, regularity of weaving, and weave size (the tightness matters, I suppose).  So those are important things to make sure they're teaching.  And so that's what they're trying to measure in their assessment.

What they aren't measuring is how a given student is taking a risk, trying something different, reaching out, finding their own path.  I don't know that those things can be measured, or how, but I know they're important aspects of student learning, probably more important than the regularity of weaving or the size of the weave.

Rethinking Goals

I thought more about the goals, more about the class, and more about stuff last night, and realized that I hadn't put in some of the stuff I think is really important.

So here's some more brainstorming for Shakespeare:

1.  I want students to get better at reading and writing about difficult texts.  (Reading in terms of strategies for reading: looking up words, thinking about sentences, reading aloud, and so forth; writing in terms of putting the reading skills into words on paper.)

2.  Develop understanding of Shakespeare's work in relationship to standard literary genres, techniques, and style (Thanks, Sapience!)

 3.  I want students to get a sense of the historic, cultural, and social contexts of early modern theater. 

4.  I want students to begin to learn research (asking questions, finding information, reading critical articles for the argument).


One of the problems I have with these sorts of exercises is that over the course of 15 weeks, I want students to learn a lot, and to think hard, and to begin to put things together, but I know different things will stick with different students, and that I can't do everything, but that there are probably 20 things I want to stick with someone in some way.

Goals, Outcomes, and Madness

In the ever constant race to update and revise and be prepared for our next departmental review, we're doing these projects to make our courses "outcomes based."   For every course, we're supposed to have goals (which, as I understand it, are supposed to have to do with what we want students to learn), outcomes (which, as I understand it, are supposed to have to do with what students produce to show that they've reached the goals), and sub-outcomes (which, as I understand it, is what students actually produce to show they've reached bits and pieces of the bigger goals and outcomes).  We're supposed to have between 1 and 4 goals, 1 and 4 outcomes (one for each goal), and for each outcome, 1-3 sub-outcomes.

I want to brainstorm a bit here, because I find this sort of stuff intensely frustrating and not fun, and I don't think I find it nearly as helpful as the edu folks keep promising I should.

Let's take, say, an intro Shakespeare course, a genre-based Shakespeare course.

What should the goals be?

1.  I want students to get better at reading difficult texts.

2.  I want students to get a sense of how damned good Shakespeare is.  (I think of this as building in an understanding of some plays, how genre works, how imagery and such work, how character development works, and on and on.)

3.  I want students to get a sense of the historic, cultural, and social contexts of early modern theater.  (Yes, that's huge.  HUGE!)


Do those goals make sense?  Are there better goals?  Are there more goals?

Edited to add:  There should be something about writing in there, maybe in the first one, reading and writing about difficult texts?

Time Time Ticking...

I'm about to head off to a meeting which, based on the agenda distributed, looks like it will be a bit of talking, more talking, and no decisions needing to be made.  I hate these sorts of meetings generally, but I will go to this one because it's in the job description.

We've been working on assessment stuffs, mostly on doing stuff that says we'll assess our courses in this given way for these given assessment points, which all include at least three subpoints.

I've spent probably a couple hours on this for the class I was supposed to do (and did).  If everyone in my department spent just about two hours on theirs (we pretty much all had a class to do), then we've wasted two weeks of work.

Wasted?  I'm sure the latest assessment guru would tell us that this is vital work, and that it's all important for making sure that we teach what we've decided is worth teaching, and that our students learn what we've decided is worth learning.  And by the way, there's that ever present threat that if we don't do it to ourselves, "they" will do it to us.

And in three years more, having done these and many more, having filled out assessment bubble things, and written reports, all of which will show that our students aren't learning as much as we think they should, but some of them are, and some not at all, the assessment guru will tell us that no, we've been doing it wrong, and now there's some cool new way to do assessment, so we should start over again.  And it will be treated as practically an emergency situation, so we all need to do that new thing lickety split.

And then we'll need a new assistant to the recently hired assessment guru, because there will be ever more work to do in assessment.

One of my colleagues and I were chatting today, and she expressed her frustration at the constant near-emergency attitude from the administration, and at the ways that this assessment work takes time away from other work we need/want to do, work such as, you know, teaching, prepping classes, designing courses, and so forth.

Wagging the Dog

Over in the underwater basketweaving department, they're working on assessment projects like mad because the new campus assessment person has said jump in a very loud voice, and the headmaster's assistant has cracked the whip.

They have this program, a certificate in underwater basketweaving, which is a bit less than a minor, but gives students practical skills for managing and working with basketweaving projects later.  The certificate has six classes, Basketweaving basics (UBW 10), Underwater safety (UBW 20),  Introductory reed work (UBW 101), Basketweaving theory (UBW 103), and Integrative basketweaving (UBW 190, a sort of capstone experience), with one elective (at the 100 level or above) to be chosen from a group of basketweaving classes.

They were told that they had to write out specific goals and outcomes for the certificate program, and that for every goal, there had to be at least one outcome, and for every outcome, at least one way to demonstrate that every student did it.  The pressure from the campus assessment person is that we all have to do assessment stuff "in a meaningful way," and so he hints that we should all limit the numbers of outcomes.

The goal for this certificate is that students should be able to weave basic underwater baskets, and that they should do so safely, and they should understand how basketweaving works.  That gets broken down into a couple of outcomes, which I'll number:

1)  Weave underwater baskets.

2)  Use safe and appropriate techniques.

3)  Understand basic basketweaving issues.  (Which in assessment outcome talk is going to come out as something like "demonstrate an understanding of basic basketweaving")

Then they were told to make a grid, showing where each of the outcomes was taught, and where each would be assessed.  Let's use "T" for courses where things will be taught, and "Ass" for where they'll be assessed

Basketweaving basics (UBW 10):  1T; 3T

Underwater safety (UBW 20):  2T

Introductory reed work (UBW 101): 1T; 3T

Basketweaving theory (UBW 103): 3T

Integrative basketweaving (UBW 190, a sort of capstone experience):  1A, 2A, 3A


Since they don't know which of the electives students will take (Advanced Underwater Safety, Intermediate Basketweaving, whatever), they don't put the electives on the list.

Then they're asked to fill out another form, telling how UBW 190 is going to assess each of the outcomes.  So they've chosen as their assessment the final project, the production of an underwater basket without drowning or losing body parts.  Oh, and they also have to use something written to demonstrate understanding, but some people don't give an exam, and some people require an extensive reflection piece about the woven underwater basket, while others do give an exam, but do it as an oral.  It's a mess, but they're working on it.


And now, they're a bit worried, because budget cuts are coming down the pike along with an emphasis on getting students through programs as fast as possible, and they're going to be driven by assessment.

UBW 103, the theory class, doesn't seem to teach any of the outcomes that aren't also taught in the other classes, so are the beancounters going to come back and say, "Well, it doesn't contribute to the outcomes you've assigned, so why is it required?  If it were important, you would have put it on your grid."

"And the elective?" the beancounters will ask, "Do students even learn anything in the elective?  It's not on your grid, so they must not learn anything."


Coming soon to a campus near you, because it's for the students!