Grading Lessons from Cognitive Psychology

You may also like...

12 Responses

  1. Howard Wasserman says:

    Not related to grading, but people on the meat market should keep in mind the advice from your sister’s piece.

  2. Orin Kerr says:

    I always regrade the first few exams I graded for each question, until the regraded scores match the initial scores. At least for me, it often takes a few exams to settle on a fairly-applied standard.

  3. Steph Tai says:

    I feel sort of vindicated, because I’ve done both of these things ever since I’ve been teaching. But I can’t take credit for drawing from the insights of cognitive psychology or anything. It’s just how we used to grade exams when I was in chemistry graduate school–we the TAs would all get together in a room with some pizza, then we’d go through question by question to grade (drawing randomly from a huge pile of exams) so that when ambiguous answers came up, we could share thoughts on how to deal with them consistently. Drawing from a big pile of exams meant that it’d be pretty random what we got, but that we as a group would end up “finishing” grading the question at the same time: when the middle pile ran out. Then we’d put everything back into that middle pile again and start with the next question.

    One other thing I also do (drawing from my earlier experience) is try to mimic that process of discussing ambiguity, which I think helped us reach greater consistency. What I do now (since I don’t have a group to grade with) is write down on a notepad each ambiguous grading issue I come across (as well as how I decided to deal with it) to remind me to do the same with similar issues in subsequent exam answers. Having it down on paper really helps, I think.

  4. Aaron Zelinsky says:

    Howard: a good point. Take the early interview if you can!

    Orin: Thanks. I found myself doing that too, particularly for more complicated questions, but on an ad hoc basis.

    Steph: Glad to vindicate you! My twin brother, a mathematician, had a similar reaction. I wonder if that style of grading is more common in the sciences because multiple people split the work? The ambiguity check sheet is a good idea. I wish I had done it. I would occasionally reach back into the pile to find an old exam, which is a lot less precise.

  5. Steph Tai says:

    Aaron: “I wonder if that style of grading is more common in the sciences because multiple people split the work?” Yeah, it probably is, though I know grad students in non-science departments who are graders for classes where they have to split the work; they seem to just split up the exams beforehand, and then each grader handles their pile of exams however they see fit. So I think it might be more a math/science thing. But I’m not sure where that norm (if there is one) comes from.

    I love the ambiguity checksheet! It’s also helpful for talking to students about the exam later, because it helps you contextualize for them their exam performance with the rest of the exams out there.

  6. Aaron Zelinsky says:

    Steph:

    Interestingly, it doesn’t seem to be the norm in psychology — or at least Kahneman doesn’t think it is — since he calls grading by exam the “conventional” way.

  7. Steph Tai says:

    Oops, this is really embarrassing. I went to college at a school where “science” just referred to the physical/natural sciences (seriously, you could fulfill what we generally referred to as our “humanities” requirement–which technically included social sciences but that wasn’t part of the colloquial reference–using some psychology classes.) As someone who writes about different “sciences” in my own work (and am aware that the categorization of psychology in academia sometimes varies from school to school, actually, mostly between “social sciences” and “biological sciences”), I should and do know better than that. But I slip sometimes, and this is an example. I meant to say “So I think it might be more a math/natural science thing.” Bad me!

  8. Aaron Zelinsky says:

    No problem! I was an econ major. I don’t think anyone considered us part of the “sciences.”

  9. Steph Tai says:

    Heh, in college, we could fulfill parts of our “humanities” requirement by taking econ courses, too. :)

  10. Ken Rhodes says:

    Orin: You have addressed the “Olympic scoring problem.” In subjective scoring, the judges tend to withhold high scores from early competitors so they (the judges) will have an upside potential for grading subsequent competitors.

    In the Olympics they try to address that problem by putting the lower rated gymnasts/skaters at the front of the rotation. That way, it is not unreasonable or unexpected when the early competitors “leave some points on the mat (or ice).” The fairness problem, though, is that a lower rated competitor who performs far above his expected level is screwed by his early position. Your approach of regrading the early competitors at the end of the rotation addresses that. (But it makes extra work for you.)

  11. Adam Garcia says:

    I believe sort of vindicated, because I’ve done general health things ever since I’ve ended up teaching. But I can’t bring credit for drawing with the insights of cognitive mindset or anything. It’s the way in which we used to grade exams whenever i was in chemistry graduate school–we the TAs would all party in a room by using some pizza, then we’d endure question by question so that you can grade………. to know more Call Sacramento Bankruptcy Attorney

  12. Marsha Cohen says:

    Kahneman’s book is on my to-read pile — but I wonder why reading-through-each-exam is deemed the “conventional” way to grade? Did he do a survey?

    I started teaching decades ago, and could never imagine grading law essay exams other than question-by-question. And before I grade any, I read a random sample before finalizing my grade sheet. After all, I may think I know what the answers are, but since I was NOT a learner in my own classroom I feel I need to see what got communicated. Oh, and I mark scores on a grade sheet which I tuck into the exam where I won’t see it when grading the subsequent question. Is there a professor who doesn’t feel a sense of surprise/shock/worry (did I grade this correctly?) when recording a question score and seeing it is way out of line with the others? (When this happens, and it does, it underscores the necessity of question-by-question blind grading.)

    I always randomize, so the same person’s first question isn’t read at the same “place” in the stack as his or her second question, etc. This is not because of a concern for score distribution, but rather that one’s view-of-the-best-answer (even having read a sample first) might change over the course of reading dozens and dozens — and whether one gets tougher or easier, and is or is not successful fighting one’s boredom, it just always seemed fair to randomize the potential for any of these subjective human factors to impinge on individual students.

    Grading is, as I’ve always told students, an art not a science. I have never regraded an entire set of law school exams just to see what would happen. I’m confident both that there would be some scoring changes and that they would be minor.