Exam Writing History: The Fall of the Monster Issue Spotter?
posted by Dave Hoffman
Like many professors, I’ve been spending much of my time this past week putting together my exam (this semester, I’m teaching a Human Behavior Colloquium and Corporations). In doing so, I’ve been thinking about the tension between an old poll I did on best exam writing practices and the comments that it engendered. In particular, I was interested in the possibility that short-answer questions and multiple choice have been on the rise of late in law school exams.
I believe, based on conversations with older lawyers, that back-in-the-day (which is much more recent than it used to be) most law school exams consisted of one, two, or at most three large issue spotter questions. (LISQs) LISQs in turn gave rise to IRAQ, which is a terrific idea if executed with enough preparation and facts on the ground. But over time, in seems, more professors have turned to multiple choice, true-false, short-answer, and other forms of evaluation.
You might be tempted to think that this choice was caused by the movement toward higher academic output standards, since there is a well-known relationship between the intensity of teaching/grading and articles one produces. Or you might think this is pure laziness/Arschlochkeit in action. But I think that there’s an alternative hypothesis: the law has gotten much more technical and complex over time, making it harder to test in one LISQ even a fair proportion of the materials covered in class. Compare, for example, administrative law or corporate law from a generation ago to today, and I think you will find exponential increases in the amount of caselaw, density of the regulatory statutes, and the theoretical nuance of the commentators. Since most professors feel some obligation to test what they teach, smaller, more discrete, questions become more attractive. This despite the fact that law professors are given almost no guidance on how to write exams, and (especially) how to design multiple choice or true/false questions so that they don’t mislead students.
We could test this hypothesis if we could identify an area of law that has resisted the turn toward complexity, and compared the design of old and new exams. I’m thinking perhaps property law fits the bill? Shucks, now that RAP is dead, that course is a cinch.
December 8, 2008 at 7:07 pm
Posted in: Law School (Teaching)
Print This Post
Responses (14)
German Speaker - December 8, 2008 at 7:41 pm
Arschlochkeit ist not a word. I know it is fashionable to use German terms once in a while, but please…
Hillel Levin - December 8, 2008 at 7:44 pm
I have assumed that the rise in alternative evaluation measures relates in some way to the rise of alternative teaching techniques. Some people teach differently because they believe that the “old method” was sub-optimal for students; and some people test differently because they believe that the “old method” was sub-optimal for students.
In other words, maybe it is neither laziness nor changes in the law, but rather changes in what some people think makes for good evaluation.
david hoffman - December 8, 2008 at 7:48 pm
German Speaker: don’t blame me, blame moneylaw, who’ve been trying to make the term a general one.
Hillel: I appreciate your optimism. But what kind of data supports it? I mean, I think that MC exams, in particular, are not particularly good for students, especially in law, if used without caution.
reader - December 8, 2008 at 7:58 pm
Another factor might the semester in which the exam is given–it could be by chance, but my professors gave shorter questions and multiple choice exams in the spring, when grades for 3Ls were due shortly after the exam.
Hillel Levin - December 8, 2008 at 8:41 pm
Dave–
There are really two separate questions: (1) do people THINK that alternative exam question types are good for something, and (2) is there empirical evidence to support it? My comment went to the first rather than the second.
I haven’t used MC and haven’t decided whether I will. But if I do, it will be because there are, in fact, right and wrong answers in the law, and being able to spot them is worth something in the legal profession. Equally important, the Bar uses MC extensively, and to the extent we can prepare our students for that, well, we’ve done something.
I’m not sold on MC because I think it is very difficult to come up with good questions; and if you don’t succeed in coming up with good questions, you probably just get noise.
Let me flip the question to you. Is there any data supporting the notion that people who do better on long issue spotters (or anything else, for that matter) are somehow going to be better lawyers?
Mike Guttentag - December 8, 2008 at 8:49 pm
Based on my own experience, I think the change in testing methodologies has a lot to do with who becomes a law school professor these days. I teach corporations, and would not have the slightest idea how to write an LISQ. Why? I did not come straight into law teaching from law school (or even from the practice of law). The issues that I emphasize when I teach this material come largely out of my past business experience and the scholarly debates I am engaged in. Neither is conducive to a LISQ. The same is probably true for scholars who have PhDs as well as law degrees.
ps: congrats on tenure!
Orin Kerr - December 8, 2008 at 9:21 pm
I tend to think law has much simpler and less technical to learn in the last 50 years or so. We now have many more resources and it’s much easier to find the law, so opinions are much better and much more coherent and the law is much easier to understand.
I wonder if the bigger difference is that we professors play hide the ball much less often, and if you play hide the ball you can see more of a gap between those who found the ball and those who didn’t.
A.J. Sutter - December 8, 2008 at 11:33 pm
Orin, are you being ironic in your first paragraph? I hope so, and in that case I apologize for the superfluity of the rest of this comment. I’m not a prof, but I’ve taught some occasional law-related classes, as well as prepared materials for in-house training at law firms and companies. Suffice it to say that real-life practice gives one ample inspiration to come up with baroque LISQs, if that’s what you’re aiming for. Truth is stranger than fiction. Moreover, I find the methodology of answering LISQs — at least, the issue-spotting component — is very helpful in real practice.
Tim - December 9, 2008 at 2:37 am
I just took my take-home Trademark exam today…I worked on it for 12 hours….anyway, the first portion was 22 T/F questions. The other section was a LISQ – the first 22 were incredibly easy and I probably could have done it without the book. The issue spotter wasn’t difficult either because I had the book. Seemingly the hardest parts of LISQ’s are A – the writing part, I’m not a particularly strong writer B – keeping it under the word limit – which I imagine is good practice for real life.
Nate Oman - December 9, 2008 at 11:50 am
Dave: I think that there is something to the complexity point. It seems to me that in order to have good MC you need two conditions. First, areas of the law where there are clearly right answers. Second, areas where the complexity of the problems for which there are clearly right answers is sufficiently high that the MC questions are not so easy as to not test anything. Here, I think that the UCC is a good test case. There are actually quite complex fact patterns and legal questions for which there are absoltuely clearly right answers under the UCC. Accordingly, Article 9 or Article 2 would be good candidates for MC questions. If your hypothesis is correct, we would expect to see that MC questions have been present in commerical law classes for a long time. I’ve no real evidence one way or the other, but I have heard stories from grizzled (Okay, not that grizzled) UCC profs who have been writing MC questions for years.
Tim - December 9, 2008 at 12:34 pm
My Contracts Professor, Lester Brickmen, gives a MC exam at the end of each semester of his two semester course… they are 160 questions long and use fact patters and each questions changes the fact pattern a little. And is heavily UCC laden. While you can look things up, it forces you to learn it. The “hide the ball” stuff is ridiculous…if you just ask us to find it, and more so in a pressurized situation like a test, we are bound to remember it.
Jason Kilborn - December 9, 2008 at 4:53 pm
Thank you, Dave, for explaining a phenomenon that I had observed in my own exam writing recently. Bankruptcy law has become inordinately complex after the passage of a massive reform law in 2005. After that occurred, I switched to a 50/50 MC and LISQ format. Coverage issues drove me to this point, just as you described. I’m not thrilled about the movement, but I feel better about understanding why I’ve done this–you’re right.
To Nate Oman’s point, I agree that subjects like Payment Systems, where the law is complex and the answers are clear (and usually binary–yes or no, liable or not) make for good MC tests. I have used a MC test in my Payments class for years. Yes, it is VERY difficult to come up with good, valid MC questions (but the complex computer analysis that one gets from a scantron exam is useful in testing the questions). I am less enthusiastic about my Bankruptcy MC questions, and for Secured Trans, I am less concerned with binary answers than with several-step analysis, which I have found to be significantly more difficult to test in MC format. I have a record enrollment for Secured Trans next semester, though, so perhaps I’ll consider an MC component.
Kaimi - December 9, 2008 at 8:46 pm
Err, Dave? My Wills and Trusts students’ fervent wishes to the contrary, RAP is not actually dead.
A.J. Sutter - December 9, 2008 at 10:42 pm
Seconding Nate’s observation about “areas where the complexity of the problems for which there are clearly right answers is sufficiently high that the MC questions are not so easy as to not test anything”, I note that the first half of the “patent bar exam”, which is based on facility with the Manual of Patent Examination Procedures and certainly fits his description, has been MC for many years.
Leave a Reply