Is a Good Exam Broad or Deep?
posted by Dave Hoffman
It should be obvious that writing exams is much more pleasant than grading them. Still, nagging questions dog the enterprise, making it difficult to get the job done. The hardest, for me, is whether to test broadly (i.e., a fair sample of every issue the course covers) or deeply (a few issues, with lots of factual nuance and policy twists). The problem is that I think that students and teachers have different preferences. Should I trump students’ preferences, work to debias them, or satisfy them?
Maybe this problem is all theory, and no evidence. Let’s take a poll, to get some “data” on point.
April 23, 2007 at 4:12 pm
Posted in: Law School (Teaching)
Print This Post
Responses (11)
Steve - April 23, 2007 at 5:12 pm
I’m a student, and I went for very broad for two reasons. First, it takes some amount of randomness out of the process. Very specific exam questions will favor those who studied the specific topic better than others, but that won’t necessarily reflect proficiency in a course. Second, a broad exam is more likely to encourage students to study in a way that they’re likely to remember it after the test. When I cram 150 specific cases, I don’t remember them a week after the test. When I study and practice general policy and doctrine as it applies to varied fact settings, I remember the policies, etc. months later.
Ben - April 23, 2007 at 6:11 pm
Student and I went with broad. It just strikes me as unfair when I spend 20% of my studying time on the 20% of the material covered in class that doesn’t even make it onto the exam. Exams are capricious, but if we are going to pretend that my grade represents how well I know the subject, we should probably test all of it.
I like the inclusion of papers as a choice, aren’t papers the ultimate example of deep over broad?
Mike - April 23, 2007 at 7:32 pm
I would like some feedback about something related to exams. Do most law schools still base an entire semester’s grade on one exam? We had our contracts II final this a.m. and it was a 3 hour exam and the only test we took all semester. I probably invested somewhere around 200 hours on contracts this semester (class time, class prep, study time, etc.). Then, all of that comes down to one three-hour exam! Is this really the best approach? So, I guess my real preference for exams would be to have 2 or even 3 exams that go very deeply on the issues. This strikes me as a much better way to approach the material. But, it would also add quite a bit of work to professors that have to make and grade the exams.
Seth R. - April 24, 2007 at 11:22 am
If your aim is to train future law professors/federal clerks/federal judges -
Go deep.
If your aim is to produce future lawyers -
Go broad.
Because honestly, you don’t need a ton of in-depth analysis in a nuts and bolts small practice (which is where most of us end up). For a mainstreet lawyer, knowing which judge is a pain in the keister is infinitely more useful than correct analysis of caselaw.
Jeff Lipshaw - April 24, 2007 at 2:48 pm
It looks to me by the results so far that preferences don’t differ at all. It appears to run 3:1 in favor of broad for both writers and takers.
dave - April 24, 2007 at 2:58 pm
Yes, so far, it is turning out differently than I anticipated.
Rachel Godsil - April 24, 2007 at 9:14 pm
I think there is a middle ground between broad and deep. As an exam writer (and of course former exam taker), I aim for testing a few topics deeply through essay questions and then reasonably broad coverage with several short answer questions. Different areas of law lend themselves to different modes of testing. For example, I teach property — real estate transactions don’t require deep testing, takings does.
Edward Swaine - April 24, 2007 at 11:33 pm
Good question; not sure the data is good. I think the phrasing may have something to do with it. In your introduction, you contrast “fair” sample with “policy twists.” And you have “very broad” versus “hits only a small percentage of the course but crushes it” in the poll. What if the poll, for example, were “Tests shallowly on every single blinking issue mentioned” versus “Tests fewer issues allowing students to show depth and mastery”?
Mariane - April 25, 2007 at 12:17 am
The best exams I had in law school were those where the professor a) touched on absolutely everything she taught (I took one professor three times for that reason), and b) allowed us to bring in whatever relevant materials we deemed necessary to adequately answer the questions. I never was going to be a law professor, and I ran into quite a few law professors who were out of touch with their students’ needs as future practicing attorneys.
Broad exams allow students the ability to showcase thier mastery of the material in whatever way they individually see fit to do so. Those who study for depth can still do that, and those who just want a basic overview of the material are also well-served.
Dave, you indicate in the post that you thought the preferences would fall out differently than they did. Which group were you surprised by?
Justin - April 26, 2007 at 3:54 am
I am a student, and I tend to feel that deep tests are not used enough.
Most strong analyst types or natural test-takers can pick apart a weakly assembled, broad test and prove they are good at taking tests, while knowing little about the subject. I have often found myself in that category. That might be fine for some subjects and fields (and for identifying people who have seemingly strong instinctive decision making abilities, perhaps), but as I move up the hierarchy and expected level of competency, I expect to actually have to know something, and the only way I can see that happening is when a greater reliance on depth: either complex essays, or papers — even if the majority of the course is not exercised in the test. The assumption for relevance of a deep test being that the answers and the arguments will be critically evaluated based on the expected ability and knowledge, outlined criteria, and spirit of the course. Shouldn’t a student theoretically be familiar with all aspects of the course, otherwise, why include the topic in the course in the first place if it is not important enough to be a potential test item?
Dave Hoffman - April 26, 2007 at 11:53 pm
Mariane. I’m surprised that more professors wouldn’t prefer to test with depth rather than breadth. It seems like a much better way to seperate out the best students in the class. That said, I understand why students want broad tests.
Ed, you are right: the question framing likely distorts the results. But not too much, I think, as it confirms an anecdotal sense I have. (It is comforting when data confirms intuition; it is sample error when it does not.
Mike, I know of no professor teaching a first year doctrinal course who tests more than once a semester. Maybe it happens somewhere, but, as you said, it would seem to require graduate students to help with grading.
Leave a Reply