In Defense of Law Reviews

You may also like...

11 Responses

  1. Speaking as an outsider who sometimes cites to law reviews, I think the peer review problem is more critical in interdisciplinary writing. Law and economics is the obvious candidate, but I’m seeing more and more articles published in law reviews that include statistical work. Very few law students have the necessary background to evaluate the quality of such work.

    For that matter, peer review is no guarantee that articles with methodological flaws or errors in statistical reasoning won’t make it past review, but at least there is a chance that an editor will direct the paper to someone with familiarity with the methods.

    Peer review does have its problems, but I think on the whole the double-blind review process does have some advantages, and certainly avoids the issue of professors “dump[ing] their lesser work onto their own school’s students.”

  2. Marc Roark says:

    Thanks Daniel. Your voice of reason seems quite measured. The one point I’d add is that another under appreciated point of scholarship is it allows professors to refine their voice, which surfaces in the classroom, making the student experience richer and deeper. To this end, not all benefits of the law review system can be measured as “outcomes” such as the author seems to suggest.

  3. brad says:

    The problem with post hoc peer review is that it is difficult for the reader of an article to find. Maybe if there was a big Shepard’s style warning at the top of articles that said “heavily criticized” it would work, but as it stands most bad articles are simply ignored.

    While peer review is hardly perfect (as scandal after scandal shows) at least when you read an article in a well known journal in other fields you can start with a higher prior probability that it isn’t complete nonsense.

    Also, peer review would seem to prevent the crazy system of simultaneous submission everywhere and accompanying expedite process, which is one of the oddest and most tawdry aspects of the law review model.

  4. Daniel Solove says:

    The issue is not whether a peer review system in the abstract is better than the law review system. It is whether the peer review system is worth the time and effort to implement in these days where there isn’t a scarcity of publishing opportunities.

    Quite frankly, I’d rather spend my time reading pieces I know are good and working on my own scholarship than doing a lot of peer reviewing. I can tell rather quickly if a piece is going to be any good, and I can quickly move on if it isn’t. And if I don’t know whether a piece outside my field is any good, I can ask professors in that field whom I trust or see what pieces they recommend on Twitter and in blogs, etc.

    I don’t want to spend tons of time writing detailed reviews of pieces for peer review, as even with a bad piece, considerable time must be spent writing the review. All this is lost time spent with a sub-par piece. Ordinarily, I would quickly brush aside these pieces and move on with more important things, but with peer review, I must write a review. Life is short, and I’d rather not spend my time this way, and I bet many law professors think the same. So we live with an imperfect law review system. We save a lot of wasted time, and . . . yes . . . the system is imperfect, but getting perfection isn’t worth the price.

  5. phil says:

    > Life is short, and I’d rather not spend my time this way, and I bet many law professors think the same.

    That’s kind of a selfish answer. I appreciate the honesty, but still. Peer review in other fields is considered one of the things that responsible academics do. Annoying and boring perhaps, but it helps separate the wheat from the chaff, and multiplied over all the people who have to do their own personal peer review every time they are looking for an article saves net time.

    I see similar attitudes towards grading — particularly good feedback on written work, and towards committee and other institutional work. On top of that, law professors are better compensated as compared to peers in social sciences departments.

    It all makes it very hard to sympathize when law faculty comes under attack in the popular press or in university politics.

  6. Daniel Solove says:

    Phil,

    I think you’re missing my point. Peer review would matter if there were a scarcity of publishing opportunities. But there isn’t. So why bother putting so much time into front end review when there isn’t scarcity anymore and anyone can publish? Why not devote more time to other things?

    The argument that responsible academics just do peer review doesn’t really answer my point about why pre-publication evaluation is important and why post-publication evaluation won’t be sufficient.

    People can readily find the wheat from the chaff by looking at commentary on already-published works. Basically, being published just doesn’t mean much anymore, and I being published never was a particularly precise proxy for being good. It has been a proxy better than random, for sure, but not a precise one, and not one I’d rely heavily upon — even with peer review.

    I think that most debates about peer review seem to be stuck in a bygone mentality where publishing opportunities were scarce. We’re living in a very different world now.

  7. Bob Lawless says:

    As a law professor, I am much more sympathetic toward the “pro peer review” comments. Properly done, peer review is not just a sorting mechanism but instead helps improve the research. My own papers are stronger when they go through peer review, both because of the comments that go with the peer review and because I know when preparing the paper that it will receive more careful scrutiny.

    The biggest challenge for peer-review in law is the normativity of most legal scholarship. Absent agreement on methodology, it is not apparent to me what it means to peer review a normative claim.

  8. Orin Kerr says:

    Dan, I agree with a lot of what you’re saying, but I think you’re overstating the case. There is a scarcity of publishing opportunities in top journals, and we all know that journal placement still matters. To be sure, people *should* read the article and decide for themselves. But a lot of people don’t, and the association with a journal still colors associations with an article. And we see this from decisions scholars make: There’s a reason you decided to publish your recent and forthcoming articles are in the Harvard Law Review, Columbia Law Review, and California Law Review and not secondary journals at low-ranked law reviews.

  9. Adam Levitin says:

    I want to echo Bob Lawless’s point on the problem of peer review of normative scholarship.

    I’d also observe that peer review in small fields just doesn’t work.

    Finally, I’d suggest that these days publication occurs when a work is posted to SSRN, not when it is accepted in or appears in a journal. SSRN posting is literally when the work is made public. The formal publication in a journal is a finishing process (and the inane level of footnoting actually has some virtues), but we’re really in a self-publishing world now. I don’t know anyone who reads actual print law reviews-the SSRN abstracting journals are today’s advance sheets.

  10. Considering that it’s easy and painless to upload a paper to SSRN (or a similar site), the law review serves a function in the sense that a least a paper is screened. I’ve seen some papers accessible on SSRN (I pay for SSRN access … out of my own pocket) that were so absurd — and often biased — that they would never see the light of day in a law review.

    For me, both SSRN rapid publishing and law reviews serve a purpose. Maybe if SSRN was more selective — and sometimes they are — I might opt for SSRN over law reviews. For now, I’ll stick with both.

  11. Larry Rosenthal says:

    I am happy to agree that peer review is far from a perfect system. It may also be true that peer review is difficult to apply to normative legal scholarship — but then again one of the complaints about legal scholarship is that so much of it has a “this I believe” quality. It is for this reason that legal scholarship often tries to ground itself in empirical claims. For just this reason, I think we need to worry that legal scholarship is lacking in adequate quality control because of the lack of rigorous peer review. The process of post-publication review is rather a hit-or-miss thing. It depends on the willingness of other scholars to take the time to scrutinize critically and with rigor the scholarship of others. I wonder how often this happens.

    William Stuntz was one of the most prolific and cited criminal procedure scholars in recent decades, but his work often contained broad empirical generalizations with questionable empirical support — the kind of work that might be questioned by the editors of a peer-reviewed criminology journal, but which law review editors were evidently eager to publish, and legal scholars were sub subsequently eager to cite. For years (really decades), the process of post-publication peer review described in this post seemed to smile on Professor Stuntz’s claims — not because anyone systematically examined the evidence supporting these claims, but because they offered support for others who wished to make arguments that were supported by these same claims. In other disciplines, however, empirical claims are not accepted merely because someone has already made them — they require persuasive supporting evidence. Indeed, criminologists never seemed to take Stuntz especially seriously, but law professors did.

    In their recent reviews of Professor Stuntz’s final book largely recapitulating Stuntz’s law review articles, Stephen Schulhofer and Donald Dripps have dismantled a great many of Stuntz’s central empirical claims. But I wonder whether Stuntz’s work ever would have enjoyed the prominence that it attained if law reviews were more like criminology journals. The more that legal scholars do the work of other disciplines, the less defensible is our system of peer review, or so it seems to me.

    Larry Rosenthal
    Chapman