A Modest Defense of Law Reviews
posted by Nate Oman
It is a pretty common place observation that one of the virtues of markets is that they manage to aggregate a great deal of disaggregated information. Obviously, folks disagree rather vehemently as to how effectively markets do this, but most people, I suspect, would admit that the market is frequently smarter than particular actors within the market. The same is true, I think, of law reviews.
The classical critique of law reviews is that they are staffed by dumb students who don’t know anything. Obviously, there is a lot of truth to this. However, I think that the case for the incompetence of law reviews can be overstated. While obviously no law review is doing a perfect job, citation studies suggest that the article selection process is far from random. All things being equal, articles that appear in top journals get cited more often than articles that don’t appear in top journals. Of course, it may be that law professors are simply dolts who are taking the name of the journal as a signal of quality. However, it seems at least equally probable to suppose that the law reviews — or at least the top law reviews — are identifying important legal scholarship considerably better than the classical critique suggests.
The mistake of the classical critique, in my opinion, is that it assumes that the only meaningful difference between law reviews and other academic journals is peer review. There is, however, a second and equally important factor: law reviews allow multiple submissions. This fact, coupled with the expedited review process, means that law reviews in effect compete with one another for pieces. Hence, you have a system in which a lot of smart but basically clueless students are desperately trying to find good articles. They are grabbing their information from lots of sources: discussions with friendly professors, a quick review of the literature on Westlaw, perhaps even blogs. None of these sources is perfect, and none of them amounts to any sort of expertise. Nevertheless, student editors are not making their choices at random. They are acting on the basis of information. When they act by making an offer, this places pressure on the “price” of the article by providing an additional reason for other reviews to consider it. The result is that taken as a whole, the system of multiple submission and expedited review will tend to aggregate information.
Peer review, in contrast, relies on individual expertise rather than the institutional aggregation of information. Indeed, the process of peer review is designed so as to insulate the peer reviewers from institutional pressure. They don’t compete with other journals. The process is slow. They are deprived of signals about what other people think of the article. Obviously, there are a lot of virtues to this system, and I suspect that in a lot of areas — sophisticated empirical research, for example — it dominates the law review model. However, the important point is to see that it is a model that relies on the conscious and personal aggregation of information, and that this is by no means the only way that information can be aggregated.
Does this mean that the law review system is perfect? Far from it. Obviously, factors such as reputation and law school letter-head have a huge impact on the process. Some of this is probably not such a bad thing. It is not as though reputation and law school letter head tells you absolutely nothing about the possible quality of the piece. Still, I have submitted articles as a practicing attorney and I am sure that there is bias against such pieces in the selection process. As an editor, I reviewed more than one piece that no one on the committee would have looked at but for a big name author. Law reviews, however, are not as bad as people argue they are.
The oddest thing of all is that in a sense it shouldn’t matter. Suppose that we believe that peer reviewed publications are decisively better signals of quality. Who cares? Presumably, the main consumers of academic publications are academics. In other words, they are people who ought to have the expertise to independently assess quality, and therefore don’t require the signal provided by where a piece is published.
November 9, 2005 at 4:54 pm
Posted in: Law School, Law School (Law Reviews)
Print This Post
Responses (9)
Joe Liu - November 9, 2005 at 5:35 pm
I suppose the choice of which structure best aggregates information depends on what kind of information you are trying to produce. If I want to know the best treatment for a cold, I might surf the internet for the conventional wisdom. If I want advice on whether to undergo a very complicated surgical procedure, I will be calling on an expert. Law reviews may do a decent job of sorting legal scholarship into broad categories, but not as good a job of distinguishing between solid and truly excellent scholarship.
I think your final point actually underscores why people often object to the existing system. The existing system would not be problematic if nothing really hinged on placement, since, as you note, experts in the field know what is a good article regardless of where it is placed. Yet I think it’s true that, when it comes to tenure and related decisions, people often rely too much on placement as a proxy for quality.
Finally, is SSRN an example of an even more efficient market? Everything gets published. Then people vote with their feet (or mice, as it were). I suppose there are distortions here as well (based on links to blogs, subject matter, etc.). But what if SSRN could track numbers of downloads from legal academics? Or academics within a particular discipline? Maybe the end result will be an Amazon-like rating system (”I give Sunstein 3.5 starts …”).
Nate Oman - November 9, 2005 at 6:37 pm
Joe: Do you think that peer review would do a better job of picking the difference between the solid and the truely excellent scholarship? I can imagine that they might, but it is by no means obvious. Certainly, citation studies (which admittedly are a really really bad proxy for quality) suggest that most of what gets published in peer reviewed locales gets ignored. Certainly, I have done enough research in peer reviewed journals (philosophy & economics) to notice the tendency of some authors to publish the same article several different times, etc.
I suppose that any time you set up a system where some marker is going to act as a proxy for something else, you will have incentives to game the system. (Writing a program to download your pieces from SSRN, etc.) Certainly SSRN aggregates information better than say posting working papers on your personal website. I doubt that it is perfect. I suspect that at the end of the day, there is no way of avoiding the need to read a paper and access its quality for oneself.
If I am right about institutional aggregation of information, however, law profs can engage in a little less hand wringing about law reviews. Obviously, SOME hand wringing is still in order, but I think that often times the criticism fall into two traps. First, they ignore the extent to which law reviews often get it right. It is certainly not random or at least totally random. Second, they often have an unrealistically idealized picture of what the peer review process is like.
Kaimi - November 9, 2005 at 7:20 pm
Nate,
Are you by chance referring to some of the conversation taking place in this thread? http://www.theconglomerate.org/2005/11/law_school_repu.html
Because if not, your post is awfully coincidental.
Nate Oman - November 9, 2005 at 7:51 pm
Kaimi: The Conglomerate thread is a good discussion, but my post was actually prompted by a discussion that I had with Dan and Professor Litowitz somewhat strident attacks on law reviews and the law professoriate in general on your thread.
Aaron Wright - November 9, 2005 at 10:47 pm
I think there is another benefit to Law Reviews, which is that it helps train law students to write legal scholarship. If you have an editorial position on a law review or comprable journal, the editing process teaches an aspiring professor many things–common structures for articles, appropriate citation conventions, etc. This doesn’t even include the note process.
I think that these are tangible benefits that also get overlooked.
Joe Liu - November 9, 2005 at 11:48 pm
Nate: I guess I may be referring more to review by peers than peer review. If I want to know which articles in a particular area are truly path-breaking, I would ask folks in that area whose judgment I trust. And I do think that they would do a better job making that determination than the law reviews could, even collectively.
Now it may be that the peer review *process* has defects in it that undercut the benefit of expertise. If the folks doing the peer review are not themselves good judges of quality, if the process is politicized, or if the people don’t make the effort, that’s problematic. So I guess I don’t really know enough about the peer review process, as it is actually implemented, to make a comparative judgment. (I may be guilty of idealizing the process).
But more generally, I guess I’m a bit more skeptical that the collective “wisdom of the law review crowd” is always enough to overcome the substantial advantages of experts when trying to determine which works are truly excellent. While it’s true that the law review system as a whole may aggregate a lot of information, I think we still need to look critically at the quality of that information and compare it to the information available to a smaller group of experts.
That being said, I agree with your general point that the critique of law reviews can be overstated. I’m actually relatively content with the existing system, since pretty much everything gets published and, as you say, experts in a particular area know what’s good regardless of where it’s published.
Christine Hurt - November 10, 2005 at 9:49 am
Aaron — were you an editor at Cardozo? Were you my editor? If so, what a small world! If not — gosh there are a lot of Aaron Wrights in the world.
Concurring Opinions - November 29, 2005 at 2:07 pm
History of the Book
Folks here at concurringopinions have been talking a lot about books recently–Nate Oman’s had posts on the appeal of law books (particularly old ones) and law reviews and Dan Solove’s posted about the open library. I find student-edited law…
Concurring Opinions - November 29, 2005 at 2:08 pm
History of the Book
Folks here at concurringopinions have been talking a lot about books recently–Nate Oman’s had posts on the appeal of law books (particularly old ones) and law reviews and Dan Solove’s posted about the open library. I find student-edited law…
Leave a Reply