1. A.J. Sutter - August 1, 2010 at 2:50 am

   1. Prof. Uelmen’s count is 6 distinct Cal Supreme Court opinions published in past 5 years, relying on (i.e., not citing in dicta ) law review articles — not six citations, as you mention of your search. Just wondering, did your quick-and dirty search conform to Prof. Uelmen’s criteria?

   2. Your criterion of usefulness seems to apply to litigation only (and indeed, only to cases that make it to an adjudicated appeal). As a transactional practitioner, I can say that I can’t recall any article from a student-edited law review that has helped me, despite my continually seeking for them, in a quarter-century of practice. (Out of an excess of caution, I may have estimated this as “one or two” in prior comments to other posts on this blog in the last couple of years.)

   3. There are also questions raised by, and questions not asked in, the Schwartz & Petherbridge article that should temper your view that “the conventional wisdom so completely wrong on this point.”
   (a) S&P don’t mention the absolute number of law review articles cited — maybe a small number get a lot of cites, just as with the statistics of websites.
   (b))They also don’t mention the proportion of published articles cited. Nor do they mention the age distribution of articles cited. (They assume that Justice Scalia is right when he says the shelf life of a law review article is 5 years, but why should they accept this anecdotal judgment when they “prove” that Justice Roberts’s anecdotal estimation is unreliable?) As Prof. Uelmen pointed out, the number of law journals has exploded in the past 20 years or so. So even assuming for the sake of argument that the absolute number of articles cited has gone up, the relative number of “useful” articles may have decreased. That alone could explain why most judges and practitioners complain.
   (c) S&P mention that a relatively small cohort of judges (< 14%) are responsible for ~ 50% of all citations. They don't mention who these judges are — might they be concentrated in particular circuits?
   (d) S&P also don't discuss the content of the articles cited, or the types of cases that tend to cite them. E.g., patent cases might correlate with the Federal Circuit, which didn't exist during the first half of the period discussed by S&P. Or article content might reflect a particular ideological shift, e.g. more weight to a law & economics approach, which might correlate with particular judges as well as explain, due to the temporal dynamics of ideological reception, a recent up-trend in cites.
   (e) Especially in light of the possibility of an ideological correlation, one also needs to ask: if it's true that law review articles are having more influence on judges despite being more oriented to theory and "law &…," is that unequivocally a good thing for society (as distinguished from being good for law profs)? What is the substantive content of what's being cited?

2. lawanon - August 1, 2010 at 3:23 am

   Your hypothesis re judges citing articles that are *not* doctrinal analysis is easily testable, and I wish you attempted to test it before posting it. Look at the six mentioned CA supreme court opinions and see whether law review references are related to doctrinal analysis or non-doctrinal research. I’d bet it’s the former, though we’ll never know until someone does the reading and reports here.

3. TJ - August 1, 2010 at 5:51 am

   One point that everyone seems to miss is that there is no shortage of doctrinal scholarship being published in law reviews. They are called student notes. Lawyers and judges who complain that law reviews are useless because they don’t contain doctrinal analysis thus (1) overestimate how useful they actually find doctrinal scholarship, (2) are ignorant about what is actually in a law review, or (3) have a rather elitist (is there a better word here?) preoccupation with the prestige of the author rather than the merit of the content.

4. James Grimmelmann - August 1, 2010 at 9:06 am

   Law professors do have something important to contribute even when they “merely outline, discuss, or analyze doctrinal areas.” We have time: time to read and time to think. Even a highly conscientious judge will receive an issue as it has been framed by the parties, and is under institutional pressure not to go off on research fishing expedition. A professor who takes the time to study an issue thoroughly can bring back fresh insights, unexpected precedents, compelling syntheses, and surprising challenges to seemingly settled authority. This kind of work can be done well or done poorly, but done well, it is institutionally valuable.

   Also, TJ’s point about student notes is excellent. Advisors: steer your students away from common topics. A student who writes about whether the DMCA is stifling creativity will say nothing new and will never be cited. A student who notices that Pennsylvania courts apply two startlingly inconsistent choice-of-law tests in insurance cases will say something interesting and may well have it picked up by the courts.

5. Corey Rayburn Yung - August 1, 2010 at 4:54 pm

   Hi A.J.,

   I am not sure I can properly distinguish the difference between “citation of” and “relying on” and I do not know how Uelman made such a distinction (if he did). I’m dubious of your dicta differentiation because I can’t imagine an instance where a law review is actually part of the holding. It seems like such a citation is always dicta (unless the facts of the case actually pertain to a law review which would be irrelevant to this discussion).

   As for usefulness, I agree that nothing I said pertains to transactional lawyers. Since I was primarily focused on judges and their collective claims of the irrelevance of law reviews, I only meant to speak to litigation.

   As to your methodological objections, you would have to speak directly to the authors. However, I wouldn’t expect them to present the data with every breakdown possible. And since some judges and commentators have argued that judges have no use for law reviews, I feel that they have effectively attacked that claim.

   Hi anon,

   Since this was not my study, I did not attempt to determine if the doctrinal citations are more or less common. I would disagree that it would be “easy” to do since it would require a substantial sample size and a lot of labor to categorize the law reviews cited. Perhaps Schwartz and Petherbridge will perform such an analysis, but as the mere author of a blog post, I didn’t want to start such a large scale project.

   Hi James,

   I did not mean to discount the value of doctrinal work. I actually wrote a purely doctrinal article about the Adam Walsh Act and it has been cited by a couple of federal courts perhaps for the reasons you cite. I only meant to point out that the shift away from doctrine might end up being more helpful to judges insofar as cases involve areas of research in which judges are not naturally experts.

   Corey

6. Mark McKenna - August 1, 2010 at 9:02 pm

   I have always been unpersuaded by the claim that legal scholarship is unimportant because it is not cited by judges (and that was even assuming, perhaps contrary to the evidence, that it was in fact true that judges were not citing law review articles). Citations in decisions are simply not a good proxy for influence of legal scholarship. First of all, our scholarship affects our teaching, and the students are exposed to the ideas in the scholarship whether they know it or not. Trends in the scholarship therefore shape new generations of lawyers. I doubt lots of judges were citing articles written by the legal realists, but certainly they were having an important impact on the law. (I suspect something similar could be said about the originalist movement). Second, in many areas of law, including my own, professors regularly write amicus briefs and/or assist practicing lawyers. Whether they are cited directly or not, the arguments in academic articles are presented to courts and shape the form of the arguments. So I just don’t see why the citation count is important.

7. Dave Schwartz - August 2, 2010 at 1:06 am

   As a co-author of the study, I wanted to chime in. First, I agree with Corey Yung’s responses to the comments. Second, I also agree with A.J. Sutton and Mark McKenna that there are additional uses of scholarship beyond citation in opinions. Our paper makes this exact point. There is also a substantial body of literature relying upon citation counts. Our paper engages the citation count literature. I believe we have provided new information to the debate, and analyzed citation data in new ways from what has been done in the past. Many of A.J.’s questions — such as what percentage of published articles are cited and what are the subject matters of the opinions citing legal scholarship — are merely possible directions for future research. In fact, we flagged several of these areas in our paper. His inquiry about the small cohort of judges who cite scholarship is easily answered. (A.J.: “S&P mention that a relatively small cohort of judges (< 14%) are responsible for ~ 50% of all citations. They don't mention who these judges are — might they be concentrated in particular circuits?"). The 10 judges who cited to scholarship in the most opinions from 1990-2008 reside in 7 different circuits, and no circuit is home to more than 2 of them.

8. A.J. Sutter - August 2, 2010 at 5:04 am

   Thanks, Corey & Dave, for your replies. My objections were less to methodology than to the inferences Corey’s post made from the results.

   The study mentions that about 3.4% of opinions cited legal scholarship in 1950-1979, and about 4.8% cited it from 1980-2008. First of all, that’s not a whole lot, in absolute terms. Neither is the the 6.21% during the shorter period of 1999-2008 — it means that > 93% of the time, the appellate courts didn’t cite scholarship. If my clients told me that 93% of my work was useless, I’d rightly feel like a failure. Of course, the same would be true if they felt that way about a considerably smaller percentage (just to put your “other uses” argument into perspective). You might reasonably argue that not all of that 93% is intended to help lawyers and judges anyway, but that makes their complaint more plausible, not less so.

   In relative terms, it’s about a 40% increase in citation between the two periods. Here’s where the issues raised in my items (a) and (b) are pertinent. Let’s first assume that the cites are evenly distributed among published articles (i.e., probability of citation is the same for each published article). It’s very difficult to establish this probability density a priori (and it isn’t very plausible anyway — otherwise there wouldn’t be “top tier” journals, etc.), but it’s a simplifying assumption that provides an easily-tested case: if the average annual rate of publication during the later period increased more than 40% compared to the rate during the earlier one, the proportion of useless articles will have increased. (Since my own alma mater has > 2x as many law journals as it did when I attended in the early 1980s, I find such a large increase in the rate of publication prima facie plausible.) If instead the citation of articles isn’t evenly distributed, then it could be a much more complicated calculation, but one possible result is that an increase in the rate of publication of < 40% can nonetheless lead to an increase in the proportion of useless articles.

   A greater proportion of useless articles would certainly justify the complaints of lawyers and judges, and yet could be entirely consistent with S&P's published data. So in fact the S&P study is inconclusive as to this controversy.

   BTW, I’m not so sure it’s reasonable to be so blasé about the uselessness of law review articles to transactional attorneys. Law school isn’t solely for the training of litigators and judges — Mark McKenna’s comment that your scholarship affects your teaching is pertinent on this score. And in fact a lot of the subject matter taught in law school is predicated on the existence of transactions.

9. Matt - August 2, 2010 at 6:58 am

   AJ- surely you don’t think there’s a valid inference from “not cited by a judge in an opinion” to “not useful for a judge in understanding an area of law.” (And, of course, something can be useful for law students and other lawyers without being useful for a judge, too.) That’s a complete non-sequitur. I don’t, of course, make the converse claim that all (or even most) legal scholarship is _useful_ for judges, but the inference you make here is completely unwarranted, as I’m sure you’ll see.

10. A.J. Sutter - August 2, 2010 at 7:06 am

    Slight correction: strike my comment from “If my clients told me that 93% of my work was useless, …,” since mathematically this is too simplistic. There isn’t any necessary connection between the percentage of appellate cases citing legal scholarship, and the percentage of sources of such scholarship that are cited. E.g., even if 100% of appellate cases in a given year cited legal scholarship, that wouldn’t necessarily mean that 100% of legal scholarship was cited; and even if 100% of legal scholarship were cited, it wouldn’t necessarily mean all cases included at least one citation.

11. A.J. Sutter - August 2, 2010 at 7:07 am

    Sc. strike from that phrase to the end of that paragraph only. Apologies for the itchy mouse fingers.

12. A.J. Sutter - August 2, 2010 at 7:54 am

    Matt, the issue in my earlier comments was Corey’s inferences from the case cite data in the S&P article. As for your point, we don’t have any way of knowing whether an article is influential or not in the way you propose. Law professors and students can think of ways in which their work might have influence invisibly, and I’m not claiming they (you) are categorically wrong. However, I’m not sure why this merits more weight than the testimony of judges and lawyers themselves about what they find useful vel non.

    I also don’t think most practitioners and judges are arguing the absolute, that law review articles are never useful, including in the diffuse way you suggest — it seems to me that the point is a more relative one, like there being so much stuff to wade through, and so much of it not informed by practical experience or concerns, that it’s getting noticeably harder to find something useful than it used to be. It’s not my wont to put things in such terms, but perhaps one could say that for practitioners the transaction costs of dealing with legal scholarship are higher than they used to be.

    Or another economic analogy: for law scholars to “prove” their own usefulness through statistics (not that S&P have done so) is a bit like economists saying householders are “better off” because per capita GDP goes up. If householders nonetheless report feeling worse off, that’s something worth listening to.

13. Corey Rayburn Yung - August 2, 2010 at 11:14 am

    Hi A.J.,

    I did not intend for my comments to be construed as an inference from the study. I was arguing in favor of one of many possible working theories. Further research would be needed to test my hypothesis.

    And I certainly do not believe that court citation is near the most significant factor in assessing the importance of scholarship. I just find it interesting that judges have been arguing that law reviews are increasingly irrelevant while their actions seem to tell a different story. I think the distance between the ivory tower and the courts is smaller than most believe and the S&P study supports that conclusion.

    As for the importance to transactional lawyers, I have no comment. Since my primary fields are criminal law and federal courts, I’m almost exclusively focused on litigation issues.

    Corey

Leave a Reply