If people were confident about how the the rankings were done, then that influence might be acceptable.  But most faculty do not think that the methodology used by U.S. News is sound.  I’ve noted before that they give no weight to [...]]]> One of the most common complaints that you hear from law professors and deans is that the U.S. News and World Report rankings exert too much influence over legal education.  If given a choice between doing something to boost its ranking or doing something to help students, the incentives for a school are heavily weighted towards boosting the ranking.  This is true because rankings are widely publicized and provide a simple way for prospective students, alumni, and other interested constituencies to evaluate law school performance.

If people were confident about how the the rankings were done, then that influence might be acceptable.  But most faculty do not think that the methodology used by U.S. News is sound.  I’ve noted before that they give no weight to student or faculty diversity, and Malcolm Gladwell wrote an essay observing that the rankings do not take cost-effectiveness into account (which is especially strange in this era).  Granted, coming up with a standard that everyone would agree upon is impossible, but we can do better.

What is to be done?  The answer to monopoly is competition.  We need other organizations to conduct law school rankings. This would give people more information, especially if the alternatives explicitly take factors into account (e.g., cost) that are absent from the U.S. News rankings. It would also diminish the power of any single organization or person over law schools, and make gaming the ranking system far more difficult.

No single school can be trusted to do this for conflict-of-interest reasons, but there are plenty of other candidates.  The ABA and the AALS are two obvious ones assuming that no other commercial outfit wants to compete with U.S. News.  Or, dare I say it, a consortium of law blogs could organize and then disseminate these rankings for free.  It’s time to stop whining about U.S. News and start doing something to give schools better incentives to improve legal education.

Sorry for the blogging hiatus.  I’ve been writing.  I’m sorry also to have missed the latest NYT attack on legal education — in the form of a misleading hatchet job on NYLS.  The article – one of a shoddy series by David Segal – struck an academic nerve already made sensitive by Justice Roberts’ dismissal of legal scholarship.

Of course, arguments about law school’s worth and scholarship’s consequence are evergreen – they drive blogging traffic and comments & promise to motivate engagement between blogs by practicing lawyers and the academy.  But quite often, unfortunately, these discussions go nowhere.

On law professor blogs, there’s a tone of tetchy defensiveness: “the market tells us that we’re worthwhile – [...]]]>

This guy has seen the same debate so many times it broke his back

Sorry for the blogging hiatus.  I’ve been writing.  I’m sorry also to have missed the latest NYT attack on legal education — in the form of a misleading hatchet job on NYLS.  The article – one of a shoddy series by David Segal – struck an academic nerve already made sensitive by Justice Roberts’ dismissal of legal scholarship.

Of course, arguments about law school’s worth and scholarship’s consequence are evergreen – they drive blogging traffic and comments & promise to motivate engagement between blogs by practicing lawyers and the academy.  But quite often, unfortunately, these discussions go nowhere.

On law professor blogs, there’s a tone of tetchy defensiveness: “the market tells us that we’re worthwhile – just look at the continuing number of lemmings pounding at the gate!”, or “of course our scholarship is consequential, let’s count the citations”; or, “no one ever promised that a JD was a job guarantee!”; or, “what’s their BATLS?” [The last is a truly obscure negotiation joke if there ever was one.]

For reporters, it feels like the scene in the Wire when they are talking about what to cover in the coming year. Sure, you could talk about complexity and globalization and economic markets and the changing nature of legal practice.  Or you might talk about the relationship between ABA regulation, thoughtless paternalism, and resulting distributional inequalities in education.  But that’s a set of sprawling stories – lacking an obvious villain to muckrake.  Rather, then, the news blames the dickensian aspect of law schools.  Reporters write articles that stir the pot, but aren’t recognizable to insiders, making them less likely to actually motivate change.

Last, not least, the practicing lawyers often articulate resentment toward ivory tower academics who ignore the realities of “trench lawyering”. (This happens even when the “academics” in question are actually practicing lawyers.)  Basically: impractical law professors versus practical lawyers.

Why does this “debate” feel so tired?  I have a partial hypothesis: because we ignore history. I had a great research assistant, Alex Radus, collect quotes about the ferment about legal education in the 1930s-1940s.  (Which is highlighted in Prosser’s famous 1948 speech to Temple’s law faculty, Lighthouse No Good.“)  After the jump, you’ll see some fantastic quotes from that era and before, which remind us that “what has been will be again / what has been done will be done again /there is nothing new under the sun.”

Starting pretty early, Blackstone wrote in his Commentaries that there are

“some who warmly recommend dropping all liberal education as of no use to students in the law; and placing them, in its stead, at the desk of some skilled attorney in order to initiate them early in all the depths of practice and render them more dextrous in the mechanical part of the business . . . . If practice be the whole he is taught, practice must be the whole he will ever know; if he be uninstructed in the elements and principles upon which the rule of practice is founded, the least variation from established precedents will totally distract and bewilder him . . . “

By contrast (and leading the charge) there was Jerome Frank [Why Not a Clinical Lawyer-School?, 81 U. Pa. L. Rev. 907, 909 (1932)]

“Students trained under the Langdell system are like future horticulturists confining their studies to cut flowers, like architects who study pictures of buildings and nothing else. They resemble prospective dog breeders who never see anything but stuffed dogs. And it is beginning to be suspected that there is some correlation between that kind of stuffed-dog study and the over–production of stuffed shirts in the legal profession  …

[T]he acceptance of the Langdell-Harvard method meant that the university law school teachers, with few exceptions, were those who had never practiced or practiced for only a brief interval. It is probably true that a majority of the teachers in some of our university law schools have never met or advised a client, consulted with witnesses, negotiated a settlement, drafted a complicated contract, lease or mortgage, tried a case or assisted in the trial of a case or even written a brief or argued a case in an upper court.”

What is to be done?  Harold D. Lasswell & Myres S. McDougal summarized the possibilities: [Legal Education and Public Policy: Professional Training in the Public Interest, 52 Yale L. J. 203, 206 (1943).]

“Proposals for escape from all of this confusion and inadequacy have, of course, been legion . . . . Lecture versus case; large class versus small class; seminars versus courses; group work versus individual work; specialization versus “well-rounded” training; vocational training versus cultural training; Bills and Notes (substitute any course) in the second year versus Bills and Notes in the third year; prescribed pre-legal training versus eclecticism; three-year curriculum versus four-year curriculum—these and many similar questions have been raised like quills on a startled porcupine.”

Shucks, adapting law school was all they talked about!  [Roscoe B. Turner, Changing Objectives in Legal Education, 40 Yale L. J. 576, 576 (1931)]

“It may be said without much question that there is more activity in the law school world today than there has been at any time within the last generation or two. Things are in a state of flux—the culmination of a long period of suggestion and counter-suggestion with little change. The addition of new courses to the curriculum, the general re-arrangement of existing courses to admit various types of non-legal materials, new approaches in legal thought, new ideas concerning teaching methods, the advent of fact research, and a dawning awareness of the existence of other social sciences than law, all testify to a rapidly changing world.”

But maybe there was no problem at all. [Sidney Post, Continuing Education of the Bar, 59 Harv. L. Rev. 694, 695 (1946]

“The law schools need offer no apologies for not preparing men for immediate practice … They have their own job, and should concentrate on doing it better. Academic instruction by professional teachers is the best method yet found for imparting certain kinds of knowledge.“

Or maybe there was: [Elliott E. Cheatham, Legal Education – Some Predictions, 26 Tex. L. Rev. 174, 180. (1947)]

“The United States is the only country of the Western world where a man can be admitted to the bar without having had any contact with, or any real knowledge of, the active profession.”

Finally, of course in the 1930s the job situation for graduates was pretty grim.  [Charles E. Clark & Emma Corstvet, The Lawyer and the Public: An A.A.L.S. Survey, 47 YALE L.J. 1272 (1938)]

 “At its annual meeting in 1933 the Association of American Law Schools considered at length the economic condition of the Bar . . . Among lawyers there was some difference of opinion about the present situation of the Bar. One man, himself successful, remarked that most lawyers could make a living if they were not lazy, and another that there were not too many lawyers, not enough, in fact, if conditions became better. But the general opinion indicated anxiety: lawyers were having a hard time—there was little work for the young lawyer—there were too many lawyers—a man couldn’t get started unless his forebears had been lawyers—banks and trust companies were taking away the business—and even collection agencies obtained powers of attorney to represent clients in the small claims court.  . . .  Many remedies were suggested by the lawyers interviewed. Real estate men and banks should not do so much legal work. Possibly the lawyer should be paid a certain amount by the state and do minimum work for people at large. Average earnings should be published, so the public could see they are not large. The law should be changed so that lawyers must certify that income figures are taken from records. A law might be passed that no legal instrument is legal unless a private lawyer attaches his signature. There is too much talk about crooked lawyers. It is all right to go after the crooks, but to talk and do nothing gives the public the impression that all lawyers are crooks. The Bar should keep its skirts clean, but does not. It should disbar those that need it, instead of letting them degrade the entire Bar. And finally a feeling that the public regarded lawyers as unreliable, and lumped the good with the bad, led many to be in favor of advertising, traditionally so at variance with the profession’s ethics.

The lawyers’ earnings are low, and there is a great deal of legal work undone which might become what the economists call ‘effective demand’ under other conditions. “

The Texas Bar put it starkly, and blamed law schools for pumping out graduates [Bar Section, Resolution Adopted By Dallas Bar Association, 10 TEX. L. REV. 326 (1932).]:

 “We your committee on Legal Education beg leave to submit the following report:

We believe that no part of our machinery for administering justice in Texas is more urgently in need of immediate overhauling than our system of admitting new members to the bar. Under the existing laws and the rulings of our Supreme Court, the state is being flooded with young lawyers from every state in the union, many of them with little preparation for the duties that devolve upon them as the guardians of the life, the liberty, and the property of the citizen…

Before going into the details of the situation that confronts us, your committee believes it wise to lay down a few general principles and to make some observations of a general character …

There are too many lawyers in this country. That this is true is evident to anyone at all familiar with the facts. In proportion to population, there are more than twice as many lawyers in the United States as in Great Britain, and nearly five times as many as in either France, Germany or Holland. If all our lawyers were busy and rendering a useful service, there would be little reason to complain, but many of them are not busy. On the contrary, many of them are on the margin of starvation and are constantly tempted to foment litigation, to indulge in legal blackmail and to engage in other reprehensible and illegal practices. It is not merely a waste of man power. It is a prolific source of injustice and social corruption.

“Such are the standards, or the lack of standards, for admission to the bar of Texas. Our bar, already overcrowded, is held out as an asylum for the lame, and the halt, and the blind from the law schools of this country. And they are coming. ”

You gotta love those Texas lawyers!

Legendary computer scientist Brian W. Kernighan (co-author of the classic textbook on the C programming language) wrote a delightful editorial on rankings last fall:

In the 1980s, statisticians at Bell Laboratories studied the data from the 1985 “Places Rated Almanac,” [...]]]> Paul Caron brings news of the ranking system from Thomas M. Cooley School of Law, which pegs itself at #2, between Harvard and Georgetown. Caron calls it “the most extreme example of the phenomenon we observed [in 2004]: in every alternative ranking of law schools, the ranker’s school ranks higher than it does under U.S. News.” I just wanted to note a few other problems with such systems, apart from what I’ve discussed in earlier blog posts and articles on search engine rankings.

Legendary computer scientist Brian W. Kernighan (co-author of the classic textbook on the C programming language) wrote a delightful editorial on rankings last fall:

In the 1980s, statisticians at Bell Laboratories studied the data from the 1985 “Places Rated Almanac,” which ranked 329 American cities on how desirable they were as places to live. (This book is still published every couple of years.) My colleagues at Bell Labs tried to assess the data objectively. To summarize a lot of first-rate statistical analysis and exposition in a few sentences, what they showed was that if one combines flaky data with arbitrary weights, it’s possible to come up with pretty much any order you like. They were able, by juggling the weights on the nine attributes of the original data, to move any one of 134 cities to first position, and (separately) to move any one of 150 cities to the bottom. Depending on the weights, 59 cities could rank either first or last! [emphasis added]

To illustrate the problem in a local setting, suppose that US News rated universities only on alumni giving rate, which today is just one of their criteria. Princeton is miles ahead on this measure and would always rank first. If instead the single criterion were SAT score, we’d be down in the list, well behind MIT and California Institute of Technology. . . . I often ask students in COS 109: Computers in Our World to explore the malleability of rankings. With factors and weights loosely based on US News data that ranks Princeton first, their task is to adjust the weights to push Princeton down as far as possible, while simultaneously raising Harvard up as much as they can.

Kernighan has also recently given talks on innumeracy, describing how easy it is to distort important debates with misleading or false numerical indicators.

So it’s clear that ranking systems can either be structured to produce certain results, or provoke gaming once they are structured. Perhaps only a diversity of rankings can solve that problem. But as James F. English shows in his book, The Economy of Prestige, in many cases the “alternative rankings” must migrate toward the opinions of the establishment rankings, or risk irrelevance. Network power makes it difficult to break out of the pack, however worthy the effort may be.

We can at least be thankful that the extant ranking criteria are public, so we can detect arbitrary weighting. In other spheres of life, ranking systems are secret—a problem I’m exploring in a book I’m writing called The Black Box Society. For an entertaining example of a secret scoring system, check out this article on the “web’s social scorekeepers:”

People have been burnishing their online reputations for years, padding their resumes on professional networking site LinkedIn and trying to affect the search results that appear when someone Googles their names. Now, they’re targeting something once thought to be far more difficult to measure: influence over fellow consumers.

The arbiters of the new social hierarchy have names like Klout, PeerIndex and Twitalyzer. Each company essentially works the same way: They feed public data, mostly from Twitter, but also from sites like LinkedIn and Facebook, into secret formulas and then generate scores that gauge users’ influence. Think of it as the credit score of friendship or, as PeerIndex calls it, “the S&P of social relationships.”

Zach Bussey, a 25-year-old consultant, started trying to improve his social-media mojo last year. “It is an ego thing,” says Mr. Bussey, who describes himself as a social-media “passionisto.” One of the services he turned to was TweetLevel, created by public-relations firm Edelman. It grades users’ influence, popularity, trust and “engagement” on a scale of 1 to 100. He decided to try to improve his score by boosting the ratio of people who follow him to the number he follows. So he halved the number of people he was following to 4,000. His TweetLevel score rose about 5 points and his Klout score jumped from a 51 to a 60.

The Klout rankings remind me of Avvo rankings for lawyers (which I discussed in this book). I can’t say too much about them because their algorithms are secret. But I think any system that ranks Justin Bieber (K=100) over Lady Gaga (K=90) is inherently suspect.

What do these ranking games teach us? Perhaps the key is to “choose the right pond,” or find the niche one is best suited for:

Photo Credits: Leo Reynolds; gordasm.

Sayeth Kaplan:

1,383 aspiring lawyers who took the October LSAT . . . [were] asked “What is most important to you when picking a law school to apply to?” According to the results, 30% say that a law school’s ranking is the most critical factor, followed by geographic location at 24%; academic programming at 19%; and affordability at 12%. Only 8% of respondents consider a law school’s job placement statistics to be the most important factor. In a related question [...]]]> For those of you who had any doubts, our friends at Kaplan have just confirmed it:  Aspiring law students care more about law school rankings than anything else, including the prospects of getting a job, quality of program, or geography.

Sayeth Kaplan:

1,383 aspiring lawyers who took the October LSAT . . . [were] asked “What is most important to you when picking a law school to apply to?” According to the results, 30% say that a law school’s ranking is the most critical factor, followed by geographic location at 24%; academic programming at 19%; and affordability at 12%. Only 8% of respondents consider a law school’s job placement statistics to be the most important factor. In a related question asking, “How important a factor is a law school’s ranking in determining where you will apply?” 86% say ranking is “very important” or “somewhat important” in their application decision-making.

Mystal at ATL expresses shock–shock!–that potential law students could be so naive. Surely, he fairly observes, they should care most about job prospects.

Yes, that would be true if they were rational.  Yet, we all know from the behavioral literature that we apply a heavy discount rate to long-distance prospects.  How much can I or  should I care today about what may happen 3 (or 4) years from today?

If you think about it from the perspective of any law school applicant today, the one concrete thing they can lock onto that has present value is the school’s ranking:  It is simple, quantified, and–perhaps most important–tauntable.  No one’s face burns with shame because their enemy (or friend)  got into a law school with a better job placement rate.  Jealously and envy–the daily diet of anxious first-years–are driven by much simpler signals:  Is mine bigger (higher) than yours?

This is not to defend the students who place so much faith in numbers that have repeatedly been shown to be incredibly stupid.  It just means that Kaplan’s survey (and I have not seen the instrument or data) makes intuitive sense.

Which leads to me to offer two modest (and probably unoriginal) proposals:

1.  Perhaps the  LSAT itself should include some behavioral assessment portion that mirrors the Kaplan survey.  After you’re done solving logic puzzles such as whether Aunt Jean wore a blond wig to dinner based on Wisconsin’s BCS ranking, you should be asked what sorts of things matter to you in a law school.   As the Kaplan survey shows, the answers give insight not only into market preferences (for better or worse) but, imho, the judgment of test takers: Demerits for poor judgment if you answered as most of these students apparently did.  Even if you couldn’t quantify and rank the responses (and, really, how could you?), it would tell schools something about the kind of students they’re getting.

We know that behavioral testing of all sorts is important in a variety of fields.  Having taught at least one student who went postal (not my fault, I was assured), I think law schools would do well to have some insight into the psychological fitness of their students. To my knowledge, that’s not something the analytic grind of the LSAT measures.

2.  Why isn’t there competition among ratings?  We know that in a world of J.D. Powers, Kiplingers, Consumer Reports–to say nothing of at least three credit rating agencies–it is certainly possible to come up with multiple, credible ways to evaluate any given product. I know that valiant (and not-so-valiant) efforts have been made to compete with USNWR, but so far nothing seems to have stuck.  Why hasn’t some media powerhouse come up with its own system?  It’s clearly been lucrative for U.S. News–I suspect it is the only thing keeping the brand alive.

But perhaps I answered my own question, above:  No one really wants more than one ranking  precisely because of the clear signal sent by a single, if flawed, metric.

Still, if they’re smart enough to become lawyers, you’d think they’d be smart enough to assess more than one set of measures.  Isn’t that what judging is all about?

Going beyond the reasonable notion that you should periodically take a deeper look at what you’re doing, pedagogical reformers of many sorts get convert zeal and treat assessment as a moral imperative.  But, when a religion has enough zealous adherents, it might suddenly become mainstream.  And when it goes mainstream, it goes from being pure to being mass market lowest common denominator oversaturation.  The word “assessment” is no longer just confined to careful examinations of how [...]]]> Academics – driven by their accrediting agencies - have a new buzzword.  We are all now charged with thinking about assessment.  How well are we doing at the goals we set out for ourselves?  How do we know?  How do we know if our processes of assessment are appropriate?  As an academic (non-legal) blogger observed:

Going beyond the reasonable notion that you should periodically take a deeper look at what you’re doing, pedagogical reformers of many sorts get convert zeal and treat assessment as a moral imperative.  But, when a religion has enough zealous adherents, it might suddenly become mainstream.  And when it goes mainstream, it goes from being pure to being mass market lowest common denominator oversaturation.  The word “assessment” is no longer just confined to careful examinations of how well something is working.  It isn’t even just applied to a bureaucratic ritual of report-writing focused on the curriculum.  It’s applied to every piece of paper, every report, every bit of data, any and every piece of bureaucracy and hoop-jumping and report-generating.  The odds are good that a time sheet will soon be marked “Hours assessment” and an account statement will be marked “Fiscal assessment.”

This proselytizing ideal has obviously caught on in the ABA’s self-study process, which requires not just a strategic plan and a strategic planning process, but also that the school show that it regularly evaluates its self-assessment and thinks about whether the school’s goals are good ones.  Schools which fail to have a process, plan, and plan assessment will be disapproved until they come to their senses.

It’s no small irony – nor I’m sure am I the first to note – that there is no evidence at all that schools which regularly engage in planned reflection produce better outcomes for students or for society than schools who muddle through with less formal techniques.  I’m not even sure that it is possible to design an experimental study that would make the case for assessment, given external validity concerns.  The case against self-reflection is pretty simple: deciding what academics ought to maximize is a hard problem, and any answer arrived at by any group of people will necessarily be too vague to provide hooks for truly useful tactical choices, especially when the time spent planning uses up  productive resources .  Indeed, it’s possible that designing ever-more-particularly assessment metrics (and plans for achieving those metrics) encourages us to set ever-more-narrow goals, which are then, comfortably, met.

All in all, I’d give the current assessment trend a 23.3 on an A to ∂ point scale, where our goal is to hit a ß.

Penn                92.86%

Temple            92.34%

Villanova        89.87%

Widener (H)   87.67%

Pitt                  86.93%

Duquesne       86.47%

Penn State      83.64%

Widener (D)   82.61%

Drexel             81.32%

Rutgers (C)    77.85%

Because failing the bar can be economically devastating, bar passage is a very, very important marker of  a law school’s success – certainly more so than SSRN download rankings!!   Being above the state’s average is a big deal, and worth celebrating.  Temple had a problem on this score about a decade ago, and we made serious efforts to help our students be better prepared to enter practice.  I’m glad to see [...]]]> Pennsylvania’s Bar results came out last week.  Congratulations to all passers, and especially to my graduating students who are now licensed lawyers.  The statewide pass rate for first-time takers was 84.68 percent.  The rates for the Pennsylvania area law schools were:

Penn                92.86%

Temple            92.34%

Villanova        89.87%

Widener (H)   87.67%

Pitt                  86.93%

Duquesne       86.47%

Penn State      83.64%

Widener (D)   82.61%

Drexel             81.32%

Rutgers (C)    77.85%

Because failing the bar can be economically devastating, bar passage is a very, very important marker of  a law school’s success – certainly more so than SSRN download rankings!!   Being above the state’s average is a big deal, and worth celebrating.  Temple had a problem on this score about a decade ago, and we made serious efforts to help our students be better prepared to enter practice.  I’m glad to see that our efforts are bearing fruit.

Incidentally, the combined rates (July/Feb) are consistent – though Penn falls a bit — and I’ve posted that list after the jump:

Temple            92.05%

Villanova        89.35%

Penn                87.76%

Widener (H)   86.67%

Pitt                  86.50%

Duquesne       86.44%

Drexel             81.05%

Penn State      80.65%

Widener (D)   80.22%

Rutgers (C)    77.14%

According to this measure, the  following are the top-25 student-edited general interest law reviews published in the United States.   The list looks congruent with my sense of generally accepted understandings among law faculty of law review standings.  At first it may make one wonder whether tools like this are useful because they verify knowledge or useless because they don’t tell us anything new.   But, on second thought, people new to this profession may neither know nor want to ask. 

1. Harvard  Law Review

2. Yale  Law Journal

3.  Stanford Law [...]]]> The latest way to measure scholarly influence is the eigenfactor, a term to describe various algorithms used to quantify aspects of knowledge.  The linked web site enables people to find top lists using assorted measures, including the top law reviews using article influence proxied by citation histories. 

According to this measure, the  following are the top-25 student-edited general interest law reviews published in the United States.   The list looks congruent with my sense of generally accepted understandings among law faculty of law review standings.  At first it may make one wonder whether tools like this are useful because they verify knowledge or useless because they don’t tell us anything new.   But, on second thought, people new to this profession may neither know nor want to ask. 

1. Harvard  Law Review

2. Yale  Law Journal

3.  Stanford Law Review

4. Columbia Law Review

5. NYU Law Review

6. Virginia Law Review

7.  Texas Law Review

8. Michigan Law Review

9.  California Law Review

10.  Penn Law Review

11.  Chicago Law Review

12.  UCLA Law Review

13.  Northwestern Law Review

14.  Duke Law Journal

15.  Minnesota Law Review

16. Cornell Law Review

17.  Georgetown Law Journal

18.  Vanderbilt Law Review

19.  George Washington University Law Review

20. Notre Dame Law Review

21. Iowa Law Review

22. Indiana Law Journal

23.  Illinois Law Review

24. Boston University Law Review

25.  University of Southern California Law Review

Eigenfactor’s linked list also included faculty-edited and subject-limited reviews.  Among those appearing within the top 25 overall were: Journal of Legal Studies (5); Law & Society Review (15);  Harvard International Law Journal (22); and Harvard Civil Rights-Civil Liberties Law Review (23).  The link also enables using an alternative scoring system, which mostly only slightly reorders the foregoing list, though also adds, bizarrely, periodicals like Fortune and the New Republic, as well as one additional journal, Fordham Law Review.

Update: To find these results at the linked site, select the word Law from the “Eigenfactor subject category” drop-down menu then click search.  These results are for 2008, the latest available today.

In law, lacking a tradition of co-authorship, there appears to be at best a weak norm that the first author is the primary contributor. That [...]]]> Other disciplines don’t kid around about the ordering of authors in publications.  In political science and economics, alphabetical or reverse-alphabetical ordering is the dominant approach, even though it distorts hiring decisions.  In science, the first and last names matter – woe to the middle men!  Harvard is so concerned about the trend that it instructs its faculty to “specify in their manuscript a description of the contributions of each author and how they have assigned the order in which they are listed so that readers can interpret their roles correctly [and] prepare a concise, written description of how order of authorship was decided.”

In law, lacking a tradition of co-authorship, there appears to be at best a weak norm that the first author is the primary contributor. That results in a set of interrelated problems:

1)  To law audiences, the first author did the most work, and is rewarded in two ways.  The first is qualitative, and pops up at tenure, promotion, and lateral review — “he was the driver on that piece,” or “she was just the second author.”  Quantitatively, the bluebook foolishly permits multiple author works to be et al’d, meaning that the second through nth authors never get to see their name in the citation print.  Given the rudimentary nature of impact citation analysis in the legal academy, this mean that people who are listed first get the citations and the people who aren’t don’t. This might be less troublesome if the “first author” norm was correct — that is, if first authors in law reviews actually did more work. But my bet is that given letter head bias, many co-authored pieces list as the first author the most prominent author (or at least the author at the best-ranked school).  The upshot: first authors in law reviews are rewarded for being first in both qualitative and quantitative terms, though it’s not clear they ought to be.

2)  To other disciplines, this is fundamentally screwy and is another reason not to publish in a law review.  But interdisciplinary co-authored work published outside of the law reviews becomes that much more difficult as a result.  If a law professor and a non law professor were to publish in an economics journal, my sense of the norm is to alphabetize. [Correct me if I'm wrong here.]  Non legal audiences look at this and understand that it doesn’t signify relative contribution.  Law audiences don’t have that filter on, and the result (again) is that the second author is punished, here for having a last name at the back of the alphabet.

3)  Making sense of this mess requires coordination, which is quite hard because we lack a learned society that is sufficiently respected to impose change from above.  We do have, however, a few very strong journals that have had remarkable success in changing otherwise intractable scholarly pathologies like article bloat.  If the Harvard Law Review could -almost singlehandedly – impose a 25,000 word limit, surely it could fix this problem too.  In my view, the top few journals (HYS) ought to, as a part of their blue-booking project, agree to impose something like the Harvard faculty author order guidelines on folks who are publishing joint projects in their pages. The default ought to be reverse alphabetical listing.  Each article should state the respective contributions of the authors and, to the extent that they have deviated from the alphabet, why.  Finally, HYS ought to reform the bluebook to insist that the first citation of any work include the names of all contributors to the piece, rather than permitting et al. treatment.

Here’s an example.  A few weeks back, Bob Morse issued a stern warning to law school administrators out to game his rankings.  In response to a problem created by “openness about our ranking model” Morse took a strong step in the direction [...]]]> Why is Bob Morse, USWR’s ranking guru, so unafraid of competition? There’s plenty of evidence that his rankings are fragile – for example, changes at the “bottom” of the scale have unexpected influences on schools at the top because the magazine engages in a statistically bizarre forced normalization – and that they measure factors that have little relevance to an underlying “quality” measure of importance in the market for legal jobs.  But he persists in acting like an incumbent politician, prioritizing optics over real reform.

Here’s an example.  A few weeks back, Bob Morse issued a stern warning to law school administrators out to game his rankings.  In response to a problem created by “openness about our ranking model” Morse took a strong step in the direction of reform by…wait for it…threatening certain schools with punishment for gaming their employed-at-graduation statistic.  For those who follow the rankings, this was a particularly galling and obnoxious post.  The rankings model isn’t at all “open”: for most categories of concern, USNews engages in hidden manipulations of dubious value which make replicating the results quite difficult.  See, e.g., LSAT percentile scoring, COLA adjustments; normalization, treatment of missing data, etc.  Indeed, the rankings would likely fail the very low bar for openness and replication set by even a student-edited law journal, let alone a peer reviewed publication.  And the irony is that savvy administrators would find the at-employment graduation statistic of very little interest, because its relative contribution to a school’s rank is significantly dampened by lack of variance.  To put it another way, Morse seized on “gaming” of a factor that has the second least connection with overall ranking success (behind library volumes).  Leiter said it best: “fiddling while rome burns.”

Why did Morse’s team focus on this particular statistic, as opposed to working on real reform? It’s all about the perception of legitimacy: an increasing number of schools weren’t reporting their employment numbers (because the formula for imputing missing values in this input factor produced a helpful number).  When Paul Caron pointed this out, it was embarrassing for Morse, especially when Caron’s post was picked up widely. Real reform might result in dramatic changes that called into question earlier rankings, as well as adding cost and expense.  This change, on the other hand, will have at most a marginal effects on scores, and costs essentially nothing.  Mission accomplished!

Still, the question remains unanswered: what makes Bob Morse so convinced that incumbency renders his flawed product insensitive to normal market corrections?

Or Duke’s claim is just . . . er . . . false.  But I would never accuse a Top 11 school of misrepresenting the facts.

We caught up with Bob Morse, the director of data services for U.S. News, who said in his estimation, the 1-5 options generally speaking matched up with the level of knowledge held by the raters. “We’ve felt that the level of judgment isn’t granular enough to provide a wider scale.”

He also said that because the survey reports the results of the reputation question out to the tenths place, “we’re actually publishing it on a scale of 50; the results average out to be more granular.”

Morse defends the granularity of the US [...]]]> Over at WSJ Blog, Ashby Jones contacted Robert Morse to get his reaction to my post about how raters should fill out the US News law school rankings forms:

We caught up with Bob Morse, the director of data services for U.S. News, who said in his estimation, the 1-5 options generally speaking matched up with the level of knowledge held by the raters. “We’ve felt that the level of judgment isn’t granular enough to provide a wider scale.”

He also said that because the survey reports the results of the reputation question out to the tenths place, “we’re actually publishing it on a scale of 50; the results average out to be more granular.”

Morse defends the granularity of the US News rankings by pointing to the fact that the average scores do have decimal points.  Although this is true, it doesn’t address the problem I pointed out in my post — for the individuals doing the ratings, they can’t accurately reflect their sense of how a school compares to other schools.

Who gives Yale a 4 on only a 1-5 scale?  Or Harvard?  Either this person has a very different theory of law school reputation or is trying to game the system.

Let’s say that there are 10 reviewers, and they rate as follows:

Yale scores: 5, 5, 5, 5, 5, 5, 5, 5, 5, 4 = 4.9 average

Harvard scores: 5, 5, 5, 5, 5, 5, 5, 4, 3 = 4.7 average

Morse would conclude that the difference between Yale and Harvard is meaningful.  I would conclude that the difference is attributable to either (1) a fluke due to quirky beliefs of a very small number of raters; or (2) gaming by some raters.  I just don’t see how, on a 1-5 scale, Yale or Harvard would get any less than a 5 on all the forms.  Their averages should both be 5.0.  Any differences are the result of flukes or gaming and shouldn’t be taken seriously.

These problems exist beyond Yale and Harvard — they persist for the entire US News survey because there’s not enough granularity.  If there’s a granularity problem for individual raters that makes their ratings flawed, then the problem doesn’t just disappear by aggregating flawed ratings.

With all due respect to Morse, I must also disagree with his first point.  As my post demonstrated, Morse is wrong in his statement that “the level of judgment isn’t granular enough to provide a wider scale.”  If he’s right, then readers of my post should conclude that my hypothetical dean is way outside the norm.  But I’m willing to bet that among those in the academy, the consensus on the schools I mentioned in the post is that they deserve different ranking levels: Yale, Michigan, Cornell, USC, Emory, and American.  My sense is that there’s a consensus that most raters would rate them in the order listed, and would think that they are all at different reputational levels.

If this is the consensus, then Morse’s 5 point scale isn’t granular enough for individual raters.  And aggregating scores that are assigned based on a system of rating that isn’t granular enough doesn’t fix the problem.  It just means that the outliers control the outcome, and those outliers are either people with views way outside the norm or people who are gaming the system.  I don’t think we want the ratings to turn on what the outliers are doing.

I appreciate Morse’s response to Ashby Jones, and would be interested in his response to my points above.

Please note that I’m not an expert on statistics, so I’m open-minded about my claims.  If there’s a statistics expert among our readers, I’d be very interested in your thoughts.

When studying it, Deans may find it worthwhile to rearrange in any number of ways. In most rearrangements, data are likely to appear in line with the data as presented, though some interesting variations may appear. As an example, Professor Leiter’s report ranks schools according to weighted citations of all scholars within [...]]]> Just as Law School Deans heed US News rankings, for better or worse, they may heed scholarly impact rankings, including recently-released tabulation performed by U. Chicago Professor Brian Leiter. It counts high citation scholars over the past five years, with attribution and ordering (a) by school, (b) within school, and (c) across a dozen subject areas. Hats off to Professor Leiter for taking what must have been scores of hours to assemble the data.

When studying it, Deans may find it worthwhile to rearrange in any number of ways. In most rearrangements, data are likely to appear in line with the data as presented, though some interesting variations may appear. As an example, Professor Leiter’s report ranks schools according to weighted citations of all scholars within the school. An alternative would rank schools according to the number of times a school’s scholars are named in the high-impact lists by subject matter.

In that rearrangement, there are small differences in rankings of the top seven schools, but many more marked differences further on the line.  Following are the top seven, ranked by number of times scholars appear in high-impact rankings by subject (that number appears in parentheses, followed in separate parentheses with the ranking in Professor Leiter’s report).

1. Harvard (28) (2nd)

2. NYU (26) (5th)

2. Yale (26) (1st)

4. Columbia (20) (6th)

5. Chicago (16) (3rd)

6. Berkeley (13) (7th)

6. Stanford (13) (4th)

The foregoing comparative rankings differ, but slightly, especially compared to the next cohort:

8. Michigan (11) (11th)

9. George Washington (8) (18th)

9. Penn (8) (14th)

9. Texas (8) (17th)

12. Georgetown (7) (20th)

13. San Diego (6) (outside 25)

13. Vanderbilt (6) (10th)

13. UCLA (6) (15th)

16. Cornell (5) (11th)

16. Duke (5) (11th)

16. Illinois (5) (21st)

16. Northwestern (5) (8th)

20. Minnesota (4) (18th)

20. USC (4) (outside 25)

22. Chicago-Kent (3) (outside 25)

22.  Emory (3) (23rd)

22. Virginia (3) (16th)

25.  Seven Schools Tied (2) (various)

NB: These data are unaudited.  They are compiled by conducting a word-search and count protocol on the original subject matter lists appearing in Prof. Leiter’s post.

Let’s assume a knowledgeable dean filling out the form in good faith.  How is he or she to go about filling out the form?

Here’s my hypothetical dean’s stream of consciousness:

Okay, I think Yale [...]]]> Every year, US News compiles its law school rankings by relying heavily on reputation ratings by law professors (mainly deans and associate deans) and practitioners and judges.  They are asked to assign a score (from 1 to 5) for the roughly 200 law schools on the form.  A 5 is the highest score and a 1 is the lowest.  While many factors that go into the US News ranking have been criticized, the reputation ratings by and large are considered one of the best components in the ranking system. But should it be?

Let’s assume a knowledgeable dean filling out the form in good faith.  How is he or she to go about filling out the form?

Here’s my hypothetical dean’s stream of consciousness:

Okay, I think Yale is the top law school, so I’ll give it a 5.

What about Michigan?  Great school, but not quite as high as Yale.  I’ll give it a 4.

Cornell is an excellent school too, one of the best.  But it’s not Yale or Harvard, so I can’t give it a 5.  It’s not as good as Michigan in my view, so I can’t give it a 4.  I gave Penn and Berkeley 4′s too, and I think Cornell isn’t quite at the same level.  So it’s a 3.

What about USC?  Another excellent school, but it’s not as high as Cornell.  So it’s a 2.

Ruh-roh!  I’m not even out of the top 20, and I have 160+ law schools to assign scores to, and I only have one number left.  But I must go on!

How about Emory?  That’s a bit lower than USC in my view, so I’ll give it a 1.

What about American?  Another terrific school, but I think Emory’s better.  I can’t give American a 0.    What do I do?  Okay, I guess I’ll give it a 1 as well.

But I’m not even out of the top 50.  Yikes!  I’ve run out of numbers.  Maybe I’ll call Robert Morse and ask him if I can start assigning negative numbers.   What do I do?

Time to try some math.   To make things easy, I’ll assume there are roughly 200 law schools.  And I have 5 numbers to assign — 1, 2, 3, 4, and 5.  Assuming an equal number of schools assigned to each number, that’s 40 schools for each number.  OMG!  So I need to give schools I rank 1-40 a score of 5, schools I rank 41-80 a 4,  schools I rank 81-120 a 3, schools I rank 121-160 a 2, and every other school a 1.  But that’s ridiculous.  The law school I think ranks #40 isn’t anywhere near the law school ranked #1.  This system is impossible.

Okay, maybe I give a score of 5 to the top 10 law schools, then a score of 4 to the next 90 law schools, then 3′s and 2′s to 100-150, and 1′s to the rest.   But that still lumps too many schools together.  If every person filling out the form did what I did, then there would be no way to distinguish the top 10, and no way to distinguish schools in the top 100.  Only real outlier scores would determine the difference.

Dear Mr. Morse — what am I to do?  Please help me!

Does anyone have any advice for our poor dean?  How are people to fill out the US News ranking forms in good faith to reflect accurately their sense of law school reputations?

In March, Fred contributed an amazingly insightful, thoughtful, reflective, and useful series of seven posts called Thoughts on Choosing a Law School.   The 7-part series broke down as follows: (1) limited utility of popular rankings; (2) curriculum;  (3) faculty staffing of instruction; (4) subject matter distinction; (5) faculty strength; (6) physical facilities; and (7) faculty publishing record.

These were formally directed to students considering which law school’s admissions offer to accept; and they also mean a great deal to we suppliers of legal education.   We’re grateful for these contributions.  And we’re delighted Fred will be back to contribute more wisdom, on these [...]]]> We are delighted that our esteemed guest blogger, Professor Alfred Yen (Boston College), with us this past month (and before), will stay another one.  (You can see my post introducing Fred, my former colleague, here.)

In March, Fred contributed an amazingly insightful, thoughtful, reflective, and useful series of seven posts called Thoughts on Choosing a Law School.   The 7-part series broke down as follows: (1) limited utility of popular rankings; (2) curriculum;  (3) faculty staffing of instruction; (4) subject matter distinction; (5) faculty strength; (6) physical facilities; and (7) faculty publishing record.

These were formally directed to students considering which law school’s admissions offer to accept; and they also mean a great deal to we suppliers of legal education.   We’re grateful for these contributions.  And we’re delighted Fred will be back to contribute more wisdom, on these and the many other subjects within his capacious expertise.

NOTE: These aren’t the real rankings and this isn’t a memo by Robert Morse.  This is a joke.  If you don’t realize it is a joke, then something is seriously wrong with you.

To illustrate, I’ll reveal a bit about two classes I have taught: copyright and evidence. I’ve published a reasonable amount about copyright, including a casebook published by West. By [...]]]> In this post, I’m going to argue that prospective students should care whether a law school’s faculty publishes. Not everyone agrees, and we’ve all had professors who were great scholars but indifferent classroom teachers. I also freely concede that teaching ability does not necessarily go hand-in-hand with scholarly ability, so that a school’s best teachers need not be its best publishers. Nevertheless, I still think that faculty who publish have a better chance of offering outstanding classes than those who do not.

To illustrate, I’ll reveal a bit about two classes I have taught: copyright and evidence. I’ve published a reasonable amount about copyright, including a casebook published by West. By contrast, I’ve published nothing about evidence, with my background in that area coming from my work as a litigator.

Students have rated both of these classes well. In fact, I don’t think there’s any significant variation in the numbers. Yet, I firmly believe that I teach better a copyright than evidence class because the things I learn from research and publishing enable me to give copyright a deeper and more nuanced treatment. I know more about the overall structure of the area, respond better to student questions, and challenge students in more ways in copyright than in evidence.

Now granted, I don’t think this is something that students always pick up. My evidence class is pretty “black letter,” sticking to how lawyers need to work through evidentiary problems in courtrooms. This makes sense given how students will use evidence, and I think students feel that the course serves them well. Nevertheless, I am aware that I don’t blend in the “big theory” issues as well as I could because I don’t know them that well.

By contrast, I pack a lot into my copyright course. This sometimes frustrates students. Some only want “black letter” law (something that is very elusive in copyright at best). Some dislike what they consider theoretical digressions from what they need to know for practice. I could teach copyright to that lower common denominator, but I choose not to. And I like to think that my students come to appreciate that the complexity they encounter ultimately serves them well when they deal with that subject’s frustrating ambiguity in practice. In short, although I teach what I think is a good, competent evidence course, the academic “ceiling” in my copyright class is much higher.

To be clear, I am not saying that publishing is the only thing that prospective students should care about in evaluating a law school’s faculty. As I suggested in an earlier post, some law schools clearly value teaching and their professors are accessible to students in ways that can matter a great deal. Students should visit schools, talk to existing students, and see if classes are well-received. Such inquiry will probably identify a number of schools that appear to have good teaching. At this point, I think it makes sense for a prospective student to then compare publication records of the faculties to see how often they will learn from professors who are at the forefront of their fields.

The most important space for students is the classroom. When you visit a school, look at some large and small classrooms and evaluate the sight lines and acoustics, preferably by sitting in on a live class. Do students sit in a pattern where they can see and hear each other? Can they hear the professor? You might be [...]]]> Every prospective student notices the physical facilities of a school when he or she visits. Wood paneling, marble floors, and grand foyers create impressions about whether a law school is well-funded and a “nice” place to study. I’d like to suggest a few other ways in which prospective students should evaluate a school’s facilities.

The most important space for students is the classroom. When you visit a school, look at some large and small classrooms and evaluate the sight lines and acoustics, preferably by sitting in on a live class. Do students sit in a pattern where they can see and hear each other? Can they hear the professor? You might be surprised at the number of classrooms where heating or air conditioning interferes with voices. This might not seem bad in the traditional lecture class you had in college, because professors can always wear a mike. But in law school, the Socratic method makes it important to hear what your classmates say. It’s impossible to follow along if you can’t. In addition to sight lines and acoustics, you might also look at the front of the room. Is there full audio-visual capability with a computer for the professor? Is there enough white or blackboard? Is the screen large enough for easy viewing by students?

Next, I would suggest looking at the individual and group work space available for students. Individual work space exists primarily in the library. There needs to be ample seating to support students during high demand periods like exams or major writing projects. Is there seating of the kind you prefer to work in? Long tables? Individual carrels? Big, padded chairs to sit in while reading? Is there ample Internet access, wired or wireless? You are going to spend a lot of time studying in law school. Unless you are sure that your apartment or house provides you with the space you need, you will likely spend a lot of time in these facilities.

Group work space exists in libraries and sometimes elsewhere throughout the school. How many small conference rooms are there that students can reserve? I personally wouldn’t be too happy with only a few. At certain times of the year such as moot court competitions, there is a lot of student collaboration going on, and demand for these spaces can get pretty heavy.

One other type of important student work space involves the facilities of any clinical programs. If the school has clinics where students actually represent clients, are there proper rooms where client meetings and interviews can be held, separate areas where students can do work and maintain case files? Clinics are expensive to run, and it is not uncommon for schools to trim those costs by providing clinic facilities that don’t fully support the clinics’ work. If you think a clinic will be a big part of your legal education, this could matter.

Finally, I suggest looking at the spaces where students can gather informally. Is there a good student lounge or other gathering place like a cafeteria? Are there seats in hallways where you can sit for conversations? Granted, these amenities may not seem terribly important, but their absence impairs the creation of a community where students get to know and support each other.

All of the things mentioned here seem pretty obvious, perhaps so obvious that one would think every law school would take care of them. It may well be the case that the schools you’re comparing will all have good physical facilities. But you might also be surprised at how often schools, even some of the top schools, have facilities that don’t fully support their educational ambitions.

Professors come in many types. For purposes of this post, however, we can get along with a distinction between permanent faculty and part-time (frequently called adjuncts) faculty. For [...]]]> I thought I would say a bit about faculty – the people who teach all those classes in the curriculum. Every law school will tell you that its faculty is excellent, and with justification. Law teaching jobs are sufficiently desirable that law schools generally have many, many qualified applicants for openings. Law schools today hire very well qualified people. Nevertheless, I would like to suggest one way in which prospective students can evaluate whether a particular faculty will provide a good educational experience.

Professors come in many types. For purposes of this post, however, we can get along with a distinction between permanent faculty and part-time (frequently called adjuncts) faculty. For permanent faculty, law teaching is their full-time job. Part-time faculty, as their name implies, generally have another job and devote a relatively small amount of their time to law teaching. They generally teach one class at a school, often in the early morning or evening, and they frequently do so from year to year.

A good school should have the vast majority of its courses, particularly first year courses and basic doctrinal upper year courses, taught by permanent faculty. This is not to say that part-time faculty can’t do a good job. Many are good, dedicated teachers. Nevertheless, full-time faculty are at the school, present for students in ways that would be impossible for part-time faculty. Those professors have more time to focus on teaching, and they bring cutting edge expertise based on their research to the classroom. There are, of course, areas in which part-time faculty can do a better job than permanent faculty. For example, skills courses or courses focused on specialized topics related to practice (e.g. business planning) benefit from the day to day practical experience of adjuncts.

Accreditors give significant importance to the principle that law students should be taught primarily by full-time faculty, and accreditors will give law schools trouble if the principle is violated. Surprisingly, however, law schools sometimes overuse part-time faculty. This happens because, at some schools, permanent faculty do not want to teach first year or other basic courses. Student enrollments in those classes are high, so teaching those classes takes more time than teaching smaller seminars that may be more closely related to a faculty member’s research. It’s obviously hard for schools to force tenured professors to teach classes they don’t want to teach. Indeed, faculty who don’t want to teach a class may not do a good job.

For prospective students, a law school that does not put its full-time faculty in basic classes raises a question that needs to be answered. Do the school and its faculty really give sufficient priority to teaching students? Every school will of course answer yes, but sometimes actions speak louder than words.

A school that offers, for example, 24 intellectual property courses surely offers far more [...]]]> Law schools compete for students by touting the strength of their curriculum, and with every school claiming that it is strong in a particular area, it’s sometimes hard to get a handle on whether a particular school really would be better than another for a student interested in, for example, corporate law or environmental law. One possible way to assess this might be the raw number of courses in a particular area, and in a sense more can be preferred to less. That having been said, I’d encourage prospective students to look beyond raw numbers when evaluating claims of curricular excellence.

A school that offers, for example, 24 intellectual property courses surely offers far more courses than an individual student could ever take. That doesn’t mean that the large number of offerings is valueless. Rather, a student should think carefully about how many courses one can profitably devote to concentration in a particular area versus the general education that forms the foundation for the successful practice of law. For example, a student may want to specialize in intellectual property, but she should also make room in her curriculum for corporations, commercial law, antitrust, employment law, and other areas that arise when considering IP issues. Additionally, I think it’s important for students to take a class or two devoted to perspectives on law like jurisprudence, law and economics, or legal history. They greatly enrich a legal education. When one adds these classes to requirements such as professional responsibility and courses people take because the subjects appear on the bar, there aren’t that many open slots for specialization. At some point, adding classes is nice, but perhaps overkill.

A student should also evaluate whether the courses offered by a school permit effective progression from basic study to advanced possibilities. Each field has basic courses that serve as entry points of study. In the corporate law area, that would probably be a course like corporations or business associations. More advanced doctrinally oriented courses might include corporate finance, securities regulation, and mergers and acquisitions. Beyond that, students might branch out in a couple of different directions. One would be toward increasingly advanced theoretical or policy study, perhaps in a seminar with a large research project. For example, a school might offer a seminar on theories of corporate governance or applications of the efficient capital markets hypothesis. The other would be towards practical application of knowledge and skills training. These classes would include classroom skills courses like drafting or trial practice, live client clinics where students actually practice under the supervision of faculty, and externship placements in law firms, companies, or government offices.

Obviously, the course content of a particular curriculum is not the only thing that determines its quality. A lot depends on who does the teaching, a subject I will address in another post. But for now, students can probably identify schools that will serve their needs by considering not only the number of offerings in an area of interest, but also the structuring of the curriculum to provide opportunities for intellectual depth and development of skill.