Law School Ranking: Measurement vs. Characterization
posted by Frank Pasquale
I have long been concerned about negative externalities from ranking systems. Perhaps people and institutions are always prone to try to distinguish themselves. If so, Brian Leiter’s expert consultation for the MacLean’s rankings of Canadian law schools may be a good thing, since, as he states, it resulted in “a ranking system that can not be gamed, that does not depend on self-reported data, and is not an indecipherable stew of a dozen different ingredients.”
However, Benjamin Alarie at Toronto has critiqued Leiter’s efforts. Here are a few issues:
One of the central problems with how Faculty quality is measured is that it doesn’t assess influence in publications aside from 33 Canadian law journals. As an initial matter, I think it is fair to say that academics seek to publish in places with the most active audiences for particular types of research—for example, the best journals to publish law and economics research in are likely to be American peer reviewed journals such as the Journal of Legal Studies, or the American Law and Economics Review, or even professional economics journals.
[I]t is unlikely that frequency of citation is a perfect proxy for quality; for example, overly provocative papers are sometimes cited for being so provocative.
It is unclear what the threshold used was for including the firms as among the “elite firms” used by Maclean’s.
[A national reach] measure . . . misses “international reach” of the law schools that regularly place students in the excluded top New York and Boston firms, in international NGOs, and in various other attractive positions.
[By the way, I put the links into those block quotes above.]
I think these are all valid points, but the problem is even larger. The consumers of these rankings are, by and large, students looking for a good education and firms looking for well-trained lawyers. Why so much focus on whether the law schools are feeders for “top” firms? Perhaps the best law teaching is that which manages to train people for diverse careers in law.
Finally, a more philosophical point.
I fear that a misguided quest for objectivity can lead us to present many judgments that ought to be characterizations as, instead, measurements or rankings. Leiter deserves great credit for not trying to mash an incommensurable array of statistics into a single figure. But I still think true accuracy in law school assessment might be better found in words, not numbers–in a long-form assessment of the school’s service to students, to the academy, and to its community. Perhaps such an evaluation would take a few hundred pages to cover all of Canada’s law schools. But if someone about to embark on a career in law is not willing to try to make sense of such a document, perhaps they should not be a lawyer in the first place.
Obviously such long-form documents would not achieve the concision of a ranking system. As someone who’s decried information overload, I see rankings’ appeal. But I also recall a favorite saying of Hilary Putnam on approaches to epistemology: “any theory that fits in a nutshell belongs there.” Perhaps the same can be said of rankings. There are some virtues to “knowing” less.
September 13, 2007 at 9:04 pm
Posted in: Law School (Rankings)
Print This Post
Responses (3)
Michael Lee - September 13, 2007 at 11:25 pm
Just as the law cannot command a useless act, so should we not undertake or acknowledge a useless act, i.e., to pay attention to wholly subjective rankings.
Matt - September 14, 2007 at 9:43 am
The sorts of criticisms noted in the quoted section can be taken in two ways- one reasonable and the other stupid. The reasonable way is as either noting things that might be improved on in the future or else as noting things that should lead smart consumers of the rankings to ask more questions or otherwise serve as caveats on using the rankings. I’m sure that Leiter has no objections to such remarks. The stupid way to mean them would be to believe that there could be a perfect ranking system, one that combines all desirable elements and has no undesirable ones. It’s hard to tell here which way the remarks are meant so I’ll assume it’s the good way. Many critics of rankings, however, seem pretty clearly to mean the stupid thing. (Leiter, for example, has given some good reasons for focusing on Canadian journals. They might not be fully persuasive in the end, but it wasn’t an arbitrary choice, and including American journals would have some clear draw-backs, too.)
Finally, the idea that rankings of schools in general is bad seems silly to me. Unless one thinks there is no significant difference between the schools (an unlikely proposition) then rankings can be useful to students. They are not perfect but of course anyone who uses them in a stupid way probably ought not go to law school, even more so than someone who would not read a 300page narrative account.
frank cross - November 29, 2007 at 2:44 pm
I find lawyers’ reaction to numbers and rankings very odd. These evaluations are made all the time, which is the better law school, who’s the best scholar, etc. The rankings simply provide information on this question. They aren’t perfect but your subjective opinion on the issue sure as heck isn’t perfect either.
There’s a strange fear of numbers here. There are various reasons for the fear of numbers, I think. One common, and distressing, reason is that they are less manipulable to reach the result the person wants, than is subjectivity. But the fear I see mostly is the thought that the numbers are somehow the “answer.” That they magically end the discussion. That’s wrong. They simply inform the discussion. May they be misused? Sure, but that is true of everything and hence is a criticism of everything (and hence, of nothing).
Leave a Reply