Improving the US News Rankings: A Wish List
posted by Daniel Solove
A new article in the ABA Journal profiles Bob Morse, the US News & World Report “rankings czar.” I recently corresponded with Bob when he wrote to me about my parody of the rankings. He took my humor in good spirit. According to the ABA Journal article:
Since it began the rankings in 1987, the magazine is often attacked as wielding too much power; its methodology is denounced as easily manipulated and too subjective to carry such inordinate weight.
No one understands this more than Robert Morse, the man who created the law school rankings for U.S. News. As the magazine’s data research director, Morse says he, too, feels a high level of anxiety each year when the law school rankings are revealed. . . .
He also feels the heat from those who resent their enduring influence. For a ratings czar, he is a very reluctant despot. Far from being impervious to complaint, he maintains a blog where he explains his rankings and encourages constructive criticism. He’s been known to show up unannounced at gatherings likely to denounce him.
The article goes on to note that Bob Morse is open to suggestions for improving the rankings:
Morse says he understands and agrees that the rankings are not perfect, and he would like nothing more than to discuss with law school deans ways to improve them.
“Deans are welcome to call me or come by my office in Washington,” Morse says. “I want to work with them to improve the rankings.”
For better or worse, the US News rankings are here to stay. They are tremendously influential, and despite our constant complaints, I doubt that the influence of the rankings will diminish. So we can continue to gripe and grumble, with probably little effect. Or, we might be to work with the magazine to improve the rankings. Bob says he’s amenable to suggestions for improvement:
Bob Morse has his own blog that invites comments and criticisms. He’s shown up uninvited to university symposiums dedicated to fighting the U.S. News rankings he created. He wants to hear what the critics have to say.
So since Bob is listening, I pose the question: How ought the rankings to be improved?
The current US News methodology is here.
Here are a few things I’d recommend:
1. The reputation surveys are not sent out broadly enough. They go out only to deans and to newly-tenured professors. A broader cross-section of law school faculties should be used in the poll.
2. A more granular reputation scoring system should be used. The current 1-5 score isn’t granular enough. For starters, how do top schools like Yale and Harvard have average scores less than 5. Who’s giving them a score of 4? Seems fishy to me. Suppose a dean thinks Yale is the best and that Chicago is excellent — not quite as good as Yale, but very close. Yale therefore gets a 5. Does that mean Chicago gets a 4? That’s a big drop. Giving Chicago a 5 says it is equal, which may not be the dean’s view. There’s a problem here — the scale isn’t granular enough.
3. The number of library volumes shouldn’t be a part of the scoring system. This strikes me as a silly factor in ranking law schools.
These are just a few ideas. What are yours? The purpose of this thread is not to gripe about the rankings, but to propose fixes and improvements, so please focus your comments on suggestions for reforming the US News rankings.
March 25, 2008 at 8:11 pm
Posted in: Law School (Rankings)
Print This Post
Responses (23)
CB - March 25, 2008 at 9:55 pm
Create a ranking for law schools that wish to participate in the national arena. Then have separate regional rankings for schools that aren’t striving to be as competitive in the national arena, but have a lot to offer prospective students that are planning on in a particular state or region.
CB - March 25, 2008 at 10:01 pm
oops… that should read “planning on working or practicing in a particular state or region.”
Frank - March 25, 2008 at 10:16 pm
There was an outstanding conference at Indiana on the rankings:
http://www.law.indiana.edu/front/special/2005_rankings_nextgen/
My sense is that Jeff Stake’s article did a great job explaining how the rankings can be manipulated by extraneous accounting and other factors…so perhaps the best way to improve them is to do the opposite of the practices that Stake criticizes.
Joe Miller - March 25, 2008 at 11:16 pm
Use an aggregation formula that doesn’t force the lowest score to be zero. It’s been shown ( I can’t remember the author’s name, but the paper came out last year ) that this forced-to-zero-range creates all sorts of statistical artifacts / oversensitivity to data noise.
Joe Miller - March 25, 2008 at 11:16 pm
Use an aggregation formula that doesn’t force the lowest score to be zero. It’s been shown ( I can’t remember the author’s name, but the paper came out last year ) that this forced-to-zero-range creates all sorts of statistical artifacts / oversensitivity to data noise.
Seth R. - March 26, 2008 at 12:43 am
Bar passage rates and job placement rates.
End of story.
I’d also appreciate it if they rewarded schools that refrain from flooding the market with law grads who drive down the price of legal services, but that seems a tad overreaching.
Anon - March 26, 2008 at 3:44 am
Seth R: Bar passage rates vary wildly jdx to jdx, and job placement rates are very normative. This may work better outside the t14 where 100% employment is not taken for granted, but what job placement stats would you use to distinguish between Chi and Yale? All are employed at graduation. Chi grads may have a higher mean salary, but is that because of higher talent level or because of self-selection at yale into public interest and faculty positions. For disclosure’s sake, I am a student at Stanford Law, who chose it more for the weather and the faculty members within my very narrow subspecialty than for any particular career or prestige concerns…
Seth R. - March 26, 2008 at 9:48 am
My point was that, from the student’s perspective, after those two concerns – passing the bar and employment, everything else is really of secondary or minor importance.
merevaudevillian - March 26, 2008 at 9:49 am
Too many schools attempt to game the rankings by accepting a large body of transfer students as 2Ls (e.g., Georgetown); by having sizeable part-time student bodies that may not “count” in the same way as full-time student bodies (e.g., Fordham); by attracting a stronger student base than normal because of a geographic location to a higher-ranked school, but inevitably losing many of the best students to that transfer school (e.g., BU and BC to Harvard); or calculating or spending exorbitant amounts of money on mundane services (e.g., Illinois’s Lexis “tab”).
One statistic I might use, on account of these problems, is the LSAT and GPA medians of the 2L class as a separate calculation. While it might “punish” schools who accept students with low LSAT/GPA numbers but who perform admirably their first year and deserve a transfer, the effect would be negligible if the transfers were, in fact, a negligible proportion of the 2L class. Additionally, it would provide a more accurate portrait of who sticks around for a second year.
John - March 26, 2008 at 10:24 am
Ranking based on the number of books in the law libraries is a way to encourgage the sale of more law books which in turn makes law profs happy and more supportive of the rankings.
Brian Dalton - March 26, 2008 at 10:33 am
My company, Vault, released our law school rankings a few days ago. We focused on a single criterion: the “employability” of the schools’ grads as seen from the firms’ perspective. Results here: http://www.vault.com/lawschool/top25/
Our take is obviously subjective and narrow in its emphasis on private firms. There’s no perfect survey or rankings scheme, so the more the merrier. Some reaction to our findings: http://www.abovethelaw.com/2008/03/because_the_world_needs_more_l.php#more and http://randazza.wordpress.com/2008/03/25/rankings-are-garbage-but/
Jim - March 26, 2008 at 10:41 am
Actually, the way you inflate volume counts in the library is not by buying more books, but by keeping huge runs of print reporters (in three or four copies), retaining superseded volumes and never weeding the collection. The thing is nobody but a few rich schools can afford the space to do that any more. Fortunately, volume count is around 0.75% of the total US News rankings.
Maryland Conservatarian - March 26, 2008 at 10:55 am
I think you should rank law schools by their proximity to quality major league ball parks…alright!, Maryland is #1
Mark Johnson - March 26, 2008 at 11:35 am
I think the average level of debt at graduation should be matched up with the average pay level of graduates.
John - March 26, 2008 at 11:48 am
As a graduate of a law school that last I looked was ranked 5th among the three law schools within its state, I have an axe to grind with US News and their rankings. I think the rankings have value but their value has to be put into context by considering a couple of things.
1. It is pointless and stupid to attempt to rank law schools precisely 1 through 100 or whatever. Debating whether Harvard or Yale is really better than Chicago or Stanford is the equivalent of publishing a drunken debate at the Algonquin Club. The schools should be rated by tiers without ranking each school specifically within the tiers.
2. It is pointless and stupid to compare schools that serve different functions. Some schools like the aforementioned Harvard and Yale are national schools who aim to fill a niche of producing new wage slaves for the top firms, under qualified political appointees, judicial clerks, and new law professors. Other schools are there to provide actual practicing lawyers to their local communities. The two types of schools should not be compared or evaluated by the same criteria. For example, among the regional schools, bar passage rate is an extremely important factor. Since everyone at the national schools gets a job and many don’t bother to take the bar because they go into alternative careers, bar passage rates are less important. Since salaries vary from region to region, starting salaries are not a very good measure of regional law schools. Since the graduates of the national schools compete for largely the same jobs at a few elite law firms, they are a very good measure for the national schools. The same goes with national reputation of the professors. It doesn’t take a genius or a national reputation to teach basic civil procedure but it does take a good teacher. The two don’t always match up in the same person. No one goes to place like Harvard because the of the teaching quality. They go there to say that they went there and be able to say they had Larry Tribe for Civil Procedure. It really doesn’t matter whether or not Larry Tribe can actually teach. In the same way, to a studen at a regional law school, it really doesn’t matter if his civil pro teacher has published many law review articles, it matters whether the professor can teach because the student at a regional law school is going to be expected to pass the bar and be able to know something and won’t just have a job handed to him. Therefore, national reputation of the faculty should be an important factor in judging national law schools but not that important when judging regional ones. For regional law schools, overall student satisfaction with their education should be more important than reputation.
The rankings really ought to be put into several tiers. There should be a two tier rating system for “national law schools” in which the top ten recieve top tier status and the other twenty or thirty are divided into second and third tier status. Then the other law schools ought to be divided into regions, judged differently, and placed into three tiers, top, middle and lower among each region.
This system would provide real valuable information to prospective students. Not every student wants to go to a national law school or work for a big firm. Some students want to live in a particular area and practice law. This system would tell them for example, “if you want to get a job at a good firm in Portland or city X, you don’t have to go to Harvard, you can go to this school which is the most respected regional school in the area with the highest bar passage rates.” That would make a lot more sense than ranking schools that serve different functions 1 through 100.
John - March 26, 2008 at 11:49 am
As a graduate of a law school that last I looked was ranked 5th among the three law schools within its state, I have an axe to grind with US News and their rankings. I think the rankings have value but their value has to be put into context by considering a couple of things.
1. It is pointless and stupid to attempt to rank law schools precisely 1 through 100 or whatever. Debating whether Harvard or Yale is really better than Chicago or Stanford is the equivalent of publishing a drunken debate at the Algonquin Club. The schools should be rated by tiers without ranking each school specifically within the tiers.
2. It is pointless and stupid to compare schools that serve different functions. Some schools like the aforementioned Harvard and Yale are national schools who aim to fill a niche of producing new wage slaves for the top firms, under qualified political appointees, judicial clerks, and new law professors. Other schools are there to provide actual practicing lawyers to their local communities. The two types of schools should not be compared or evaluated by the same criteria. For example, among the regional schools, bar passage rate is an extremely important factor. Since everyone at the national schools gets a job and many don’t bother to take the bar because they go into alternative careers, bar passage rates are less important. Since salaries vary from region to region, starting salaries are not a very good measure of regional law schools. Since the graduates of the national schools compete for largely the same jobs at a few elite law firms, they are a very good measure for the national schools. The same goes with national reputation of the professors. It doesn’t take a genius or a national reputation to teach basic civil procedure but it does take a good teacher. The two don’t always match up in the same person. No one goes to place like Harvard because the of the teaching quality. They go there to say that they went there and be able to say they had Larry Tribe for Civil Procedure. It really doesn’t matter whether or not Larry Tribe can actually teach. In the same way, to a studen at a regional law school, it really doesn’t matter if his civil pro teacher has published many law review articles, it matters whether the professor can teach because the student at a regional law school is going to be expected to pass the bar and be able to know something and won’t just have a job handed to him. Therefore, national reputation of the faculty should be an important factor in judging national law schools but not that important when judging regional ones. For regional law schools, overall student satisfaction with their education should be more important than reputation.
The rankings really ought to be put into several tiers. There should be a two tier rating system for “national law schools” in which the top ten recieve top tier status and the other twenty or thirty are divided into second and third tier status. Then the other law schools ought to be divided into regions, judged differently, and placed into three tiers, top, middle and lower among each region.
This system would provide real valuable information to prospective students. Not every student wants to go to a national law school or work for a big firm. Some students want to live in a particular area and practice law. This system would tell them for example, “if you want to get a job at a good firm in Portland or city X, you don’t have to go to Harvard, you can go to this school which is the most respected regional school in the area with the highest bar passage rates.” That would make a lot more sense than ranking schools that serve different functions 1 through 100.
TC - March 26, 2008 at 1:07 pm
The result attempted here is futile. Law School rankings are ultimately subjective. Dramatic appeal ultimately outweighs any consideration of statistical accuracy.
As someone describes such processes, this US News Holy Writ appears to be a splendid example of sham science and spurious specificity. Ranking consists of r= a+b+c…+z. The problem with some of the components, say “p” for peer review, is that it is arbitrary, subjective, guess work. Survey-responding profs & administrators from Harvard, Yale, Columbia always have a warm, fuzzy feeling toward their Ivy League brethren & non Ivy league responders have the same warm, fuzzy feeling toward Ivy League schools. Also a school like Stanford wound up with two recent Supremes (Rehnquist & O’Connor) & were raised from the hoi barbaroi to the Sweet 16, & even sometimes to the Top Ten.
Interesting that one of the Blog commenters notes that he/she is
“a graduate of a law school that last I looked was ranked 5th among the three law schools within its state”.
5th outta three? Presumably a typo & I don’t want to start a war over a typo.
Also, one of the Blog commenters takes a shot at Fordham Law School thusly
“Too many schools attempt to game the rankings …by having sizeable part-time student bodies that may not “count” in the same way as full-time student bodies (e.g., Fordham)….”
Not sure know how this works. Does it mean having a Four-year Night school?
TC - March 26, 2008 at 1:11 pm
Should’ve been that Stanford “was raised”.
John - March 26, 2008 at 1:21 pm
“a graduate of a law school that last I looked was ranked 5th among the three law schools within its state”.
That wasn’t a typo is was a joke. The football coach John McKay while he was coaching the woeful expansion Tampa Bay Bucaneers that the experts had picked them to finish “sixth in a five team division”. I was merely commenting on the low esteem US News holds my law school.
Kyle - March 26, 2008 at 2:13 pm
Solution: don’t rank ‘em. Take a cue from Hillary Clinton, who can find a variety of post-hoc arguments why she is winning the democratic primary, even when she is losing by the agreed upon measures. Let every school say they are the best and tell everyone why they believe that is true. The marketplace can then sort out whose arguments are the best, rather than some purported objective third party
By contrast — look at baseball. Everyone wants to play for the winning teams because they want to win (and get paid more). Then, the next year, the better teams have even better players.
Michael - March 26, 2008 at 3:44 pm
merevaudevillian said:
“by attracting a stronger student base than normal because of a geographic location to a higher-ranked school, but inevitably losing many of the best students to that transfer school (e.g., BU and BC to Harvard)”
I’m sure this affects some schools, but your example is faulty. HLS’s transfer class is tiny, 2-5 students a year, and exists merely to get the class to the historical number of 556 should it be short due to dropouts or whatever (or at least this was true as recently as 2002 when I graduated). You basically had to be top of your class AND have a good reason for wanting to come there (to study with a specific prof or pursue a specific specialty, or be closer to family or spouse). I knew most of the transfers, those that I remember came from Michigan, Georgetown, Tulane, Boalt, Penn, and Yale (yep, and one of ours transfered to Yale). I don’t remember any from BC or BU.
Sertorius - March 26, 2008 at 6:34 pm
The reputation survey should be focused on hiring attorneys at firms of different sizes and in different areas of the country.
Outside of the tiny, tiny minority of people who want to go into academia, who cares what Dean X or Professor Y thinks about your law school?
What prospective students should care about is what Hiring Partner Z thinks.
MichaelM - April 2, 2008 at 12:38 pm
The Vault has rankings based on surveys from partners who make hiring decisions. The survey ascertains “which schools prepare their students the best for law school.” This is much more useful when doing your tangible or intrinsic net present value analysis of law school, because it gives you an idea of the implied economic value of your degree.
The real problem here is that U.S. News purports to be–and is the closest thing to–a comprehensive ranking system. In truth, the issue of law school quality is so multifaceted that its difficult if not impossible to do this fairly.
A more atomized set of surveys looking both quantitatively and qualitatively at different law schools would be more meaningful. Of course, there are many such ranking systems out there, but because they don’t purport to be the all-encompassing gold standard in law school rankings (which I might be unfairly conjecturing that US News does deliberately), they don’t share the ubiquity of the US News rankings.
Leave a Reply