How to Fill Out the US News Law School Rankings Form
posted by Daniel Solove
Every year, US News compiles its law school rankings by relying heavily on reputation ratings by law professors (mainly deans and associate deans) and practitioners and judges. They are asked to assign a score (from 1 to 5) for the roughly 200 law schools on the form. A 5 is the highest score and a 1 is the lowest. While many factors that go into the US News ranking have been criticized, the reputation ratings by and large are considered one of the best components in the ranking system. But should it be?
Let’s assume a knowledgeable dean filling out the form in good faith. How is he or she to go about filling out the form?
Here’s my hypothetical dean’s stream of consciousness:
Okay, I think Yale is the top law school, so I’ll give it a 5.
What about Michigan? Great school, but not quite as high as Yale. I’ll give it a 4.
Cornell is an excellent school too, one of the best. But it’s not Yale or Harvard, so I can’t give it a 5. It’s not as good as Michigan in my view, so I can’t give it a 4. I gave Penn and Berkeley 4′s too, and I think Cornell isn’t quite at the same level. So it’s a 3.
What about USC? Another excellent school, but it’s not as high as Cornell. So it’s a 2.
Ruh-roh! I’m not even out of the top 20, and I have 160+ law schools to assign scores to, and I only have one number left. But I must go on!
How about Emory? That’s a bit lower than USC in my view, so I’ll give it a 1.
What about American? Another terrific school, but I think Emory’s better. I can’t give American a 0. What do I do? Okay, I guess I’ll give it a 1 as well.
But I’m not even out of the top 50. Yikes! I’ve run out of numbers. Maybe I’ll call Robert Morse and ask him if I can start assigning negative numbers. What do I do?
Time to try some math. To make things easy, I’ll assume there are roughly 200 law schools. And I have 5 numbers to assign — 1, 2, 3, 4, and 5. Assuming an equal number of schools assigned to each number, that’s 40 schools for each number. OMG! So I need to give schools I rank 1-40 a score of 5, schools I rank 41-80 a 4, schools I rank 81-120 a 3, schools I rank 121-160 a 2, and every other school a 1. But that’s ridiculous. The law school I think ranks #40 isn’t anywhere near the law school ranked #1. This system is impossible.
Okay, maybe I give a score of 5 to the top 10 law schools, then a score of 4 to the next 90 law schools, then 3′s and 2′s to 100-150, and 1′s to the rest. But that still lumps too many schools together. If every person filling out the form did what I did, then there would be no way to distinguish the top 10, and no way to distinguish schools in the top 100. Only real outlier scores would determine the difference.
Dear Mr. Morse — what am I to do? Please help me!
Does anyone have any advice for our poor dean? How are people to fill out the US News ranking forms in good faith to reflect accurately their sense of law school reputations?
April 5, 2010 at 11:05 am
Posted in: Law School, Law School (Rankings), Law School (Scholarship)
Print This Post
Responses (12)
Vladimir - April 5, 2010 at 11:32 am
This is just brilliant, Dan!
Tim Zinnecker - April 5, 2010 at 12:06 pm
How about tossing the form in the trash can?
anon - April 5, 2010 at 5:45 pm
Ever heard of decimal numbers? Problem solved.
Adam - April 5, 2010 at 6:06 pm
But the question isn’t whether each individual dean’s rankings can appropriately distinguish between law schools. The only real question is whether the average rankings across all deans do. And those are two different things. For example, say 98% of deans give a 5 to Yale and 2% give a 4, while 94% of deans give a 5 to Stanford, while 6% give a 4, and 75% give a 5 to NYU, while 24% give a 4, and 1% give a 3. Yale ends up with a reputational ranking of 5, while Stanford gets a 4.9, and NYU gets a 4.7.
The point is, because we’re aggregating scores, what ends up mattering is not just what scores deans can give, but also how those scores are distributed over a large number of deans. That will make the results more accurately reflect reality, even if most deans are giving schools 5′s against their will.
All that being said, your general point that giving deans a scale of only 1 to 5, rather than, say, 1 to 100 will make for less precise results is true, and I don’t really see what harm there would be in allowing for finer gradations.
Problem solved - April 5, 2010 at 6:42 pm
Here’s what you do, Dean Solove: George Washington gets a 5. Every other school ranked anywhere near you gets a 1. Schools in the bottom 100 where your buddies are deans can have 3s or 4s.
Then your ranking goes up (or barely keeps pace, as your peers race to out-manipulate you) and U.S. News gets to publish its “rankings.”
Daniel Solove - April 5, 2010 at 6:44 pm
Adam,
But how many deans, acting in good faith, on a 1-5 scale, would give Yale or Harvard a 4? Or a 3? The deans doing this likely have either a vastly divergent sense of law school reputation than the norm or they’re gaming the system.
notgood - April 5, 2010 at 7:42 pm
4,
Considering the publishing of reputations/rankings tends to reinforce them, we might steer away from finer gradations if we thought it would increase the, already present, distortion of an individual’s personal valuation of the schools and that such an increase was undesirable. However, from a consumer’s perspective, I do appreciate the additional detail.
Orin Kerr - April 5, 2010 at 11:43 pm
Excellent questions, Dan.
Question - April 6, 2010 at 11:33 am
Regardless of the score, the “reputation” factor is self-serving. What region are the practitioners and judges from? I’m guessing the bulk of them are from NYC, Boston, LA, DC, and other “elite” cities. Because of that bias, I doubt a school from the South or smaller cities will ever have reputation scores as high as schools in those “elite” cities.
I know respondents are “supposed to” skip rating schools they are unfamiliar with, but is that really the case? I’m sure a lot of people mark schools they are not familiar with as “1″ – if they are not familiar with them, they are not “reputable.” Since we don’t know the people surveyed or their geographic distribution, I don’t see how they can use “reputation” as such a strong factor in the rankings.
Justice Scalia - April 12, 2010 at 1:32 am
I’ve got an idea.
Why don’t the powers that be at US News rank the law school deans first (1-5 scale) and then weight their responses based on the higher ranking deans having more influence.
That’d be great. More worthless statistics.
Anon - April 14, 2010 at 11:04 am
Those who suggest decimal points or small numbers of people down-voting a school fix the problem are confusing precision with accuracy. It’s not a more accurate number just because it has more decimal points.
For example, suppose that I’m thinking of a number between one and five. All of you should try to guess what it is. The fact that we average your guesses doesn’t make it any more “accurate.” In fact, the decimal points on the average would ensure that the average was necessarily wrong. Only if you all unanimously guessed what the actual number is would the average be accurate. I’m not suggesting that “reputation” is the same as an imagined number, but the creation of more decimal places as a proxy for accuracy is ludicrous.
Further, the idea that we can accurately declare a school that gets 98% 5s to be better than a school that gets 96% 5s (from a different base of voters, no less, if the deans are truly only voting on schools about which they have some knowledge) is absurd. What’s worse, the “reputation” is largely influenced by last year’s ranking in US News, isn’t it? Yale is not a better school than Harvard. Nor is Harvard a better school than Michigan. Or Cornell or Stanford or any of the others. The rankings are the journalists revenge on law schools and their friends who went to law school instead of journalism school.
P.S. The number I was thinking of was 3.
Anon - April 14, 2010 at 11:08 am
I wish there were no typo in the word journalists above, but that is the uneditable nature of an anonymous post.
Leave a Reply