Bob Morse’s Stunning Complacency
posted by Dave Hoffman
Why is Bob Morse, USWR’s ranking guru, so unafraid of competition? There’s plenty of evidence that his rankings are fragile – for example, changes at the “bottom” of the scale have unexpected influences on schools at the top because the magazine engages in a statistically bizarre forced normalization – and that they measure factors that have little relevance to an underlying “quality” measure of importance in the market for legal jobs. But he persists in acting like an incumbent politician, prioritizing optics over real reform.
Here’s an example. A few weeks back, Bob Morse issued a stern warning to law school administrators out to game his rankings. In response to a problem created by “openness about our ranking model” Morse took a strong step in the direction of reform by…wait for it…threatening certain schools with punishment for gaming their employed-at-graduation statistic. For those who follow the rankings, this was a particularly galling and obnoxious post. The rankings model isn’t at all “open”: for most categories of concern, USNews engages in hidden manipulations of dubious value which make replicating the results quite difficult. See, e.g., LSAT percentile scoring, COLA adjustments; normalization, treatment of missing data, etc. Indeed, the rankings would likely fail the very low bar for openness and replication set by even a student-edited law journal, let alone a peer reviewed publication. And the irony is that savvy administrators would find the at-employment graduation statistic of very little interest, because its relative contribution to a school’s rank is significantly dampened by lack of variance. To put it another way, Morse seized on “gaming” of a factor that has the second least connection with overall ranking success (behind library volumes). Leiter said it best: “fiddling while rome burns.”
Why did Morse’s team focus on this particular statistic, as opposed to working on real reform? It’s all about the perception of legitimacy: an increasing number of schools weren’t reporting their employment numbers (because the formula for imputing missing values in this input factor produced a helpful number). When Paul Caron pointed this out, it was embarrassing for Morse, especially when Caron’s post was picked up widely. Real reform might result in dramatic changes that called into question earlier rankings, as well as adding cost and expense. This change, on the other hand, will have at most a marginal effects on scores, and costs essentially nothing. Mission accomplished!
Still, the question remains unanswered: what makes Bob Morse so convinced that incumbency renders his flawed product insensitive to normal market corrections?
June 13, 2010 at 2:22 pm
Posted in: Law School (Rankings)
Print This Post
Responses (4)
law teacher - June 13, 2010 at 6:05 pm
i assume that he won’t feel threatened by market corrections unless that includes law schools publishing full and candid statistical information to applicants and students. Which is not likely. So his magazine will continue to be the best source of information.
Dave Hoffman - June 13, 2010 at 6:20 pm
Law Teacher — it’s a overgeneralization. Name one piece of information that USNews provides that isn’t replicated by the LSAC data (or bettered by employment data like Henderson et al. provide, or scholarly output data provided by Leiter)? It isn’t, for example, expenditures? Hard to imagine how — USNews doesn’t actually provide the raw (or should I say, manipulated raw) information! Is it reputation among judges/lawyers, created by a survey with a notoriously low response rate?
Seriously, I’m all for data transparency — and a mark against USNews is that it isn’t transparent but pretends to be. But saying that the magazine is the “best source of information” about law school inputs and outputs is simply false. It may be the most convenient source – by virtue of salience – but that’s no good argument at all.
Joe Miller - June 13, 2010 at 6:53 pm
Dave,
You make a number of great points here.
What the status quo makes me wonder, in antitrust-ish mode, is this – What are the barriers to entry for a competing magazine? For some other publication? For a group of, e.g., law professors spread across a range of institutions? And what are the prospective returns to each? (I guess this can also be thought of as one of the barriers, in reverse). Relatedly, how good must a rankings system be, at or soon after it enters, for it to get traction? US News can be disciplined by prospective competitors only if there’s likely to be one or more of them under a realistic set of assumptions, no?. It appears Morse / US News doesn’t think there will be.
Wish there were one!
pre-tenure prof - June 14, 2010 at 6:34 pm
An interesting twist from my perspective (a pre-tenure, tenure-track prof) is the weight given to the rankings in terms of law review placements. Many tenure committees look closely at the USNWR rank of the school in determining (i) quality of scholarship and (ii) whether a piece will even count towards tenure. Further, individuals are often judged based on the number of top-20 or top-50 placements, without a meaningful examination of the actual work itself. If the academy is serious about its stated goals and beliefs regarding the relative weight of the USNWR rankings, we should take a look in the mirror to closely examine how we are making a bad situation worse.
Leave a Reply