What Should a Judge’s Reversal Rate Be?
posted by Dave Hoffman
Via TPM, I found that critics of Judge Sotomayor have made some hay of her reversal rate in the Supreme Court (50% of 6 cases heard). Though I like data more than the next guy, I will freely admit that this use of quantitative empirical legal studies is demonstrably silly. I’m going to try to demonstrate to you in this post that a reversal rate of fifty percent is exactly what the state-of-science would predict to be ordinary [or even below average], and a rate of zero (which some might think of as emblematic of a law-abiding judge) tells you that the judge in question might be departing quite severely from precedent.
Let’s start with the obvious. Most appellate court opinions aren’t subject to a cert petition; almost no cert petitions are granted; only some granted cert petitions end up in full argument and a written opinion by the Supreme Court. Throughout this long process, parties may settle their cases and exit the system. They may do soduring briefing before the appellate court or after argument (with some limitations), they may settle after an opinion issues but before a cert petitition is filed, and they may settle thereafter until the Court rules (again, with some limitations based on mootness doctrine). This potential for settlement after the appellate court issues its mandate creates selection effects.
Though such selection effects are likely less predictable & more dominated by wealth & party characteristics than the immense selection that occurs in the district court, it remains that case that the universe of cases that survive the decision to appeal and the decision to grant certiorari is significantly winnowed. That winnowing produces a distinct set of cases. Cases before the Supreme Court contain legal & factual issues more finely balanced than those that issued from the courts below. To put it another way, cases are argued (usually) because the parties both believe they are going to win. If the parties are rational & wealth neutral, an assumption that sometimes holds, we should expect that the resulting decisions from the Court will be a bit of a random walk. (See my earlier post on bankruptcy scholarship for more on this hobby-horse of mine. Also, note that much of this applies to civil cases, not criminal cases, which create unique settlement patterns. The actual rate of reversal, over all cases, ranges between 60 and 75%.)
What’s the upshot? An appellate judge’s “reversal statistic” tells you less than you think about the “merits” of her opinions, or even how such opinions stacked up against governing Supreme Court precedent. Ironically, when a judge significantly departs from precedent either for or against the plaintiff, settlement may be particularly likely, as the parties’ chances above are quite clear: thus a judge who convinces her colleagues to depart from precedent often will almost never be reviewed or reversed by the Supreme Court. Cases where the judge stuck with precedent, by contrast, may face appeal and reversal, especially if the precedent shifted due to ideological change on the Court itself.
There’s lots of good work on this, much of it recent. And what it teaches me is that we have no idea what an appropriate reversal rate for an appellate judge ought to be. It also suggests that there is no way to evaluate the quality of an appellate judge’s work except to read her opinions and decide for yourself what you think of them. This is a clear instance where statistics mislead.
May 26, 2009 at 3:20 pm
Posted in: Empirical Analysis of Law, Supreme Court
Print This Post
Responses (5)
Steven Lubet - May 26, 2009 at 3:28 pm
Doesn’t SCOTUS reverse slightly more than 60% of all cases?
Dave Hoffman - May 26, 2009 at 3:29 pm
Steve, I think that’s a blended crim/civ statistic. But you are of course right that the overall reversal rate also strongly suggests that selection & case selection by the justices is operating pretty strongly.
Howard Wasserman - May 26, 2009 at 4:22 pm
There also is the simple fact that being reversed does not tell us anything about whether the lower-court judge was right or wrong, only that her views were or were not in step with those of the reviewing court. We are back to the old saw: We are not final because we are infallible, we are infallible because we are final.
Jake - May 26, 2009 at 6:22 pm
Does it make sense to evaluate a federal circuit court judge’s reversal/affirmance rate by limiting the analysis to cases that are granted cert based on circuit splits?
Sam Baumgartner - May 27, 2009 at 8:50 am
Apart from selection effects, a “sample” of six is not exactly large enough to draw valid descriptive inference, in this case the implied inference from her reversal rate to her level of fidelity to precedent.
Leave a Reply