Judging the Measures of Judges
posted by Corey Yung
In my last post, I discussed the measure I have proposed in my article for the judicial ideologies of federal appellate and district judges. That leads to the question: how do we know if my measure is “good?” Anyone can make up a bunch of numbers and formulas and declare their measure to be better than existing ones. How can it be said with any certainty that one measure of ideology is quantitatively or qualitatively better than another? That is one of the trickier questions in empirical legal studies of federal courts.
Consider the very interesting and valuable studyby Michael Heise and Gregory Sisk of judicial ideology in religious liberty cases published in the Northwestern University Law Review a few years ago. The study found, consistent with prior research, that ideology had a modest correlation with outcomes in religious liberty cases. How was ideology measured? Using Common Space Scores. What if the Common Space Scores were actually a poor measure of judicial ideology and votes in religious liberty cases were actually a “better” indicator of a judge’s ideology? How would we know? Heise and Sisk chose to use Common Space Scores even while noting in detail the potential problems with the Scores. Of course, Heise and Sisk did so in part because they framed their study as part of a response to the firestorm created by an article by Lee Epstein and Gary King attacking empirical legal studies by legal academics. The types of inferences that can be drawn from the Heise and Sisk study would change dramatically if Common Space Scores were not strong indicators of judicial ideology.
In the article that I previously discussed, Josh Fischman and David Law tackled this sticky problem by assessing the leading measures of ideology (as well as Fischman’s consensus model) based upon their ability to predict case outcomes. Specifically, Fischman and Law looked at whether Common Space Scores of the appointing President’s party were good predictors of panel votes in Ninth Circuit asylum cases. The assumption was made that a pro-asylum vote was indicative of liberalism. They found that Common Space Scores offered very little improvement over the appointing President’s party.
Following a similar methodology as Fischman and Law, I tested my Ideology Scores against the two leading measures in criminal cases. I used 1,000 criminal cases (selected using a pseudo-random number generator) from my dataset and recalculated my Ideology Scores without those cases to prevent circularity problems. I assumed that on average a vote for a criminal defendant was a more liberal outcome. I then performed logistic regressions on the panel median and net ideology scores using the three measures against outcomes in favor of criminal defendants. Notably, and perhaps surprisingly, neither Common Space Scores nor the party of the appointing President had a statistically significant relationship with votes for criminal defendants. In contrast, my Ideology Scores showed such a relationship for both the panel median and net scores (p=0.004 and p=0.013, respectively). I also performed goodness-of-fit tests (pseudo R2 tests) which assessed whether my measure was a better fit for the data in the 1,000 criminal cases. In every test, my measure had a better goodness-of-fit than the leading measures. The figures below shows the relationship between a criminal defendant’s winning percentage before a judge and that judge’s Ideology Scores and Common Space Scores.
The first graph shows that a liberal under the Common Space Score measure is only slightly more likely to vote for a criminal defendant than a conservative. In contrast, a liberal as defined by the Ideology Scores is much more likely to vote for a criminal defendant. This difference is largely consistent with anecdotal evidence of Clinton appointees. President Clinton ran for President contending that he would be tough on crime. He signed into law some of the harshest criminal laws in history. And according to many observers, his appointments to the federal judiciary were not particularly concerned with defendant rights or sympathetic to defendants in general. The Common Space Scores and appointing President’s party measures cannot account for the priorities of the Clinton administration because every appointment he made is assumed to be liberal. The Ideology Scores, being based upon performance of the judge, allow for party switching. As a result, a clearer trend of liberals voting for defendants emerged. A similar test was performed with civil rights (Title 7, ADA, and Section 1983) cases and the results were similar. Overall, the various indicia of being a “better” measure of judicial ideology favor the Ideology Scores against both the Common Space Scores and President’s party.
So, if there is reason to believe the Ideology Scores are a useful tool for measuring ideology, what can be done with that information? In my next few posts, I will talk about some of the findings in my paper regarding judicial ideology among the circuits, based upon law school attended, and related to prior work experience.
June 18, 2010 at 3:41 pm
Posted in: Empirical Analysis of Law, Legal Theory, Uncategorized
Print This Post
Responses (2)
MJG - June 18, 2010 at 6:05 pm
Didn’t Ward Farnsworth already do this and in more entertaining fashion?
http://www.michiganlawreview.org/articles/signatures-of-ideology-the-case-of-the-supreme-court-s-criminal-docket
Corey Yung - June 19, 2010 at 4:12 pm
Hi MJG,
I’m not sure what you mean by “this.” Farnsworth’s work concerns judicial ideology, but I’m not really sure what else it has in common with my project.
As for the entertaining part, I can’t comment …
Corey
Leave a Reply