Judged by the Company You Keep
posted by Corey Yung
Last week, I tried to outline the difficulties associated with measuring judicial ideology in regards to the limited alternatives that have been offered by scholars. In this post, I hope to describe how I have measured it and attempted to overcome the various obstacles brought about by my methodology.
My idea for identifying the ideologies of federal appellate judges was to determine the rates at which such judges agree and disagree with “conservatives” and “liberals” on the bench. The assumption was that like-minded judges will vote together more often and judges with dissimilar ideologies will tend to disagree. By focusing on the agreements and disagreements among the judges, the goal was to pinpoint their respective ideologies (via “ideal points”). This is an agnostic method that necessarily faces all of the shortcomings of such an approach that I previously described.
The initial concern with such a method is that there are far too few disagreements among the judges on the Courts of Appeals. Indeed, in the 10,242 cases in my dataset, there were only 288 dissents (including partial dissents). Some judges who participated in over 100 cases were not on a panel in which there was a single dissenting vote. Looking at the Courts of Appeals alone was, thus, unlikely to offer much information. My solution was to treat the district judges being reviewed as pseudo-fourth members of the appellate panel. After all, the district judge reviewed the same legal issue as the appellate panel and rendered judgment on that very same issue. Notably, there are far more disagreements with district judges in the form of reversals. Also, by including the district judges, my methods also allowed data to be harvested from unanimous affirmances as well (as described below).
However, my solution faced numerous difficulties which necessitated adding nuance to the general concept for the measure. Importantly, I incorporated these details into my Ideology Scores:
- Case Mixes – different judges hear different types of cases. In my dataset, the most significant difference that was relevant to the measure was in relation to criminal and civil cases. District judges were reversed twice as often in civil cases. Accordingly, each judge (or other unit of measure) was analyzed assuming an average distribution of civil and criminal cases.
- Standards of Review – because appellate judges review district court judges with deferential and non-deferential standards of review, there is a need to adjust for standards used. For example, a reversal in a criminal case with a deferential standard of review is far less common than a reversal in a civil case with a non-deferential standard. So, in addition to the case mix adjustment, judicial votes were further weighted based upon the standard of review used.
- Rates, not Frequencies – judges have differing co-panelists and judges reviewing their decisions. As a result, it was essential to focus on the percentages of agreements rather than the absolute numbers. So, judges were evaluated for the percentage with which they agreed with judges appointed by Republicans and by Democrats in each of the relevant sub-categories (i.e., criminal case with a non-deferential standard). The use of rates also provides a stable baseline for comparisons so that judges who are in the minority ideology on their court are measured in the same manner as those in the majority group. Rates also incorporate affirmances into the measure – something that previous agnostic measures have been unable to do.
- Panel Effects – one of the most significant results in recent empirical legal studies has been that the composition of the panel determines to a large degree how an individual judge votes. Judges who sit with more Republican appointees will tend to vote in a more conservative direction than if they sat with Democratic appointees. Accordingly, the measure was adjusted to assume that judges had the same co-panelists. This was done by determining the differential in co-panelists arrangements between two Republican appointees and two Democratic appointees.
- Inter-Circuit Adjustments – the Courts of Appeals in my dataset have their jurisdiction determined by geography. As a result, they review differing sets of cases and differing district judges. This would tend to make inter-circuit comparisons impossible. However, the circuits are not entirely insular. Judges who have taken senior status often travel to different circuits and sit by designation. In my dataset, there were 2,472 votes issued by 26 “traveling judges” in either their home or away circuits. Using data from how those judges voted at home and away, adjustments were made to the Ideology Scores of judges on each circuit. This adjustment assumed that a judge remains constant in his or her ideology regardless of circuit. Again, accounting for all of the above adjustments, it was determined whether a particular circuit made travelling judges vote more liberally or conservatively.
Based upon all of those adjustments, individual Ideology Scores were determined for every federal appellate judge and district judge who issued an opinion in 2008 or had their opinion reviewed in 2008. However, due to sample size concerns, the article focused on 138 judges who sat on the Courts of Appeals that year. In my next post, I will discuss how my measure does against the two leading measures. Below are some example scores (in comparisons to the other leading measures) from notable judges studied in my article. Total interactions refers to the total number of co-panelists and district judges included the sample. Higher scores are more conservative and lower scores are more liberal.
| Judge | Circuit | Total Interactions | Party of Appointing President | Common Space Score | Ideology Score |
| Cook, Deborah L. | 6 | 178 | Rep. | 0.226 | 77.2 |
| O’Scannlain, Diarmuid F. | 9 | 270 | Rep. | 0.023 | 59.7 |
| Easterbrook, Frank H. | 7 | 241 | Rep. | 0.559 | 55.8 |
| Jones, Edith H. | 5 | 291 | Rep. | 0.502 | 22.0 |
| Boggs, Danny J. | 6 | 156 | Rep. | 0.339 | 17.3 |
| Thomas, Sidney R. | 9 | 341 | Dem. | -0.209 | -5.4 |
| Posner, Richard A. | 7 | 254 | Rep. | 0.034 | -9.9 |
| Reinhardt, Stephen R. | 9 | 143 | Dem. | -0.409 | -23.3 |
| Williams, Ann C. | 7 | 253 | Dem. | -0.345 | -31.5 |
| Wood, Diane P. | 7 | 277 | Dem. | -0.3795 | -37.2 |
| Sotomayor, Sonia | 2 | 240 | Dem. | -0.318 | -40.1 |
| Wardlaw, Kim M. | 9 | 255 | Dem. | -0.338 | -63.3 |
June 15, 2010 at 3:55 pm
Posted in: Empirical Analysis of Law, Legal Theory, Uncategorized
Print This Post
Responses (6)
Bill Reynolds - June 15, 2010 at 5:31 pm
Do you include unpublished opinions in your dataset? They are tricky to handle because review of unpublished opinionsis much more concentrated in central staff and, therefore, present different problems than published opinions? Moreover, unpublished opinions are 83% of the caseload of the federal appellate courts.
Corey Yung - June 15, 2010 at 5:39 pm
Hi Bill,
My dataset includes unpublished opinions. Because standards of review were my original focus (for my measure of judicial activism), that was the touchstone for how the data was accumulated. The dataset includes 2008 cases from the 11 numbered circuits (Fed. and D.C. are excluded) that used a standard-of-review-related word excluding immigration and habeas cases. I have not yet focused on the differences regarding published and unpublished opinions in regards to my measures. However, as you mention, because they are a significant part of the federal docket and because prior databases had largely omitted them, I felt it important to include unpublished opinions.
Corey
Dan Culley - June 15, 2010 at 5:59 pm
It seems odd to reduce ideology to a single score, even an agnostic one, unless I’m missing something. Even within a single issue, there’s rarely a linear spectrum of possible ideologies. It seems like it might make more sense to identify distinct clumps of judges who tend to vote like one another within some given threshold.
Bryan Gividen - June 15, 2010 at 6:46 pm
Dan,
Corey addresses that shortcoming in his full paper (http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1618665). He admits his linear ideology scale is merely a starting point for further research.
Ken Rhodes - June 16, 2010 at 1:02 pm
In ops research there is an important concept called “Pareto optimal.” A set of solutions, or a “solution space,” is Pareto optimal iff no solution is better than others in the set, and no solution outside the set is better than the ones in the set. (That’s a simplification, but it gets the main point across.)
When one has a diverse set of objective functions, as do most of us in trying to define our political preferences, then a single linear measure of satisfying those objectives is frequently not practical.
In the current context, I think measuring “ideology” is seen as an attempt to measure the degree of conformance to a set of solutions that conform to our personal objectives.
This can be a VERY complicated problem.
Corey Yung - June 19, 2010 at 4:11 pm
Hi Dan and Ken,
Bryan is right that I think more research needs to be done on this. I would also add a few other points. First, it is very difficult to identify in an objective manner a variety of different viewpoints of judges even on a single issue. For example, I’m quite sure that there are numerous perspectives of free speech restrictions of obscenity. Judge Kozinski’s in particular might stick out. However, how many judges could you really identify as belonging to different camps? To have an effective measure of ideology, you would probably need to identify multiple persons in each camp on each circuit (due to the limited interaction of the judges on the circuits). And how would you objectively figure out who fits in what camp? There are no easy answers to those questions right now.
Second, at least in one sense, viewpoints in judicial cases can usually be reduced to two sides: the parties. Usually one wins and one loses. From that standpoint, in the aggregate, we can identify where a judge fits. For example, if a judge is really liberal and often feels that his/her colleagues are not liberal enough, he or she is still not likely to dissent from their decisions. However, he or she might be more likely to dissent from moderate conservative decisions.
Third, in our American two-party system, there is a common collection of views as defined by our political system. Because federal judges are appointed as a product of that system, it should not be surprising if a great many of them are reducible to our normal ideological continuum.
If someone could identify: 1) more strains of ideology; and 2) an objective means of coding for them, they would really have something worthwhile. However, given the state of current ideology measures, I think that is a project for another day.
Corey
Leave a Reply