How Can Judicial Ideology be Measured?
posted by Corey Yung
In my last post, I talked about the shortcomings with the leading measures of judicial ideologies. There are strong reasons, however, why those measures have dominated empirical legal research. If a scholar wants to assess the ideology of a judge, he or she is likely to try a technique that fits within one of these three categories: Case Outcome Coding, External Proxies, and Agnostic Coding. Each of these types of measures has advantages and disadvantages for federal judges not serving on the Supreme Court.
Case Outcome Coding – this category relies on a researcher going through a sample of cases and coding whether the judge’s or panel’s vote were “liberal” and “conservative” in ideological direction. While this technique can work reasonably well at the Supreme Court level (although there are shortcoming there as well), it is extremely difficult to apply to the Courts of Appeals or federal district courts. There is an enormous amount of labor required for sufficient samples to be accumulated for individual judges. Further, coding decisions are much more subjective than at the Supreme Court level as most of the federal docket is filled with cases that have little political salience. As Tonja Jacobi and Matthew Sag recently observed, “the last four decades of empirical scholarship have proceeded without a sophisticated objective measure of case outcomes.” Given the low level of disagreement among judges on the Courts of Appeals (due to consensus norms, “easy” cases, or other strategic incentives), there is an additional problem of making valid assessments without an enormous sample of data for each judge. It is also possible that a researcher could try to code the ideological direction of methods instead of outcomes, but such a technique would tend to accentuate the difficulties described above. As a result, such measures have never been attempted on a comprehensive basis for individual judges on the federal appellate or district courts.
External Proxies – this category includes techniques which rely on a party external to the judges reviewing the cases to assess the ideology of the judges. The President’s political party and Common Space Scores are external proxies. Such measures are often easy to implement, but rely on sometimes questionable assumptions about the quality of the proxy utilized. Especially for the lower court judges, it is unclear to what degree ideology is the driving force in a President’s appointment of a judge. I have addressed the general shortcomings with this type of measure in my previous post.
Agnostic Coding – this category of measures is based upon identifying voting blocs among judges. I believe the term “agnostic coding” was created by Fischman and Law and, in my mind, perfectly captures how these metrics work. Agnostic coding relies on the premise that like-minded judges will vote together more often than judges of opposing ideology, but does not require any ideology coding of the case outcome. By looking at who an individual judge votes with and against most often, assessments of that judge’s ideology can be made. The Martin-Quinn scores, which are becoming the leading measure of ideology at the Supreme Court level, rely on an agnostic technique. Agnostic coding allows for analysis of large amounts of data because of the limited, typically objective, coding involved. However, such a technique has generally been thought to be impossible to apply to the Courts of Appeals and federal district courts. For district judges, there are no “voting blocs” because each judge sits alone. For appellate judges, simulation techniques like those used by Andrew Martin and Kevin Quinn are largely hindered by the very low dissent rate on the Courts of Appeals (less than 2% in my data). Since agnostic coding does not include the ideological direction of cases, techniques in that category have not been able to harvest data from unanimous panels. Beyond ignoring the overwhelming majority of cases, agnostic coding using current methods also requires strong prior assessments of what constitutes “liberal” or “conservative” ideology. Without such assessments, the results may indicate different groups of judges, but there won’t be any clarity as to what ideologies that those groups embody.
Given the numerous obstacles for each possible methodology technique, it is not surprising that scholars have not added to the arsenal of measures for judges on the Courts of Appeals and federal district court judges. In my next post, I will attempt to outline my method for measuring judicial ideology of federal appellate and district judges and address how I confront the various difficulties that I have discussed.
June 10, 2010 at 11:11 am
Posted in: Empirical Analysis of Law, Legal Theory
Print This Post
Responses (1)
Anon - June 11, 2010 at 1:12 am
For economic liberalism, you could look at cases that involve individuals or gov agencies v. corporations.
Leave a Reply