Measurable Things
posted by Dave Hoffman
The Misleadingly Convenient Source of Information
A common criticism one reads of ELS is that “too much of the work is driven by the existence of a data set, rather than an intellectual or analytical point.” It’s ironic that this is the very critique that the realists made of traditional legal scholarship. Consider the great Llewellyn:
“I am a prey, as is every man who tries to work with law, to the apperceptive mass. I see best what I have learned to see. I am a prey, too — as are the others — to the old truth that the available limits vision, the available bulks as if it were the whole. What records have I of the work of magistrates? How shall I get them? Are there any? And if there are, must I search them out myself? But the appellate courts make access to their work convenient. They issue reports, printed, bound, to be had all gathered for me in the libraries. The convenient source of information lures. Men work with it, first, because it is there; and because they have worked with it, men build it into ideology. The ideology grows and spreads and gains acceptance, acquires a force and an existence of its own, becomes a thing to conjure with: the rules and concepts of the courts of last resort.”
Or to put it differently, all of our work – quantitative empiricists, doctrinalists, corporate finance wizards, administrative regulation parsers, legal philosophers, and derivative social psychologists alike – is driven by the materials at hand. For most lawyers and legal academics, appellate opinions are the most convenient pieces of information available; we use such opinions to create mental models of what the “law” is, and (ordinarily in legal scholarship) what it ought be. Indeed, whenever trial court opinions are cited, they are often discounted as aberrant or transitory, in part because they are known to be unrepresentative!
Why, you might wonder, is the convention of data-driven-scholarship a particular problem in quantitative empirical work? ELS’s detractors make three interrelated claims:
- If the availability of the data drives your research question, it’s not obvious that the questions you are asking are interesting, nor that they related to a real-world problem of note;
- ELS, like law and economics, appears to feel like science — it tends to want to displace “softer” disciplines, such as the unpacking and categorization of appellate court opinions. Because legal scholars lack empirical training, the black box effect looms large. Therefore, it’s important, but hard, to be especially on guard against shoddy work; and
- It’s normal & appropriate to read appellate opinions to know what the “law” is – because the “law” isn’t really (or mostly) about how litigants dispose of their cases, but rather how judges justify the imposition of force.
The first claim is true, though of course it would sweep almost all legal scholarship under its rug. (Which wouldn’t be so bad! Most legal scholarship, like most judicial opinions and all blog posts, influences no one and matters not a whit to the turning of the world.) The empirical work I do usually involves hand-coding of datasets, in part to avoid precisely this problem.
The second claim is harder to operationalize – but has bite. True: the rhetoric of empirical legal studies tends to be triumphalist: “I have found a statistically significant result: therefore, you are wrong!” (Or, better by far, “I haven’t, so you may be right!”) But isn’t arrogance about method, subject and approach a sin of almost all legal scholarship? As for the argument about training, the cost of learning how to become an educated reader of a regression isn’t terribly high, especially for a legal academic who has summers off.
The third claim to me feels like twaddle, with some minor exceptions. To me the more interesting questions about law turn on how citizens experience and understand legal practices and rules, not how they are crafted and justified in the tribunals above. Even if this is just a preference, I tend to think that the claim about what the law is represents exactly the data-driven ideology that Llewellyn called out. And at the very least, those who subscribe to the ideology of appellate opinion practice are just as likely to reach beyond their data to tell us what is happening in the world as are the new quantitative ELSers.
February 26, 2012 at 1:10 pm
Posted in: Behavioral Law and Economics, Empirical Analysis of Law
Print This Post
Responses (3)
A.J. Sutter - February 26, 2012 at 7:04 pm
“The more interesting questions about law turn on how citizens experience and understand legal practices and rules” — looks like you should be employing qualitative research methods, then. And talking to real litigants, not hypothetical ones brought “into the laboratory,” as you once put it.
Dave Hoffman - February 26, 2012 at 10:50 pm
AJ
I am sorry that you are fixating on the word laboratory, when obviously we meant in the paper a psych experiment with real money incentives, where the “experiment” is performed in real time and the subjects under observation. This is a common term in social psychology, maybe it isn’t translating well.
I am in favor of a big tent, including qualitative research methods. Please let’s not get distracted and hijack the thread…
A.J. Sutter - February 27, 2012 at 8:21 am
Don’t worry, I’m not fixating, just needling. But regardless of the terminology, such social psychology experiments don’t seem like a good way to get at how citizens experience, etc. The experience of citizens with the real legal system seems more to the point. If the claim is that it’s difficult to access them and that toy experiments with “real money incentives” may be the most convenient substitute, that would tend to validate the criticism of ELS that you’re attempting to counter at the outset of this post.
Leave a Reply