Everything Is Empirical
posted by David Fagundes
What is “empirical” legal scholarship? I think most people in the contemporary legal academy would respond that “empirical” work refers to the formal statistical analysis of data about law. Indeed, if you search the Googles for “empirical scholarship”,* you get all manner of links to workshops for learning how to analyze data about law using statistical methods as well as papers about and some criticism of this approach. This august site’s very own subcategory for “Empirical Analysis of Law” includes posts only about statistical analysis of law. And my quick, incomplete scan of the last CELS program yielded almost exclusively papers in this quantitative vein.
Something’s always struck me as a little strange about this use of “empirical.” The weirdness has nothing to do with the merits of statistical analysis of data relating to law (which I generally find, despite my not being conversant with it, really interesting). Rather, it’s a much narrower question about terminology, and in particular the tendency of the legal academy to refer only to statistical analysis of data as “empirical.” The term “empirical”, outside the legal academy at least, is not limited to quantitative studies of data using statistics. Rather, the term refers more broadly to reasoning that derives from observations about the world rather than from theoretical propositions (this roughly but inexactly tracks the distinction between inductive and deductive reasoning).
So while “empirical” work is undoubtedly empirical, so is (at least by the definition I’m using above) much legal scholarship that is not typically so described. I say more about this anomaly, and why it matters, below the fold.
Much scholarship that is empirical (though not commonly included within the contemporary mantle of “empirical” with respect to law scholarship) uses qualitative rather than quantitative methodology. My very own dean has interviewed lawyers from around the world as the basis for his work on international commercial arbitration. I recently wrote a paper about the extralegal regulation of roller derby pseudonyms using a similar methodology (I even called it an “empirical” paper in the abstract to see if I’d get any reaction to that—I didn’t). These papers may use words and text rather than numbers and graphs as their data, but they both start with observations about the world (as opposed to abstract theories) in order to induce general points, and in that sense neither is any more “empirical” than the other.
I once lobbed this idea at some colleagues who are also stats enthusiasts, and received (somewhat to my surprise) a basically positive reception. They suggested that legal scholarship could avoid any inaccuracy by simply calling work in this vein either quantitative or qualitative empirical work. And for a while this seemed like a pretty decent resolution.
Then I got to thinking that the qualitative/quantitative distinction does not sufficiently describe the universe of legal scholarship that is empirical. To take just one example, consider traditional legal scholarship that seeks to describe the doctrinal contours of some area of law. This kind of work starts with data from the world (cases and various state statutes relating to the law of X), and then seeks to draw general conclusions based on that data (the law of X has the following general qualities). So perhaps even enfolding qualitative work into the empirical universe understates the extent to which legal scholarship generally is strongly empirical.
Not all legal scholarship meets the definition of “empirical” I’ve been using in this post, though. A piece that starts with a theory about the way the world should be and then suggests how law should be changed to conform to that theory doesn’t seem empirical in this way (i.e., doesn’t use inductive reasoning, which may not be quite the same thing). Hence the title of this post may overclaim a bit (but I’m keeping it because it’s a reference to that book that came out some time ago that everyone said was good but that I never got around to reading).
One other answer to all this is, of course, so what? One might say that this is just a semantic point, and that since everyone knows that in law scholarship, “empirical” is a synonym for “quantitative,” there’s no confusion and hence no concern. For at least one reason, though, I think this is a distinction with a difference. “Empirical” is not just a neutral term that happens to describe a particular methodology. It may be understood to connote, rightly or not, a certain degree of rigorousness and exactitude that can set it apart from, and perhaps even above, other methodologies. And for that reason, to exclusively bestow on quantitative legal scholarship the title “empirical” may give it an unwarranted patina of credibility vis a vis other approaches that equally merit that description.
*Not a typo–merely asking Google to search for “empirical scholarship” produced pages of hits exclusively about empirical *legal* scholarship, which leads me to suspect that the terminological quirk I discuss in this post is limited to legal academia.
August 11, 2011 at 9:28 pm
Posted in: Uncategorized
Print This Post
Responses (8)
Joe - August 12, 2011 at 12:24 pm
This is very apropos of a current project. I’m looking at an area of law that has evolved very differently in federal courts vs. state courts (which are nearly uniform on this issue). A few scholars have offered abstract theories to explain the difference in recent years, but no one (that I can find, at least) has ever tested these theories to see if they are, well, right. So I am going back through all fifty states and tracing the evolution of the law in this particular field to see if confirms or refutes the theories. (The results so far are unsurprisingly mixed.)
Is this “empirical” work in a broad sense? Of course. I’m using real world observations to evaluate an abstract idea. But it’s certainly not an ELS piece; there will be very few numbers at the end of the day, and my results are likely to be of the squishy type in a lot of cases (e.g., the law in State X developed sorta kinda in a way that the theories would predict, but not exclusively so).
A.J. Sutter - August 12, 2011 at 1:32 pm
Good insight, and no need to be so tentative or apologetic about suggesting why the distinction is important. If anything, you should be more assertive.
Very likely the equation of statistical (usually econometric) methods with “empirical” ones grows out of the law & economics trend, which appropriates neoclassical economics’ own math-envy of the (‘empirical’) natural sciences. My impression is that qualitative researchers in law seem to prefer to emphasize that they’re using, say, an anthropological approach or a sociological one, rather than an “empirical” one (am I mistaken?) — that too may arise from the cultural habits of sociologists and anthropologists, and their attention to differentiating one field from the other.
Your footnote is especially rich. The ELS usage (“Its’s empirical“) calls to mind the old Duck’s Breath Mystery Theater character Mr. Science, who used to trump all doubts with his boast “I have a Master’s degree — in Science!”
Bob Lawless - August 12, 2011 at 4:01 pm
David, I enjoyed your post and have never been too happy with the label “empirical legal studies” for many of the reasons you describe.
I don’t think you would get much disagreement from most anyone who does heavily quantitative empirical work. Of course, good qualitative empirical work is valuable and often can succeed on a topic where quantitative work cannot. Whether it is qualitative or quantitative, good empirical work is systematic, replicable, and valid.
Where I think you get pushback is the use of the term “qualitative empirical research” as cover for “bad empirical research.” I have heard faculty workshops where someone presents a few stray interviews as “qualitative empirical research,” when they should have said, “I called up a few people I know, and they agree with me.”
Patrick S. O'Donnell - August 12, 2011 at 9:03 pm
In July of last year there was a blog post by John Inazu (with a comment by yours truly) at The Faculty Lounge that raised similar issues: http://www.thefacultylounge.org/2010/07/some-reflections-on-empirical-legal-studies.html
dave hoffman - August 12, 2011 at 9:39 pm
Hi Dave
This is a nice post – though I agree with Bob that good qualitative work is of course empirical and considered so by people in the field. Though I believe that CELS has a bit of a quantitative bias , which is unfortunate.
I also agree that there is really a ton of bad “I sent out surveys” or “I talked with lawyers” work out there. Whether it is “empirical” or not seems somewhat besides the point.
A.J. Sutter - August 12, 2011 at 9:56 pm
Bob, qualitative research (as such, and not as “qualitative empirical research”) has a longer pedigree in anthropology, sociology and other social sciences. While it’s hard to call anything rigorous in a field that mixes positive and normative aspects, there is a good deal of learning from a history of practices — try an Amazon search on “qualitative research”. Statistical research is well-known for being capable of abuse, too, and suffers from the further drawback that, both because it is more arcane and because of our undue respect for the quantitative and for the “average” case, it’s easier to bamboozle people with statistics than with qualitative research.
David Fagundes - August 14, 2011 at 8:54 pm
Thanks, all, for the interesting comments. I think there are a few propositions that we can agree on. The first, totally uncontroversial proposition, is that good scholarship is good scholarship, regardless of nomenclature. The second proposition, slightly less obvious but still apparently unobjectionable, is that qualitative research merits the moniker “empirical” as much as quantitative research does. Lurking in there may be a third proposition, that work’s status as “empirical” or otherwise is orthogonal to its quality.
So is there anything controversial here? Possibly. There may be some disagreement as to whether it matters that some empirical work is deemed “empirical” while other empirical work is not. One could say (and I think this post by Michael Heise over at the ELS Blog suggests: http://www.elsblog.org/the_empirical_legal_studi/2011/08/more-musings-on-what-is-empirical.html) that calling work “empirical” is value-neutral, so that there are no positive connotations conferred by the term. On this view, “empirical” is just an innocuous descriptive.
In a vacuum, I think that’s probably right. But the concern I have is that referring only to quantitative work as “empirical” creates a negative implication that qualitative work is _not_ empirical. This suggests that the latter lacks the same degree of rigor and precision as quantitative methodologies, and may cause it to be taken less seriously or regarded more poorly by those in or out of the field.
Of course, it may be the case, as Bob and Dave have pointed out, that qualitative empirical work could actually lack rigor and precision, as where a few phone calls are held out as sufficient to prove some factual proposition. But I don’t think qualitative work is distinctively susceptible to this charge. AJ correctly points out that quantitative work can be executed poorly as well, and that there may be a particular concern when that happens. Most people can find the error when someone makes a few phone calls and presents that as the basis for “empirical” research. But even sloppy errors in quantitative work are hard to spot (at least for those of us without formal training).
A.J. Sutter - August 15, 2011 at 12:48 am
This thread provides evidence of a sort of paradox. On the one hand, legal scholars seem to be reluctant to recognize scholarship in other disciplines. Take the fixation on the red herring of “a few phone calls” as qualitative research: even if some conference presenters did make this silly error, some comments seem to find it prima facie plausible that maybe that’s more or less all there really is to doing qualitative research (even if some comments try to defend the research from the charge). Any cultural anthropologist would find this view about as plausible as we might find the Tea Party’s statements about the federal debt ceiling, or Sarah Palin’s pronouncements on foreign affairs. I.e., a little interdisciplinary exposure is all it takes to recognize the charge as obviously silly, and not meriting such earnest angst. If the lousy qualitative research methods are in fact more widespread in the legal academy, as Dave H. suggests, than just a few silly papers, this just proves my point about the parochial POV. That parochiality would be a better object for the angst.
On the other hand, legal scholars (among others) nonetheless crave to borrow the authority of other academic disciplines. A big reason for using the word “empirical” is to borrow the authority of the so-called hard sciences (from which the notion of “value-neutral” is also borrowed). That borrowed authority is also the source of any negative implication of being non-empirical, which David F. worries about. Such a negative implication per se may be less a cause for concern than is the power-grabbing pretense that gives rise to it.
A compromise might be to re-identify ELS as “Econometric Legal Studies”. That way the acronym could stay intact, and the methodology would be more explicit. Resistance to this could be a good diagnostic that the authority grab is the real motivator in the movement.
Leave a Reply