The Future of Empirical Legal Studies
posted by Dave Hoffman
Reading these two articles on the problems of complexity for sabermetrics, I wondered if the empirical legal studies community is coming soon to a similar point of crisis. The basic concern is that sabermetricians are devoting oodles of time to ever-more-complex formulae which add only a small amount of predictive power, but which make the discipline more remote from lay understanding, and thus less practically useful. Basically: the jargonification of a field. Substituting “law” into Graham MacAree‘s article on the failings of sabermetrics, we get the following dire warning:
“Proper [empirical legal analysis] is something that has to come from the top down ([law]-driven) rather than the bottom up (mathematics/data driven), and to lose sight of that causes a whole host of issues that are plaguing the field at present. Every single formula must be explainable without recourse to using ridiculous numbers. Every analystmust be open to thinking about the [law] in new ways. Every number, every graph in a [ELS] piece musttell a [legal] story*, because otherwise we’re no longer writing about the [legal system ] but indulging in blind number-crunching for its own sake. …
Surveying the field, I no longer believe that those essential precepts hold sway over the [ELS] community. Data analysis methods are being misapplied and sold to readers as the next big thing. Articles are being written for the sake of sharing irrelevant changes in irrelevant metrics. Certain personalities are so revered that their word is taken as gospel when fighting dogma was what brought them the respect they’re now given in the first place. [ELS] is in a sorry state.
How do we fix it? Well, the answer seems simple. [ELS] shouldn’t be so incomprehensible so as not to call up the smell of [a courtroom, or the careful drafting of the definition clauses in a contract, or the delicate tradeoffs involved in family court practice, or the importance of situation sense]. Statistics shouldn’t be sterile and clean and shiny and soulless. They shouldn’t just be about [Law]; they should invoke it. Otherwise, they run the risk of losing the language which makes them so special.”
Note: this is an entirely different than Leiter’s 2010 odd critique that ELS work was largely mediocre. The problem, rather, is that the trend is toward a focus on more complex and “accurate” models, often without the input of people with legal training, and insufficient attention to how such models will be explained to lawyers, judges and legal policymakers. (See also all of Lee Epstein’s work.)