Law’s Arbitrary Endpoints
posted by Dave Hoffman
For many purposes, a season is an arbitrary endpoint to measure a baseball player’s success. To extract utility from performance data over time, you need to pick endpoints that make sense in light of what you are measuring. Thus, if we want to know how much to discount a batter’s achievements by luck, it might not make sense to look seasonally – - because there’s no good reason to expect that luck is packaged in April-to-October chunks. Nonetheless, sabermetricians commonly do talk about BABIP seasonally — thus, Aaron Rowand had an “unusually lucky 2007,” and has since regressed himself off of a major league payroll. Jayson Werth, similarly, is feeling the bite of lady luck this “season.” (For pitchers, the analysis makes more sense, since the point of BABIP is that pitchers can’t control outcomes once the ball hits a bat. Thus, the Phillies fifth starter is supposedly not nearly as good as his haircut suggests he ought to be.)
This bias toward artificial endpoints affects legal studies, though less obviously. There aren’t legal seasons. (It’s always a time to weep, to bill, to work, to reap.) But we still organize our analyses around units which might not exactly track the underlying item of interest. We want to study disputes, but we look at the records of filing and verdicts (which are a smaller unit in time than the object of study). We wish to examine ideological voting patterns on the Court, but we organize our study by Term. We want clear signals of young lawyer quality, but we look at grades in law school, for (mostly) the first three semesters). We want to know how law schools’ influence hiring practices, but we look at deadline-generated 9-month hiring reports. Different slices at these numbers may produce quite different results — heck, one of the reasons that USNews obtains variable rankings is that they keep on moving the endpoints of the analysis in ways that are perfectly unclear.
There’s no complete solution to the endpoint problem – at least, not one that’s easily compatible with the project of data-driven legal analysis. It’s important, therefore, to be especially careful when reading studies that take advantage of convenient legal periods. A prime example is the Supreme Court’s “Term.” I have no good reason to expect that the Justices’ behavior changes meaningfully from one Term to another — absent an intervening change in personnel. So, Term analysis is convenient, but I bet it misleads. Comparing the performance of a Circuit from one Term to another is similarly odd — whatever the value of the “reversal rate” inquiry, it surely doesn’t turn on Terms!
This set of cautions might be extended to a more general one, directed at folks who are interested in doing empirical work but haven’t yet begun to collect data. If your outcome of interest is measured monthly, seasonally or yearly, consider whether that unit of measurement reflects something true about the data, or is merely a convenience. If it’s the latter, proceed with caution. Obviously, this isn’t at all a novel caution, but the persistence of the error suggests it can’t be made often enough.
September 7, 2011 at 11:40 am
Posted in: Empirical Analysis of Law
Print This Post
Responses (1)
Matt - September 7, 2011 at 1:42 pm
if we want to know how much to discount a batter’s achievements by luck, it might not make sense to look seasonally – – because there’s no good reason to expect that luck is packaged in April-to-October chunks.
I’m a bit less sure that this is unreasonable, in part because things change in the off-season in ways that make a season a plausible unit of analysis- players change, getting better or worse for various reasons, opponents change, the way fields play, and so on. If anything, a season it too long to look at for “luck” in many cases, but you’ll see this noted in most semi-careful analysis. I’m not sure if there’s a parallel for supreme court terms, though to the extent the court is influenced by the general political climate (as it seems to be), there would be some grounds for breaking analysis into some sort of chunk (though probably chunks longer than a term in most cases.)
Leave a Reply