Site Meter

Category: Empirical Analysis of Law

0

What Should We Be Working On? Empirical Civil Procedure Post CELS

Earlier this week, I argued that civil procedure empiricists are spending too much time on the Twiqbal problem.  That’s not the same as saying that Twiqbal is an unimportant set of cases.  It probably signals an important shift in federal pleading doctrine, and, arguably, some litigants we care about are being shut out of federal court. I mean to say merely this: the amount of attention paid to Twiqbal is exceeding its importance to litigants (over state and federal court).  Our focus is being driven largely by data availability and law professor incentives. We can do better.

I’m starting to make a genre of these “people should be writing about X not Y” posts.  Boy, that could get tiresome fast!  Luckily, no one actually has to listen to me except for the poor 1Ls.  In any event, it seemed useful to start a conversation about what topics are more worth writing about than Twiqbal. Use the comment thread below to generate a list and if there’s enough interest I’ll create a poll. To qualify, the topic has to be real-data-driven (i.e., not merely doctrinal analysis, not experimental, etc.); and there must be a way, in theory, to get the data.  For example,

  • Does law influence outcomes in small claims court?
  • How well do choice of law clauses work in state court?
  • When do attorneys matter?
  • What are the determinants of summary judgment grant rates in state courts?
  • Is there a way to get a handle on which cases are being “diverted” to arbitration or “carved-[back]-in“?

 

0

CELS VII: Data is Revealing Part 2

 

Shouldn't it be "data are revealing?"

Shouldn’t it be “data are revealing?”

[This is part 2 of my recap of the Penn edition of CELS, promised here. For Part 1, click here.  For previous installments in the CELS recap series, see CELS IIIIVV, and VIVII.]

Where were we?  I know: throwing stink-bombs at a civil procedure panel!

At the crack of dawn saturday I stumbled into the Contracts II panel. Up first was Ian Ayres, presenting Remedies for the No Read Problem in Consumer Contracting, co-authored with Alan Schwartz.  Florencia Marotta-Wurgler provided comments.  The gist of Ayres’ paper is that consumers are optimistic about only a few hidden terms in standard-form contracts. For most terms, they guess the content right. Ayres argued that should be concerned only when consumers believe that terms are better than they actually are.  The paper  proposes that firms make such terms more salient with a disclosure box, after requiring firms to learn about consumer’s knowledge on a regular basis. Basically: Schumer’s box, psychologically-calibrated, for everyone.  Florencia M-W commented that since standard-form contracts evolve rapidly, such a calibrated disclosure duty might be much more administratively complex than Ayres/Schwartz would’ve thought.  A commentator in the crowd pointed out that since the proposal relies on individuals’ perceptions of what terms are standard, in effect it creates a one-way ratchet. The more people learn about terms through the Ayres/Schwartz box, the weaker the need for disclosure. I liked this point, though it appears to assume that contract terms react fairly predictably to market forces. Is that true?  Here are some reasons to doubt it.

Zev Eigen then presented An Experimental Test of the Effectiveness of Terms & Conditions.  Ridiculously fun experiment — the subjects were recruited to do a presidential poll. The setup technically permitted them to take the poll multiple times, getting paid each time.  Some subjects were exhorted not to cheat in this way; others told that the experimenters trusted them not to cheat; others were given terms and conditions forbidding cheating. Subjects exhorted not to cheat and trusted not to cheat both took the opportunity to game the system significantly less often than those presented with terms and conditions. Assuming external validity, this raises a bit of a puzzle: why do firms attempt to control user behavior through T&Cs? Maybe T&Cs aren’t actually intended to control behavior at all! I wondered, but didn’t ask, if T&Cs that wrapped up with different formalities (a scan of your fingerprint; a blank box requiring you to actually try to sign with your mouse) would get to a different result.  Maybe T&Cs now signal “bad terms that I don’t care to read” instead of “contract-promise.”  That is, is it possible to turn online T&Cs back into real contracts?

Next, I went to Law and Psych to see “It All Happened So Slow!”: The Impact of Action Speed on Assessments of Intentionality by Zachary C. Burns and Eugene M. Caruso. Bottom line: prosecutors should use slow motion if they want to prove intent. Second bottom line: I need to find a way to do cultural cognition experiments that involving filming friends jousting on a bike. I then hopped on over to International Law, where Adam Chilton presented an experimental paper on the effect of international law rules on public opinion. He used a mTurk sample.  I was a concern troll, and said something like “Dan Kahan would be very sad were he here.” Adam had a good set of responses, which boiled down to “mTurk is a good value proposition!”  Which it is.

After lunch it was off to a blockbuster session on Legal Education. There was a small little paper on the value of law degrees. And then,  Ghazala Azmat and Rosa Ferrer presented  Gender Gaps in Performance: Evidence from Young Lawyers. They found that holding all else equal, young women lawyers tend to bill somewhat fewer hours than men, a difference attributable to being less likely to report being highly interested in becoming partners while spending more time on child care.  What was noteworthy was the way they were able to mine the After the JD dataset. What seemed somewhat more troubling was the use of hours billed as a measure of performance, since completely controlling for selection in assignments appeared to me to be impossible given the IVs available.  Next, Dan Ho and Mark Kelman presented Does Class Size Reduce the Gender Gap? A Natural Experiment in Law. Ho and Kelman found that switching to small classes significantly increases the GPA of female law students (eliminating the gap between men and women). This is a powerful finding – obviously,it would be worth it to see if it is replicable at other schools.

The papers I regret having missed include How to Lie with Rape Statistics by Corey Yung (cities are lying with rape statistics); Employment Conditions and Judge Performance: Evidence from State Supreme Courts by Elliott Ash and W. Bentley MacLeod (judges respond to job incentives);  and Judging the Goring Ox: Retribution Directed Towards Animals by Geoffrey Goodwin and Adam Benforado.  I also feel terrible having missed Bill James, who I hear was inspirational, in his own way.

Overall, it was a tightly organized conference – kudos to Dave Abrams, Ted Ruger, and Tess Wilkinson-Ryan.  There could’ve been more law & psych, but that seems to be an evergreen complaint. Basically, it was a great two days.  I just wish there were more Twiqbal papers.

 

 

6

CELS VIII: Data is Revealing, Part 1.

 

"If you are going to mine my data, at least have the courtesy of displaying predictive probabilities!"

“If you are going to mine my data, at least have the courtesy of displaying predictive probabilities!”

[This is part 1 of my recap of the Penn edition of CELS, promised here.  For previous installments in the CELS recap series, see CELS III, IV, V, and VI, VII.]

Barry Schwartz might’ve designed the choice set facing me at the opening of CELS. Should I go to Civil Procedure I (highlighted by a Dan Klerman paper discussing the limits of Priest-Klein selection), Contracts I (where Yuval Feldman et al. would present on the relationship between contract clause specificity and compliance), on Judicial Decisionmaking and Settlement (another amazing Kuo-Chang Huang paper). [I am aware, incidentally, that for some people this choice would be Morton's. But those people probably weren't the audience for this post, were they.] I bit the bullet and went to Civ Pro, on the theory that it’d be a highly contentious slugfest between heavyweights in the field, throwing around words like “naive” and “embarrassing.”  Or, actually, I went hoping to learn something from Klerman, which I did. The slugfest happened after he finished.

In response to a new FJC paper on pleading practices, a discussant and a subsequent presenter criticized the FJC’s work on Twiqbal. The discussant argued that the FJC’s focus on the realities of lawyers’ practice was irrelevant to the Court’s power-grab in Twombly, and that pleading standards mattered infinitely more than pleading practice.  The presenter argued that the FJC committed methodological error in their important 2011 survey, and that their result (little effect) was misleading. The ensuing commentary was not restrained. Indeed, it felt a great deal like the infamous CELS death penalty debate from 2008. One constructive thing did come out of the fire-fight: the FJC’s estimable Joe Cecil announced that he would be making the FJC’s Twombly dataset available to all researchers through Vandy’s Branstetter program. We’ll all then be able to replicate the work done, and compare it to competing coding enterprises. Way to go, Joe!

But still, it was a tense session.  As it was wrapping up, an economically-trained empiricist in the room commented how fun he had found it & how he hoped to see more papers on the topic of Twombly in the future. I’d been silent to that point, but it was time to say something.  Last year in this space I tried being nice: “My own view would go further: is Twiqbal’s effect as important a problem as the distribution of CELS papers would imply?” This year I was, perhaps impolitically, more direct.

I conceded that analyzing the effect of Twombly/Iqbal wasn’t a trivial problem. But if you had to make a list of the top five most important issues in civil procedure that data can shed light on, it wouldn’t rank.* I’m not sure it would crack the top ten.  Why then have Twiqbal papers eaten market share at CELS and elsewhere since 2011? Some hypotheses (testable!) include: (1) civil procedure’s federal court bias; (2) giant-killing causes publication, and the colossi generally write normative articles praising transsubstantive procedure and consequently hate Twombly; (3) network effects; and (4) it’s where the data are. But these are bad reasons. Everyone knows that there is too much work on Twombly. We should stop spending so much energy on this question. It is quickly becoming a dead end.

So I said much of that and got several responses. One person seemed to suggest that a good defense of Twiqbal fixation was that it provided a focal point to organize our research and thus build an empirical community. Another suggested that even if law professors were Twiqbal focused, the larger empirical community was not (yet) aware of the importance of pleadings, so more attention was beneficent. And the rest of folks seemed to give me the kind of dirty look you give the person who blocks your view at a concert. Sit down! Don’t you see the show is just getting started?

Anyway, after that bit of theatre, I was off to a panel on Disclosure. I commented (PPT deck) on Sah/Lowenstein, Nothing to Declare: Mandatory and Voluntary Disclosure leads advisors to avoid conflicts of interestThis was a very, very good paper, in the line of disclosure papers I’ve previously blogged here. The innovation was that advisors were permitted to walk away from conflicts instead of being assigned to them immutably. This one small change cured disclosure’s perverse effect. Rather than being morally licensed by disclosure to lie, cheat and steal, advisors free to avoid conflicts were chastened by disclosure just as plain-vanilla Brandeisian theory would’ve predicted.   In my comments, I encouraged Prof. Sah to think about what happened if advisors’ rewards in the COI were returned to a third party instead of to them personally, since I think that’s the more legally-relevant policy problem. Anyway, definitely worth your time to read the paper.

Then it was off to the reception. Now, as our regular readers know, the cocktail party/poster session is a source of no small amount of stress. On the one hand, it’s a concern for the organizers. Will the food be as good as the legendary CELS@Yale? The answer, surprisingly, was “close to it”, headlined by some grapes at a cheese board which were the size of small apples and tasted great.  Also, very little messy finger food, which is good because the room is full of the maladroit.  But generally, poster sessions are terribly scary for those socially awkward introverts in the crowd. Which is to say, the crowd. In any event, I couldn’t socialize because I had to circle the crowd for you. Thanks for the excuse!

How about those posters?  I’ll highlight two. The first was a product of Ryan Copus and Cait Unkovic of Bolt’s JSP program. They automated text processing of appellate opinions and find significant judge-level effects on whether the panel reverses the district court’s opinion, as well as strong effects for the decision to designate an opinion for publication in the first instance. That was neat. But what was neater was the set of judicial base cards, complete with bubble-gum and a judge-specific stat pack, that they handed out.  My pack included Andrew Kleinfeld, a 9th circuit judge who inspired me to go to law school.  The second was a poster on the state appellate courts by Thomas Cohen of the AO. The noteworthy findings were: (1) a very low appeal-to-merits rate; and (2) a higher reversal rates for plaintiff than defendant wins at trial. Overall, the only complaint I’d make about the posters was that they weren’t clearly organized in the room by topic area, which would have made it easier to know where to spend time.  Also, the average age of poster presenters was younger than the average age of presenters of papers, while the average quality appeared as high or higher. What hypotheses might we formulate to explain that distribution?

That was all for Day 1. I’ll write about Day 2, which included a contracts, international law, and legal education sessions,  in a second post.

 

*At some point, I’ll provide a top ten list.  I’m taking nominations.  If it has federal court in the title, you are going to have to convince me.

2

Stanford Law Review Online: Privacy and Big Data

Stanford Law Review

The Stanford Law Review Online has just published a Symposium of articles entitled Privacy and Big Data.

Although the solutions to many modern economic and societal challenges may be found in better understanding data, the dramatic increase in the amount and variety of data collection poses serious concerns about infringements on privacy. In our 2013 Symposium Issue, experts weigh in on these important questions at the intersection of big data and privacy.

Read the full articles, Privacy and Big Data at the Stanford Law Review Online.

 

2

The Dignity of the Minimum Wage?

[A brief note of apology: it's been a terrible blogging summer for me, though great on other fronts.  I promise I'll do better in the coming academic year. In particular, I'd like to get back to my dark fantasy/law blogging series. If you've nominations for interviewees, email me.]

WorkDetroitThis is one I’ve been meaning to write for a while.

One of the major lessons of the cultural cognition project is that empirical arguments are a terrible way to resolve value conflicts. On issues as diverse as the relationship between gun ownership and homicide rates, the child-welfare effects of gay parenting, global warming, and consent in rape cases, participants in empirically-infused politics behave as if they are spectators at sporting events. New information is polarized through identity-protective lenses; we highlight those facts that are congenial to our way of life and discounts those that are not; we are subject to naive realism.  It’s sort of dispiriting, really.  Data can inflame our culture wars.

One example of this phenomenon is the empirical debate over minimum wage laws. As is well known, there is an evergreen debate in economics journals about the policy consequences which flow from a wage floor.  Many (most) economists argue that the minimum wage retards growth and ironically hurts the very low-wage workers it is supposed to hurt. Others argue that the minimum wage has the opposite effect. What’s interesting about this debate -to me, anyway- is that it seems to bear such an orthogonal relationship to how the politics of the minimum wage play out, and the kinds of arguments that persuade partisans on one side or another. Or to put it differently, academic liberals in favor of the minimum wage have relied on regression analyses, but I don’t think they’ve persuaded many folks who weren’t otherwise disposed to agree with them. Academic critics of the minimum wage too have failed to move the needle on public opinion, which (generally) is supportive of a much higher level of minimum wage than is currently the law.

How to explain this puzzle?  My colleague Brishen Rogers has a terrific draft article out on ssrn, Justice at Work: Minimum Wage Laws and Social Equality. The paper urges a new kind of defense of minimum wages, which elides the empirical debate about minimum wages’ effect on labor markets altogether. From the abstract:

“Accepting for the sake of argument that minimum wage laws cause inefficiency and unemployment, this article nevertheless defends them. It draws upon philosophical arguments that a just state will not simply redistribute resources, but will also enable citizens to relate to one another as equals. Minimum wage laws advance this ideal of “social equality” in two ways: they symbolize the society’s commitment to low-wage workers, and they help reduce work-based class and status distinctions. Comparable tax-and-transfer programs are less effective on both fronts. Indeed, the fact that minimum wage laws increase unemployment can be a good thing, as the jobs lost will not always be worth saving. The article thus stands to enrich current increasingly urgent debates over whether to increase the minimum wage. It also recasts some longstanding questions of minimum wage doctrine, including exclusions from coverage and ambiguities regarding which parties are liable for violations.”

I’m a huge fan of Brishen’s work, having been provoked and a bit convinced by his earlier work (here) on a productive way forward for the union movement. What seems valuable in this latest paper is that the minimum wage laws are explicitly defended with reference to a widely shared set of values (dignity, equality). Foregrounding such values I think would increase support for the minimum wage among members of the populace.  The lack of such dignitary discussions in the academic debate to date has level the minimum wage’s liberal defenders without a satisfying and coherent ground on which to stand. Worth thinking about in the waning hours of Labor’s day.

 

 

0

Brian Tamanaha’s Straw Men (Part 2): Who’s Cherry Picking?

(Reposted from Brian Leiter’s Law School Reports)

BT Claim 2:  Using more years of data would reduce the earnings premium

BT Quote: There is no doubt that including 1992 to 1995 in their study would measurabley reduce the ‘earnings premium.’” 

Response:  Using more years of historical data is as likely to increase the earnings premium as to reduce it

We have doubts about the effect of more data, even if Professor Tamanaha does not.

Without seeing data that would enable us to calculate earnings premiums, we can’t know for sure if introducing more years of comparable data would increase our estimates of the earnings premium or reduce it.

The issue is not simply the state of the legal market or entry level legal hiring—we must also consider how our control group of bachelor’s degree holders (who appear to be similar to the law degree holders but for the law degree) were doing.   To measure the value of a law degree, we must measure earnings premiums, not absolute earnings levels.

As a commenter on Tamanaha’s blog helpfully points out:

“I think you make far too much of the exclusion of the period from 1992-1995. Entry-level employment was similar to 1995-98 (as indicated by table 2 on page 9).

But this does not necessarily mean that the earnings premium was the same or lower. One cannot form conclusions about all JD holders based solely on entry-level employment numbers. As S&M’s data suggests, the earnings premium tends to be larger during recessions and their immediate aftermath and the U.S. economy only began an economic recovery in late 1992.

Lastly, even if you are right about the earnings premium from 1992-1995, what about 1987-91 when the legal economy appeared to be quite strong (as illustrated by the same chart referenced above)? Your suggestion to look at a twenty year period excludes this time frame even though it might offset the diminution in the earnings premium that would allegedly occur if S&M considered 1992-95.”

There is nothing magical about 1992.  If good quality data were available, why not go back to the 1980s or beyond?   Stephen Diamond and others make this point.

The 1980s are generally believed to be a boom time in the legal market.  Assuming for the sake of the argument that law degree earnings premiums are pro-cyclical (we are not sure if they are), inclusion of more historical data going back past 1992 is just as likely to increase our earnings premium as to reduce it.  Older data might suggest an upward trend in education earnings premiums, which could mean that our assumption of flat earnigns premiums may be too conservative. Leaving aside the data quality and continuity issues we discussed before (which led us to pick 1996 as our start year), there is no objective reason to stop in the early 1990s instead of going back further to the 1980s.

Our sample from 1996 to 2011 includes both good times and bad for law graduates and for the overall economy, and in every part of the cycle, law graduates appear to earn substantially more than similar individuals with only bachelor’s degrees.

 

Cycles

 

This might be as good a place as any to affirm that we certainly did not pick 1996 for any nefarious purpose.  Having worked with the SIPP before and being aware of the change in design, we chose 1996 purely because of the benefits we described here.  Once again, should Professor Tamanaha or any other group wish to use the publicly available SIPP data to extend the series farther back, we’ll be interested to see the results.

0

Brian Tamanaha’s Straw Men (Part 1): Why we used SIPP data from 1996 to 2011

(Reposted from Brian Leiter’s Law School Reports)

 

BT Claim:  We could have used more historical data without introducing continuity and other methodological problems

BT quote:  “Although SIPP was redesigned in 1996, there are surveys for 1993 and 1992, which allow continuity . . .”

Response:  Using more historical data from SIPP would likely have introduced continuity and other methodological problems

SIPP does indeed go back farther than 1996.  We chose that date because it was the beginning of an updated and revitalized SIPP that continues to this day.  SIPP was substantially redesigned in 1996 to increase sample size and improve data quality.  Combining different versions of SIPP could have introduced methodological problems.  That doesn’t mean one could not do it in the future, but it might raise as many questions as it would answer.

Had we used earlier data, it could be difficult to know to what extent changes to our earnings premiums estimates were caused by changes in the real world, and to what extent they were artifacts caused by changes to the SIPP methodology.

Because SIPP has developed and improved over time, the more recent data is more reliable than older historical data.  All else being equal, a larger sample size and more years of data are preferable.  However, data quality issues suggest focusing on more recent data.

If older data were included, it probably would have been appropriate to weight more recent and higher quality data more heavily than older and lower quality data.  We would likely also have had to make adjustments for differences that might have been caused by changes in survey methodology.  Such adjustments would inevitably have been controversial.

Because the sample size increased dramatically after 1996, including a few years of pre 1996 data would not provide as much new data or have the potential to change our estimates by nearly as much as Professor Tamanaha believes.  There are also gaps in SIPP data from the 1980s because of insufficient funding.

These issues and the 1996 changes are explained at length in the Survey of Income and Program Participation User’s Guide.

Changes to the new 1996 version of SIPP include:

Roughly doubling the sample size

This improves the precision of estimates and shrinks standard errors

Lengthening the panels from 3 years to 4 years

This reduces the severity of the regression to the median problem

Introducing computer assisted interviewing to improve data collection and reduce errors or the need to impute for missing data

Introducing oversampling of low income neighborhoods
This mitigates response bias issues we previously discussed, which are most likely to affect the bottom of the distribution
New income topcoding procedures were instituted with the 1996 Panel
This will affect both means and various points in the distribution
Topcoding is done on a monthly or quarterly basis, and can therefore undercount end of year bonuses, even for those who are not extremely high income year-round

Most government surveys topcode income data—that is, there is a maximum income that they will report.  This is done to protect the privacy of high-income individuals who could more easily be identified from ostensibly confidential survey data if their incomes were revealed.

Because law graduates tend to have higher incomes than bachelor’s, topcoding introduces downward bias to earnings premiums estimates. Midstream changes to topcoding procedures can change this bias and create problems with respect to consistency and continuity.

Without going into more detail, the topcoding procedure that began in 1996 appears to be an improvement over the earlier topcoding procedure.

These are only a subset of the problems extending the SIPP data back past 1996 would have introduced.  For us, the costs of backfilling data appear to outweigh the benefits.  If other parties wish to pursue that course, we’ll be interested in what they find, just as we hope others were interested in our findings.

0

Brian Tamanaha’s Straw Men (Overview)

(Cross posted from Brian Leiter’s Law School Reports)

Brian Tamanaha previously told Inside Higher Education that our research only looked at average earnings premiums and did not consider the low end of the distribution.  Dylan Matthews at the Washington Post reported that Professor Tamanaha’s description of our research was “false”. 

In his latest post, Professor Tamanaha combines interesting critiques with some not very interesting errors and claims that are not supported by data.   Responding to his blog post is a little tricky as his ongoing edits rendered it something of a moving target.  While we’re happy with improvements, a PDF of the version to which we are responding is available here just so we all know what page we’re on.

Stephen Diamond explains why Tamanaha apparently changed his post: Ted Seto and Eric Rasmusen expressed concerns about Tamanaha’s use of ad hominem attacks.

Some of Tamanaha’s new errors are surprising, because they come after an email exchange with him in which we addressed them.  For example, Tamanaha’s description of our approach to ability sorting constitutes a gross misreading of our research.  Tamanaha also references the wrong chart for earnings premium trends and misinterprets confidence intervals.  And his description of our present value calculations is way off the mark.

Here are some quick bullet point responses, with details below in subsequent posts:

  • Forecasting and Backfilling
    • Using more historical data from SIPP would likely have introduced continuity and other methodological problems
    • Using more years of data is as likely to increase the historical earnings premium as to reduce it
    • If pre-1996 historical data finds lower earnings premiums, that may suggest a long term upward trend and could mean that our estimates of flat future earnings premiums are too conservative and the premium estimates should be higher
    • The earnings premium in the future is just as likely to be higher as it is to be lower than it was in 1996-2011
    • In the future, the earnings premium would have to be lower by **85 percent** for an investment in law school to destroy economic value at the median
  • Data sufficiency
    • 16 years of data is more than is used in similar studies to establish a baseline.  This includes studies Tamanaha cited and praised in his book.
    • Our data includes both peaks and troughs in the cycle.  Across the cycle, law graduates earn substantially more than bachelor’s.
  • Tamanaha’s errors and misreading
    • We control for ability sorting and selection using extensive controls for socio-economic, academic, and demographic characteristics
    • This substantially reduces our earnings premium estimates
    • Any lingering ability sorting and selection is likely offset by response bias in SIPP, topcoding, and other problems that cut in the opposite direction
    • Tamanaha references the wrong chart for earnings premium trends and misinterprets confidence intervals
    • Tamanaha is confused about present value, opportunity cost, and discounting
    • Our in-school earnings are based on data, but, in any event, “correcting” to zero would not meaningfully change our conclusions
  • Tamanaha’s best line
    • “Let me also confirm that [Simkovic & McIntyre’s] study is far more sophisticated than my admittedly crude efforts.”