Experiments in Lawyering: Does the Harvard Legal Aid Bureau Deserve a Merit Badge?
posted by Dave Hoffman
Mike Heise highlights a paper by James Greiner (Harvard) and Cassandra Wolos Pattanayak: What Difference Representation? From the abstract:
We report the results of the first in a series of randomized control trials designed to measure the effect of an offer of, and the actual use of, legal representation. The results are startling. In the context of administrative litigation to determine eligibility for unemployment benefits, a service provider’s offer of representation to a claimant had no statistically significant effect on the claimant’s probability of a victory, but the offer caused a delay in the proceeding. Because a substantial percentage of the provider’s client base consisted of claimants who were initially denied benefits but who would have that initial denial reversed as a result of the litigation, the delay an offer of representation caused inflicted a harm upon such claimants in the form of an additional waiting time for benefits to begin, this with no concomitant increase in the probability of a favorable outcome. In other words, these claimants would have been better off without the offer of representation. Other classes of claimants were unaffected, but in cases with a certain profile, the delay hurt the financing of the unemployment system, again with no concomitant benefit in the probability of a favorable outcome for the claimant. We were also able to verify a delay effect due to the actual use of (as opposed to an offer of) representation; we could come to no firm conclusion on the effect of actual use of representation on win/loss. Stepping back, we use these results as a springboard for a comprehensive review of the quantitative literature on the effect of representation in civil proceedings. We find that this literature provides virtually no credible information, excepting the results of two randomized evaluations occurring in different legal contexts and separated by over three decades. We conclude by advocating for, and describing challenges associated with, a large program of randomized evaluation of the provision of representation, particularly by legal services providers.
Greiner/Pattanayak take (very) broad swipes at a variety of previous studies of representation. Putting that aside, the results from the paper are unsettling, at least if your prior is that legal representation always helps the poor. And I wanted to pull out one part of the article that is particularly interesting. From pages 6-7:
[Our] ideas polarized the legal services community. Some organizations overcame initial nervousness about defining measurable outcomes and about ceding partial control over case selection to a randomizer and embraced the effort. In doing so, these providers demonstrated the courage necessary to subject their programs to gold-standard evaluation. Among the most courageous were the students of the Harvard Legal Aid Bureau (“HLAB”), a student-run, faculty-overseen legal services office that is part of the clinical educational program at Harvard Law School. But other organizations opposed our effort. One group did not limit its opposition to a refusal to participate on its own part. Instead, when it discovered that HLAB was conducting a randomized evaluation, it halted its previous practice of suggesting that clients it could not itself represent call HLAB.
December 21, 2010 at 5:41 pm
Posted in: Articles and Books, Civil Rights, Empirical Analysis of Law, Law School (Teaching)
Print This Post
Responses (6)
A.J. Sutter - December 21, 2010 at 9:05 pm
Your last sentence is a non sequitur, either in the absolute sense or at least with regard to the information presented in your post.
First, where is the “obstruction”? Suppose that, given what they knew at the time, other programs sincerely regarded the randomization experiment as unethical or potentially harmful to their clients. In those circumstances and under that “prior,” wouldn’t it be responsible of them to decline to participate? As for the last group, previously it had advised clients to call HLAB — but those clients might not have called HLAB anyway, or might have found HLAB even without the advice. How did its removal of such advice prevent HLAB from conducting its experiment? No matter what the reason, if I decline to participate in, or otherwise to facilitate, an experiment, am I “obstructing” it? This sounds like a notion from a bad horror or sci-fi movie, where the crazed “genius” complains that those who don’t accede to his designs are “standing in the way of science.”
Second, what’s the connection between public funds and compliance with a Harvard prof’s experiment? Or have I missed something while spending the Obama Administration abroad — a Cass Sunstein directive, e.g.?
A.J. Sutter - December 21, 2010 at 10:40 pm
Thanks for the kind shout-out and the reply, Dave. But the point of my Bayesian reference to a prior was that at the time the groups behaved in the way they did, the findings of the research weren’t yet known. So the groups might have reasonably and sincerely concluded that, far from acting in a “self-serving, self-protective” way, they were acting in the best interests of their clients by not depriving them of representation (or exposing them to the perceived risk of non-representation). From the description of the methodology in Greiner/Pattanayak (@18-20), it seems that the researchers were careful to obtain consents from participating claimants; but some of the clinics might reasonably have worried that claimants, who were already stressed by being on unemployment anyway, would not understand the potential impact of their declining to be represented. While some might consider such scruples paternalistic, that doesn’t make the non-participating clinics’ motivation necessarily self-serving or self-protective.
I’m agnostic about whether the Greiner/Pattanayak research is conclusive enough to change that “prior” from this point forward, but projecting it backwards seems like an unfair use of hindsight.
James M - December 22, 2010 at 4:41 pm
I think the study shows something else – that to be effective you need representation by someone who knows and ins familiar with both the relevant law and the administrative process used to determine the outcome.
I see lots of lawyers jumping in to help with the tide of foreclosures, but without a specialization and prior experience in foreclosure defenses they are infective. They frequently make the sort of mistakes that pro-se defendants make, and frequently don’t use the legal defenses and strategies that an experenced foreclosure defense firm would do.
I have not seen the raw data from the study, but I suspect a better comparison would be the outcomes of cases where the person was represented by good quality counsel with a track record, compared with pro-se.
It goes to show the need for effective knowledge, either through personal knowlage and experience or systematized into detailed checklists.
Airlines don’t crash very often, becuase the pilots have been well trained AND becuase everyone involved is working from pre-flight checklists which makes sure everything is done. Lawyers like to think they know it all and frequently don’t have speclised training on that specific legal process and don’t implement or use checklists.
Some do know it all, most don’t.
They don’t even know what they don’t know, don’t even know why they fail.
If some lawyers do have better than average track records in this sort of proceeding, the relevant question is WHY are the lawyers involved getting it wrong?
Jim Greiner - December 29, 2010 at 11:18 am
Hi, Dave, thanks very much for the shout-out re the paper. Two quick thoughts. First, re the question of claimants randomized to no offer of law student representation (and whether these claimants got representation from another source), it might be worth a review of pages 37-41 of the currently posted draft, where tackle that issue. No doubt many will be disappointed by the fact that the data would not support more definitive results (we’re disappointed ourselves), but we report what we see.
Second, I think the Harvard students’ decision to participate in the research really was quite courageous. We have approached other legal services providers to suggest randomized evaluations; the responses have varied, but it’s not much of an exaggeration to say that the modal response has been the finger. Without going into too much detail re Harvard Law structure, the students involved here run their own law office, and they were under no obligation whatsoever to participate in the research (they initially turned us down). As Ian Ayres suggests over at the Freakonomics blog, the only way to expand the use of randomized trials (and to benefit from the knowledge they provide) is to support those entities brave enough to engage in them.
Carolyn Elefant - January 6, 2011 at 9:18 pm
I am not an academic and I have to confess none of this made any sense to me. The paper evaluated whether an offer of representation, not actual representation made a difference. So how were the Harvard students courageous? The test didn’t evaluate work they actually did.
I also think that selecting unemployment compensation hearings in the midst of an economic downturn as a basis for evaluating effectiveness of representation was probably not a very smart choice. My guess is that ALJs are extremely sympathetic to unemployment claims in a recession and a lawyer probably would not make much of a difference. Why didn’t you examine the effectiveness of representation in foreclosure proceedings? Or debt collection cases? To me, that would have made much more sense.
Frank - February 2, 2011 at 10:21 am
Carolyn, the authors took pains to point out that they didn’t have many options–they need the clinic’s cooperation after all.
I agree that outcomes in this sort of proceeding seems less likely to benefit from lawyers. Heck, if I were an ALJ, I would have a hard time ignoring the fact that most 3L Harvard students before me are likely to draw double my salary in on year’s time.
But if it’s obvious to you that clinical students make no difference in this sort of proceeding, and since the study tends to confirm that view, isn’t the real question: “Why does Harvard commit so much time and money to a clinic that doesn’t actually help the poor?”
Leave a Reply