From: “David Grossman” <firstname.lastname@example.org>
Subject: [Lawclinic] study of case outcomes
In light of recent reports on the unpublished paper “What Difference Representation?” by an Assistant (non-clinical) Professor Jim Greiner here at Harvard Law School and statistics graduate student Cassandra Wolos Pattanayak, and subsequent concern about the breadth and impact of the study’s conclusions, we thought we should respond with some thoughts about why we entered into the study in the first place, and what we think the study does and does not tell us about the provision of free legal services.
The study, which began in 2008, randomized the eligible potential clients who would receive an offer to be represented by the Harvard Legal Aid Bureau (HLAB) at an administrative hearing adjudicating whether the client was entitled to receive unemployment insurance benefits. Professor Greiner found that on average 76% of claimants who received an offer of HLAB representation prevailed, while 72% of those who did not receive an offer prevailed. Professor Greiner also found that on average, cases receiving an HLAB offer delayed a decision by the unemployment insurance system by one to two weeks.
HLAB, a student-run legal services organization that elects a new student Board of Directors each year, has a history of experimentation, change, and self-evaluation. Through student research projects and in classroom and office policy discussions, we are constantly examining the effectiveness of the work we are doing and exploring opportunities to shift our resources to better serve our client communities. This study gave us the opportunity to work with a professional statistician to examine a question we had previously studied ourselves: whether the unemployment claims hearing system in Boston was set up to effectively serve claimants. We had some reason to believe that claimants in the system did relatively well. A retrospective study conducted by one of our clinical students three years ago found that pro se claimants won approximately 50 percent of their cases. In entering into Professor Greiner’s study, we had hoped to determine what subset of clients we could help most, and then to redirect our resources to those populations or, if it turned out the unemployment hearings system was effectively serving pro se claimants, to shift resources to our other practice areas, which include housing law, family law, wage and hour, and other government benefits.
Unfortunately, Professor Greiner’s study gives us only limited information. Although we had hoped that the study could tell us about the effect on case outcomes of legal representation this too had been Professor Greiner’s goal as the study developed, it became clear that the study could not meet this goal. First, a number of the study participants who were offered representation as a result of the randomization never accepted or followed up on that offer. Second, nearly half of those who did not receive an offer of representation from HLAB ended up receiving representation from other local legal services providers. The study thus ended up comparing outcomes for an “offer” group in which HLAB represented 90% of the claimants and the other 10% were pro se, and a “no offer” group in which 49% of the claimants were represented by other providers and 51% were pro se. Because the two groups are so heterogeneous, and because we have not been able to parse out the outcomes of the two subgroups within the control group, the offer/no offer distinction does not appear to be a sufficient proxy for representation/no representation. (Other data appear to show that, outside the study, pro se claimants win 50% of their cases; we thus far have been unable to determine the win rate for pro se claimants within the study or to compare that to HLAB’s 76% win rate.)
Although the study cannot tell us whether representation makes a difference in unemployment insurance hearings in Boston, it does tell us whether an offer of representation at least from HLAB does, but that is as compared to the study’s heterogeneous control group. At present, it is unclear how helpful that information is for decision-making, either internally at HLAB or more broadly in the legal services community. To determine how most efficiently to allocate resources, we and other providers would want to know, at a minimum, whether any offer of free representation regardless of the identity of the provider making it results in better outcomes. In order to do that, however, one would need to reanalyze Professor Greiner’s data, considering the cases in which representation was provided by other legal services providers as part of the “offer” group, not the “no offer” group. Thus far, we have been unable to reanalyze the data in this way.
Finally, because the sample size is small, Professor Greiner has been hesitant to make suggestions about what subsets of clients (for example, limited English proficient clients) would be best served by HLAB representation.
Despite these limitations, we take the study as an occasion to revisit how HLAB relates to this field. We are closely analyzing Professor Greiner’s data, and making decisions internally about how best to move forward. In light of the uncertainties of the study, we are keeping some resources in our unemployment hearings practice, while shifting resources elsewhere. We are also doing our best to figure out from his data which subsets of clients are most helped by HLAB representation, and targeting those individuals by using triggers in our intake process. At this point we are wary of eliminating our practice entirely because that would foreclose helping make systemic change in the unemployment hearing system. For example, we appealed the case of one of the clients in the study who initially lost his hearing. Because of the success of the appeal, the state agency that administers the unemployment insurance system is now making reforms to its regulations to the benefit of an entire class of claimants. In addition, the constant presence of legal services providers, including HLAB, at hearings may have an effect on outcomes at hearings broadly a point Professor Greiner mentions in his study. Finally, we are interested in understanding further how the unique, inquisitorial nature of the unemployment insurance hearings has impacted Professor Greiner’s outcomes and thus to what extent conclusions from the study can be extended to the provision of legal services in the litigation context.
Because of the study’s complexity it is subject to significant interpretive flaws, and we are concerned that the study, if read cursorily or tendentiously, will have a negative impact on legal services providers broadly. We hope that this e-mail clarifies the study’s context and significance as we understand them. We also hope that the study does not deter any of us in the legal services community from continuing to examine our priorities and our methods to ensure that we are providing the best representation that we can. To that end, please feel free to reach out to us if you would like to discuss further the impact of the study, or our experience participating in it.
President, Harvard Legal Aid Bureau