Archive for the ‘Empirical Analysis of Law’ Category

Reversal Rates, Reconsidered

posted by Dave Hoffman

What is the meaning of an appellate court’s “reversal rate”?  Opinions vary.  (My view, expressed, succinctly, is “basically nothing.”) However conceived, we ought to at least be measuring reversal correctly.  But two lawyers at Hangley Aronchick, a Philadelphia law firm, think that scholars (and journalists) have conceptualized reversal in entirely the wrong way.

According to John Summers and Michael Newman, we’ve forgotten that every case the Supreme Court takes implicitly also considers shadow cases from other circuits ruling on the same issue — that is, the Supreme Court doesn’t just “reverse” the circuit on direct appeal, it also affirms (or reverses) coordinate circuits while resolving a split.  Thus, both our numerator and our denominator have been wrong.  They’ve written up the results of this pretty interesting approach to reversal in a paper you can find blurbed here.   Among the highlights: (1) reversal is less common that is commonly supposed; (2) the Court doesn’t predictably follow the majority of circuits; (3) there are patterns of concordance between circuits in analyzing issues; and (4) even under the new approach, the ninth circuit is still the least loyal agent of the Supreme Court.

I think that this method has real promise, and I bet that folks who are interested in judicial behavior will want to check it out.

  September 30, 2011 at 12:21 pm   Posted in: Economic Analysis of Law, Empirical Analysis of Law, Supreme Court   Print This Post   No Comments

In Praise of Complexity

posted by Corey Yung

Earlier this month, right here on this very blog, Dave Hoffman pontificated about two of my favorite subjects: empirical legal studies and baseball. Primarily, Dave wondered about whether empirical legal research was facing might face the same problem as sabermatic baseball analysis: inaccessible complexity. I won’t rehash his argument because he did a very good job of explaining it in the original post. Although I completely agree with his conclusion that empirical legal studies should seek to be more accessible (which I always note at the end of my introduction of my empirical work), I disagree with his contention that empirical legal studies are facing might face widespread incomprehensibility due to growing complexity. Because I think it is a helpful analogy, I’ll borrow Dave’s example of advanced statistics in baseball. Read the rest of this post »

  September 28, 2011 at 11:32 pm   Posted in: Courts, Empirical Analysis of Law   Print This Post   3 Comments

Law’s Arbitrary Endpoints

posted by Dave Hoffman

For many purposes, a season is an arbitrary endpoint to measure a baseball player’s success.  To extract utility from performance data over time, you need to pick endpoints that make sense in light of what you are measuring.  Thus, if we want to know how much to discount a batter’s achievements by luck, it might not make sense to look seasonally – - because there’s no good reason to expect that luck is packaged in April-to-October chunks.  Nonetheless, sabermetricians commonly do talk about BABIP seasonally — thus, Aaron Rowand had an “unusually lucky 2007,” and has since regressed himself off of a major league payroll.  Jayson Werth, similarly, is feeling the bite of lady luck this “season.”  (For pitchers, the analysis makes more sense, since the point of BABIP is that pitchers can’t control outcomes once the ball hits a bat.  Thus, the Phillies fifth starter is supposedly not nearly as good as his haircut suggests he ought to be.)

This bias toward artificial endpoints affects legal studies, though less obviously.  There aren’t legal seasons.  (It’s always a time to weep, to bill, to work, to reap.)  But we still organize our analyses around units which might not exactly track the underlying item of interest.  We want to study disputes, but we look at the records of filing and verdicts (which are a smaller unit in time than the object of study).  We wish to examine ideological voting patterns on the Court, but we organize our study by Term.  We want clear signals of young lawyer quality, but we look at grades in law school, for (mostly) the first three semesters). We want to know how law schools’ influence hiring practices, but we look at deadline-generated 9-month hiring reports.  Different slices at these numbers may produce quite different results — heck, one of the reasons that USNews obtains variable rankings is that they keep on moving the endpoints of the analysis in ways that are perfectly unclear.

There’s no complete solution to the endpoint problem – at least, not one that’s easily compatible with the project of data-driven legal analysis.  It’s important, therefore, to be especially careful when reading studies that take advantage of convenient legal periods.  A prime example is the Supreme Court’s “Term.”  I have no good reason to expect that the Justices’ behavior changes meaningfully from one Term to another — absent an intervening change in personnel.  So, Term analysis is convenient, but I bet it misleads.  Comparing the performance of a Circuit from one Term to another is similarly odd — whatever the value of the “reversal rate” inquiry, it surely doesn’t turn on Terms!

This set of cautions might be extended to a more general one, directed at folks who are interested in doing empirical work but haven’t yet begun to collect data. If your outcome of interest is measured monthly, seasonally or yearly, consider whether that unit of measurement reflects something true about the data, or is merely a convenience.  If it’s the latter, proceed with caution.  Obviously, this isn’t at all a novel caution, but the persistence of the error suggests it can’t be made often enough.

  September 7, 2011 at 11:40 am   Posted in: Empirical Analysis of Law   Print This Post   One Comment

Assessing Medicaid Managed Care

posted by Frank Pasquale

The Washington Post has featured two interesting pieces recently on Medicaid managed care. Christopher Weaver reported on a battle between providers and insurers in Texas. Noting that “federal health law calls for a huge expansion of the Medicaid program in 2014,” Weaver shows how eager insurers are to enroll poor individuals in their plans. Each enrollee would “yield on average $7 a month profit,” according to recent calculations. Cost-cutting legislators see potential fiscal gains, too, once the market starts working its magic.

There’s only one problem with those projections: it turns out that “moving Medicaid recipients into managed care ‘did not lead to lower Medicaid spending during the 1991 to 2003 period,’” according to a report published by the National Bureau of Economic Research this month. Sarah Kliff is surprised to find that this is “the first national look at whether Medicaid managed care has actually done a key thing that states want it to do.”
Read the rest of this post »

  August 29, 2011 at 3:31 pm   Posted in: Empirical Analysis of Law, Health Law, Law and Inequality   Print This Post   No Comments

The Future of Empirical Legal Studies

posted by Dave Hoffman

Reading these two articles on the problems of complexity for sabermetrics, I wondered if the empirical legal studies community is coming soon to a similar point of crisis. The basic concern is that sabermetricians are devoting oodles of time to ever-more-complex formulae which add only a small amount of predictive power, but which make the discipline more remote from lay understanding, and thus less practically useful.   Basically: the jargonification of a field.  Substituting “law” into Graham MacAree‘s article on the failings of sabermetrics, we get the following dire warning:

“Proper [empirical legal analysis] is something that has to come from the top down ([law]-driven) rather than the bottom up (mathematics/data driven), and to lose sight of that causes a whole host of issues that are plaguing the field at present. Every single formula must be explainable without recourse to using ridiculous numbers. Every analystmust be open to thinking about the [law] in new ways. Every number, every graph in a [ELS] piece musttell a [legal] story*, because otherwise we’re no longer writing about the [legal system ] but indulging in blind number-crunching for its own sake. …

Surveying the field, I no longer believe that those essential precepts hold sway over the [ELS] community. Data analysis methods are being misapplied and sold to readers as the next big thing. Articles are being written for the sake of sharing irrelevant changes in irrelevant metrics. Certain personalities are so revered that their word is taken as gospel when fighting dogma was what brought them the respect they’re now given in the first place. [ELS] is in a sorry state.

How do we fix it? Well, the answer seems simple. [ELS] shouldn’t be so incomprehensible so as not to call up the smell of [a courtroom, or the careful drafting of the definition clauses in a contract, or the delicate tradeoffs involved in family court practice, or the importance of situation sense]. Statistics shouldn’t be sterile and clean and shiny and soulless. They shouldn’t just be about [Law]; they should invoke it. Otherwise, they run the risk of losing the language which makes them so special.”

Note: this is an entirely different  than Leiter’s 2010 odd critique that ELS work was largely mediocre.  The problem, rather, is that the trend is toward a focus on more complex and “accurate” models, often without the input of people with legal training, and insufficient attention to how such models will be explained to lawyers, judges and legal policymakers.  (See also all of Lee Epstein’s work.)

  August 28, 2011 at 5:41 pm   Posted in: Economic Analysis of Law, Empirical Analysis of Law   Print This Post   3 Comments

Assessing Twiqbal

posted by Dave Hoffman

Several months ago, the FJC put out a well-publicized study assessing the results of Twombly-Iqbal on motions practice.  It concluded that there was little reason, overall, for concern that the Supreme Court’s new pleadings jurisprudence had worked a revolutionary change down below.  Lonny Hoffman (Houston) has just released an important new paper which questions the methods and conclusions of the FJC’s work.  He pulls no punches:

“This paper provides the first comprehensive assessment of the Federal Judicial Center’s long-anticipated study of motions to dismiss for failure to state a claim after Iqbal v. Ashcroft. Three primary assessments are made of the FJC’s study. First, there are reasons to be concerned that the study may be providing an incomplete picture of actual Rule 12(b)(6) activity. Even if the failure to capture all relevant motion activity was a non-biased error, the inclusiveness problem is consequential. Because the study was designed to compare over time the filing and grant rate of Rule 12(b)(6) motions, the size of the effect of the Court’s cases turns on the amount of activity found. Second, even if concerns are set aside that the collected data may be incomplete, it misreads of the FJC’s findings to conclude that the Court’s decisions are having no effect on dismissal practice. The FJC found that after Iqbal, a plaintiff is twice as likely to face a motion to dismiss. This sizeable increase in rate of Rule 12(b)(6) motion activity represents a marked departure from the steady filing rate observed over the last several decades and means, among other consequences, added costs for plaintiffs who have to defend more frequently against these motions. The data regarding orders resolving dismissal motions even more dramatically shows the consequential impacts of the Court’s cases. There were more orders granting dismissal with and without leave to amend, and for every case category examined. Moreover, the data show that after Iqbal it was much more likely that a motion to dismiss would be granted with leave to amend (as compared to being denied) both overall and in the three largest case categories examined (Civil Rights, Financial Instruments and Other). Employment Discrimination, Contract and Torts all show a trend of increasing grant rates. In sum, in every case type studied there was a higher likelihood after Iqbal that a motion to dismiss would be granted. Third, because of inherent limitations in doing empirical work of this nature, the cases may be having effects that the FJC researchers were unable to detect. Comparing how many motions were filed and granted pre-Twombly to post-Iqbal cannot tell us whether the Court’s cases are deterring some claims from being brought, whether they have increased dismissals of complaints on factual sufficiency grounds, or how many meritorious cases have been dismissed as a result of the Court’s stricter pleading filter. Ultimately, perhaps the most important lesson to take away from this last assessment of the FJC’s report is that empirical study cannot resolve all of the policy questions that Twombly and Iqbal raise.”

I should disclose that I provided Lonny comments on an earlier draft, and overall I think he’s done an incredible (and generally very fair) job.  One thing to think about, as always when evaluating litigation data, is the degree to which we would expect to see any results at all given case selection effects.  That Lonny does observe such substantively significant changes notwithstanding selection tells us something about how dramatic the Twiqbal decisions really were.

  August 5, 2011 at 12:29 pm   Posted in: Civil Procedure, Empirical Analysis of Law   Print This Post   No Comments

Beneath the Lamp Post

posted by Dave Hoffman

Though many bemoan the expense and terrible functionality of PACER, the federal government’s electronic docketing system, it is vastly superior to existing state alternatives.  While some states have decent, and searchable, e-dockets, others do not, and it’s often quite hard to figure out the scope of the state databases.  The result is that a researcher (or a lawyer) who wants to study live dockets at the state level is faced with a host of known unknowns, making aggregate statistical inference basically impossible.  Even descriptive statistics about state courts are hard to verify.  It’s a black hole. (With some illumination provided by the BJS and other bodies.)

This frustrates me, and if I could wave a magic wand (or controlled Google) I would create a national e-docketing system for all state filings, permitting full-text searchers across states for comprehensive data – including searches of motions and orders – in both civil and criminal litigation.  The current state of the world, by contrast, directs much of the new empirical legal research to focus on federal cases and federal outcomes, because PACER provides access to the kinds of data that researchers need.  The problem, of course, is that PACER collects only Federal dockets, which aren’t representative of the kind or scope of litigation nationwide. Though of course studying dockets is vastly superior to studying opinions – if you want to know what judges are doing – we’re left still peering through a dark piece of glass.  Worse, I think, is that researchers end up focusing their energies on topics for which federal litigation is the dominant way of resolving legal claims.  Thus, there’s much more, and much better, docket-centered empirical work about securities law and federal civil rights statutes than there is about common law adjudication.

Our sadly patchwork court records system  doesn’t just hurt academics looking to illuminate doctrinal puzzles.  (The horror! Tenured professors can’t write more papers!)  It also means that lawyers and corporate officers may be forced to rely on anecdote and salience when deciding how to engage with the litigation system — a calculation that may lead such repeat players to develop a long-term strategy to exit the litigation system altogether.  If the state courts want to preserve their business, they need to innovate.  One way to do so would be to join forces in data collection, archival, and search.

(Image Source: Flicker)

  May 15, 2011 at 5:49 pm   Posted in: Courts, Economic Analysis of Law, Empirical Analysis of Law   Print This Post   11 Comments

Randomization Uber Alles?

posted by Dave Hoffman

Jim and Cassandra write:

“To Dave, we say that our enthusiasm for randomized studies is high, but perhaps not high enough to consider a duty to randomize among law school clinics or among legal services providers.  We provided an example in the paper of practice in which randomization was inappropriate because collecting outcomes might have exposed study subjects to deportation proceedings.  We also highlighted in the paper that in the case of a practice (including possibly a law school clinic) that focuses principally on systemic change, randomization of that practice is not constructive.  Instead, what should be done is a series of randomized studies of an alternative service provider’s practice in that same adjudicatory system; these alternative provider studies can help to assess whether the first provider’s efforts at systemic change have been successful.”

I meant to cabin my argument to law school clinics.  And I do understand that there may be very rare cases where collecting outcomes will hurt clients (such as deportation).  But what about a clinic that focuses on “systemic change.” Let’s assume that subsidizing such a clinic would be a good thing for a law school to do (or, put it another way, we think it is a good idea for current law students to incur more debt so that society gets the benefit of the clinics’ social agitation).  Obviously, randomization of client outcomes would be a terrible fit for measuring the success of such a clinic.  It would be precisely the kind of lamppost/data problem that Brian Leiter thinks characterizes much empirical work.

But that doesn’t mean that randomization couldn’t be useful in measuring other kinds of clinic outcomes.  What about randomization in the allocation of law student “employees” to the clinic as a way to measure student satisfaction in the “learning outcomes“? Or randomization of intake and utilizing different client contact techniques as a way of measuring client satisfaction with their representation (or feelings about the legitimacy of the system?)  One thing that the commentators in this symposium have tried to emphasize is that winning & losing aren’t the only outputs of the market for indigent legal services.  Controlled study of the actors in the system needn’t be constrained in the way that Jim and Cassandra’s reply to my modest proposal to mandate randomization suggest.

  March 29, 2011 at 1:23 pm   Posted in: Empirical Analysis of Law, Law Rev (Yale), Law School (Scholarship), Law School (Teaching), Symposium (What Difference Representation)   Print This Post   One Comment

Randomization, Intake Systems, and Triage

posted by Jaya Ramji-Nogales

Thanks to Jim and Cassandra for their carefully constructed study of the impact of an offer from the Harvard Legal Aid Bureau for representation before the Massachusetts Division of Unemployment Assistance, and to all of the participants in the symposium for their thoughtful contributions.  What Difference Representation? continues to provoke much thought, and as others have noted, will have a great impact on the access to justice debate.  I’d like to focus on the last question posed in the paper — where do we go from here? — and tie this in with questions about triage raised by Richard Zorza and questions about intake processes raised by Margaret Monsell.   The discussion below is informed by my experience as a legal service provider in the asylum system, a legal arena that the authors note is  strikingly different from the unemployment benefits appeals process described in the article.

My first point is that intake processes vary significantly between different service providers offering representation in similar and different areas of the law.  In my experience selecting cases for the asylum clinics at Georgetown and Yale, for example, we declined only cases that were frivolous, and at least some intake folks (yours truly included) preferred to select the more difficult cases, believing that high-quality student representation could make the most difference in these cases.  Surely other legal services providers select for the cases that are most likely to win, under different theories about the most effective use of resources.  WDR does not discuss which approach HLAB takes in normal practice (that is, outside the randomization study).  On page twenty, the study states that information on financial eligibility and “certain additional facts regarding the caller and the case”  are put to the vote of HLAB’s intake committee.  On what grounds does this committee vote to accept or reject a case?  In other words, does HLAB normally seek the hard cases, the more straightforward cases, some combination, or does it not take the merits into account at all?

Read the rest of this post »

  March 28, 2011 at 9:14 pm   Posted in: Behavioral Law and Economics, Civil Rights, Empirical Analysis of Law, Immigration, Law Practice, Law Rev (Yale), Symposium (What Difference Representation), Uncategorized   Print This Post   One Comment

How Much Enthusiasm for Randomized Trials? A Response to Kevin Quinn and David Hoffman

posted by Jim Greiner

We thank Kevin Quinn and David Hoffman for taking the time to comment in our paper.  Again, these are two authors whose work we have read and admired in the past.

Both Dave and Kevin offer  thoughts about the levelof enthusiasm legal empiricists, legal services providers, and clinicians should have for randomized studies.  We find ourselves in much but not total agreement with both.  To Kevin, we suggest that there is more at stake than just finding out whether legal assistance helps potential clients.  In an era of scarce legal resources, providers and funders have to make allocation decisions across legal practice areas (i.e., should we fund representation for SSI/SSDI appeals or for unemployment appeals or for summary eviction defense).  That requires more precise knowledge about how large representation (offer or actual use) effects are, how much bang for the buck.  Perhaps even more importantly, scarcity requires that we learn how to triage well; see Richard Zorza’s posts here and the numerous entries in his own blog on this subject.  That means studying the effects of limited interventions.  Randomized trials provide critical information on these questions, even if one agrees (as we do) that in some settings, asking whether representation (offer or actual use) helps clients is like asking whether parachutes are useful.

Thus, perhaps the parachute analogy is inapt, or better, it requires clarification:  we are in a world in which not all who could benefit from full-service parachutes can receive them.  Some will have to be provided with rickety parachutes, and some with little more than large blankets.  We all should try to change this situation as much as possible (thus the fervent hope we expressed in the paper that funding for legal services be increased).  But the oversubscription problem is simply enormous.  When there isn’t enough to go around, we need to know what we need to know to allocate well.  Meanwhile, randomized studies can also provide critical information on the pro se accessibility of an adjudicatory system, which can lay the groundwork for reform.

To Dave, we say that our enthusiasm for randomized studies is high, but perhaps not high enough to consider a duty to randomize among law school clinics or among legal services providers.  We provided an example in the paper of practice in which randomization was inappropriate because collecting outcomes might have exposed study subjects to deportation proceedings.  We also highlighted in the paper that in the case of a practice (including possibly a law school clinic) that focuses principally on systemic change, randomization of that practice is not constructive.  Instead, what should be done is a series of randomized studies of an alternative service provider’s practice in that same adjudicatory system; these alternative provider studies can help to assess whether the first provider’s efforts at systemic change have been successful.

Our great thanks to both Kevin and Dave for writing, and (obviously) to Dave (and Jaya) for organizing this symposium.

  March 28, 2011 at 9:11 pm   Posted in: Empirical Analysis of Law, Symposium (What Difference Representation)   Print This Post   No Comments

What Difference Representation: Case Selection and Professional Responsibility

posted by Margaret Monsell

Thanks for the invitation to participate in this interesting and provocative symposium.

I’m a legal services attorney in Boston. My employer, Massachusetts Law Reform Institute (MLRI), has as one of its primary tasks to connect the state’s field programs, where individual client representation occurs, with larger political bodies, including legislatures and administrative agencies, where the systemic changes affecting our clients most often take place. (The legal services programs in many states include organizations comparable to MLRI; we are sometimes known by the somewhat infelicitous name “backup centers.”) Among the programs with which MLRI is in communication is the Harvard Legal Assistance Bureau, and I would take this moment to acknowledge the high regard in which I and my colleagues regard their work.

The substantive area of my work is employment law. It is no surprise that during the past three years of our country’s Great Recession, the importance of the unemployment insurance system for our clients has increased enormously and, consequently, it has occupied a greater portion of my time than might otherwise have been the case.

I’m not a statistician nor do I work in a field program representing individual clients, so my comments will not address in any detail the validity of the HLAB study or the conclusions that may properly be drawn from it. As one member of the community of Massachusetts legal services attorneys, however, I have an obvious interest in the way the study portrays us: we are variously described as self-protective, emotional, distrustful of being evaluated, and reluctant to the point of perverseness in participating in randomized studies of the kind the authors wish to conduct. Our resistance in this regard has itself already been the subject of comment here. Happily, it is not often that one looks into what seems to be a mirror and sees the personage looking back wearing a black hat and a snarl. But when it does happen, it’s hard to look away without some effort at clarification. So I will devote my contribution to the symposium to the topic of the perceived reluctance of the legal services community to cooperate in randomized trials. It goes without saying, but the following thoughts are those of only one member of a larger community.

My understanding is that in the HLAB study, no significant case evaluation occurred prior to randomization. Many of us in legal services view with trepidation the idea of ceding control over case selection to the randomization process. Others have more sanguine views, either because they assume that randomization is already taking place or that it ought to be. For example, in his comments from a few months ago, Dave Hoffman was working under the assumption that to randomize client selection would not change an agency’s representation practices at all, and on that basis, he criticized resistance to randomized control trials as “trying to prevent research from happening.”

The authors of the study are enthusiastic about randomization not only because of its scientific value in statistical research but also because it can help to solve one of the thorniest problems facing legal services programs – the scarcity of resources as compared to the demand. As long as the demand for legal assistance outstrips the supply, Professor Greiner has said, randomization – a roll of the dice or the flip of a coin — is an easy and appropriate way to decide who gets representation and who does not.

I believe it’s erroneous to assume that randomization would not change representation practices, at least in the area of legal services in which I work. I also acknowledge that it is possible, at least theoretically, for all the cases in a randomized control trial to have met the provider’s standards for representation. This would provide some measure of reassurance. However, in one area of law, immigration asylum cases, the authors have concluded that time constraints make such an effort unworkable.

Read the rest of this post »

  March 28, 2011 at 5:11 pm   Posted in: Civil Procedure, Empirical Analysis of Law, Law Practice, Symposium (What Difference Representation)   Print This Post   No Comments

What Difference Representation: Inconclusive Evidence

posted by Ted Eisenberg

Congratulations to the authors on an excellent study that promotes and explores the importance of random assignment.

My comment supports the article’s emphasis on caution and not overgeneralizing. My focus is on the article’s Question 2: Did an offer of HLAB representation increase the probability that the claimant would prevail? My analysis of the simple frequencies (I have not delved into the regressions and ignore weights) suggests that HLAB attorneys should view the results as modest, but inconclusive, evidence that an offer of representation improves outcomes.

Based on Table 1, page 24, there are 129 No offer observations and 78 Offer observations. Ignoring weights, which I think are said not to make a huge difference, page 26 reports that .76 of claimants who received an offer prevailed in their first level appeals, and that .72 of claimants who did not receive an offer prevailed in their first level appeal.

So, those who were offered representation fared better; one measure of which is they did .04/.72 x 100, or 5.6% better. Given the high background (no offer condition) rate of prevailing, the maximum improvement (to 1.00 success rate) is .28/.72 x 100 or 38.9%.  Another measure could be the proportionate reduction in defeat. The no offer group was “defeated” 28% of the time. The offer group was defeated 24% of the time.  The reduction in defeat is .04/.28 x 100 is 14.3%. This measure has the sometimes attractive feature that it can range from 0% to 100%. So by this measure the offeree group did 14% better than the non-offeree group, a modest improvement for the offer condition.

A concern expressed in the paper is that the result is not statistically significant. This raises the question: given the sample size, how likely was it that a statistically significant effect would be detected? Assessing this requires hypothesizing what size effect of an offer would be of societal interest.  Suppose we say that lawyers should do about 10% better and move the win rate from .72 for non offerees to .80 for offerees.  This is an 11.1% improvement by the first measure and a 28.6% improvement by the second measure.  Both strike me as socially meaningful but others might specify different numbers.

We can now pose the question: given the sample size and the effect of specified size, what is the probability of observing a statistically significant effect if one exists?  I use the following Stata command to explore the statistical power of the study:

sampsi .72 .80,n1(129) n2(78), which yields the following output:

Estimated power for two-sample comparison of proportions

Test Ho: p1 = p2, where p1 is the proportion in population 1 and p2 is the proportion in population 2

Assumptions:

alpha =   0.0500  (two-sided)

p1 =   0.7200

p2 =   0.8000

sample size n1 = 129

n2 = 78

n2/n1 = 0.60

Estimated power:

power =   0.1936

A power of 0.19 is too low to conclude that the study was large enough to detect an effect of the specified size at a statistically significant level. If one concluded that an offer of representation did not make a significant difference from this study, there is a good chance the conclusion would be incorrect. To achieve power of about 0.70, one would need a sample four times as large as that in the study. If one thought that smaller effects were meaningful, the sample would be even more undersized.

I think my analysis so far underestimates the benefit of an offer by HLAB attorneys.  Perhaps we can take .72 as a reasonable lower bound on success. Even folks without an offer succeeded at that rate.  The realistic upper bound on success is likely not 1.00.  Some cases simply cannot be won, even by the best lawyer in the world. Perhaps not more than 90% of cases are ever winnable, with the real winnable rate likely somewhere between .8 and .9.  If the winnable rate was .8, then the offer got clients halfway there, from .72 to .76. If the real rate was higher, the offer was less effective but not trivial in size.  At .9, the offer got the clients 22% closer to the ideal. The study just was not large enough to detect much of an effect at a statistically significant level.

So while I agree that the study provides no significant evidence that an offer increases success, my analysis (obviously incomplete) suggests that the study provides no persuasive evidence that an offer does not increase success. The study is inconclusive on this issue because of sample size.

HLAB lawyers should not feel that they have to explain away these results; the results modestly, but inconclusively, support the positive effect of an offer because they are in the right direction in a small study.

  March 28, 2011 at 9:29 am   Posted in: Empirical Analysis of Law, Symposium (What Difference Representation)   Print This Post   No Comments

What Difference Presentation?

posted by David Udell

David Udell is the Executive Director of the National Center for Access to Justice and a Visiting Professor from Practice at Cardozo Law School.

In my line of work, I have seen many efforts in the political realm to shut down civil legal services for the poor, and have continually worked to combat such efforts.  In 1996, when the Gingrich Congress barred federally funded legal services lawyers from bringing class actions on behalf of the poor, I left Legal Services for the Elderly in order to finish a lawsuit on behalf of widows and widowers who were suing to compel the United States Treasury to fix its practices for replacing stolen Social Security payments.  When I later moved to the Brennan Center for Justice, I helped bring a lawsuit against the rules that barred legal services lawyers from participating in such class actions, I filed another lawsuit against similar rules that barred law school clinic students from bringing environmental justice cases in Louisiana, and I built a Justice Program at the Brennan Center dedicated to countering such attacks on the poor and on their lawyers.

In their March 3, 2011 draft report, What Difference Representation? Offers, Actual Use, and the Need for Randomization (“the Study”), authors D. James Greiner & Cassandra Wolos Pattanyak are right about the importance of developing a solid evidence base – one founded on methodologies that include randomization – to establish what works in ensuring access to justice for people with civil legal cases. They are right again that in the absence of such evidence, both the legal aid community and its critics are accustomed to relying on less solid data.  And they are smart to “caution against both over- and under-generalization of these study results.”  But, unfortunately, the bare exhortation to avoid over- and under-generalization is not sufficient in the highly politicized context of legal services.

While the authors obviously do not have any obligation to arrive at a particular result, they can be expected to recognize a need to avoid statements that have a high probability to mislead, especially in light of the likely inability of much of the Study’s audience to understand the authors’ methodology and findings.  In fact, because of the Study’s novelty and appearance in a non-scientific journal, it will be relied on to analyze situations where it doesn’t apply, and by people who have no background in social science research, plus it will be given disproportionate weight because so few comparable studies exist to judge it against.  It is these factors, in combination with the politicization of legal services, that make it crucial that the authors’ assertions, particularly in the sections most likely to be seen by lay readers (the title and the abstract), do not extend beyond what the findings justify.

Read the rest of this post »

  March 28, 2011 at 8:04 am   Posted in: Civil Rights, Empirical Analysis of Law, Law Practice, Law Rev (Yale), Law School, Law School (Law Reviews), Symposium (What Difference Representation), Uncategorized   Print This Post   No Comments

What Difference Representation – A Response

posted by Bob Sable

I am the Executive Director of Greater Boston Legal Services, the primary provider of civil legal services to poor people in the greater Boston area.  My program and I have a great stake in assuring that our limited resources are used where they can be most effective.  Indeed we are participating with Professor Greiner in a study of the impact of our staff attorneys’ representation in defense of eviction cases.  My comments refer to the draft dated February 12, 2011.

It is important with any study; however, to know what it concludes and what it does not.  For instance, and most importantly, the study concedes on page 43 that it could draw no conclusions about the effect on outcome for claimants actually receiving representation, as opposed to just an offer of representation.  Thus, this study should be recognized for what it is: a limited analysis of the somewhat abstract concept of “offering” assistance.  Indeed, the study wisely cautions against drawing any conclusions from the study about the usefulness of free legal assistance or even about the usefulness of offers of representation in unemployment cases in general (page 47).

I feel some changes are necessary to avoid much confusion about (and misuse of) this study’s conclusions (or lack thereof) as to the effect of representation itself, as opposed to just the offer.  For instance, given that this study’s principal conclusions are about an offer of representation and not actual representation, a more accurate title to this study would be, “What Difference an Offer of Representation?”  And the very first sentence of the Introduction on page 5 currently reads, “Particularly with respect to low-income clients in civil cases, how much of a difference does legal representation make?”  It is only a footnote that explains the study looks at offers as well as effect, and much later in the study (page 32) that no conclusions were reached at all as to the effect of representation.  Similarly, the conclusion (“Where Do We Go From Here?”) states, “the present study primarily concerned representation effects on legal outcomes affecting the potential client’s pecuniary interests.”

I am concerned also that the results reported in the study with respect to offers of representation by HLAB are misleading at best and of little utility at worst.  This is because nearly half of the control group were represented by counsel and, more significantly, probably that many and perhaps more in the control group got an offer of free representation from my program or another providing free legal services in unemployment cases.  To make an analogy to the medical world, suppose there was a Pfizer drug trial where 50% of Pfizer’s control group were offered the exact same medication from Merck.  Wouldn’t that cast serious doubt on the outcome of the study?  There is no mention of this 49% in either the abstract or introduction which unfortunately are all many readers will read.

Read the rest of this post »

  March 28, 2011 at 12:10 am   Posted in: Civil Rights, Empirical Analysis of Law, Symposium (What Difference Representation), Uncategorized   Print This Post   No Comments

Uncompensated Torts

posted by Dave Hoffman

There’s a fantastic new article out by Rick Swedloff (Rutgers-Camden) called Uncompensated Torts.  Swedloff examines the obstacles (largely insurance-related) to victim compensation for intentional torts, and offers a sobering take on possible solutions.  The one that most interests me is an insurance mandate.  Though I take it from Swedloff’s article that these kinds of mandates are perhaps less popular than the used to be?  Anyway, the whole thing is well-worth checking out.

  March 25, 2011 at 1:44 pm   Posted in: Behavioral Law and Economics, Empirical Analysis of Law   Print This Post   One Comment

What Difference Representation: Introduction to the Symposium

posted by Dave Hoffman

I am delighted to announce that Concurring Opinions will be hosting a symposium next Monday and Tuesday on What Difference Representation? Offers, Actual Use, and the Need for Randomization, the forthcoming Yale Law Journal article by Jim Greiner and Cassandra Wolos Pattanayak.  [Update: You can read all posts in the symposium by clicking on this link.]  As you may recall, What Difference has already caused quite a stir in the clinical and legal aid communities. Given our shared interest in questions of empirical methodology, and Jaya’s background in clinical legal services, we decided that bringing that debate to CoOp would be an excellent use of our time and energy. Here’s the (revised) abstract) – though you should download the article if you haven’t already:

“We report the results of the first in a series of randomized control trials designed to measure the effect of an offer of, and the actual use of, legal representation. The results are unexpected. In the context of administrative litigation to determine eligibility for unemployment benefits, a service provider’s offer of representation to a claimant had no statistically significant effect on the claimant’s probability of a victory, but the offer caused a delay in the proceeding. Because a substantial percentage of the provider’s client base consisted of claimants who were initially denied benefits but who would later have that initial denial reversed as a result of the litigation, the offer of representation inflicted a harm upon such claimants in the form of an additional waiting time for benefits to begin, this with no discernible increase in the probability of a favorable outcome. In other words, within the limits of statistical uncertainty, these claimants would have been better off without the offer of representation. The size of the delay (around two weeks, depending on how measured) was not large in absolute terms, and would have been negligible in many other legal settings, but was relevant in the context of this particular administrative and legal framework, one in which speed has remained a special concern for decades. Moreover, in a small number of cases with a certain profile, the delay caused the unemployment system to continue paying benefits erroneously for a longer period of time, potentially imposing costs on the financing of the unemployment system. We were also able to verify a delay effect due to the actual use of (as opposed to an offer of) representation; we could come to no firm conclusion on the effect of actual use of representation on win/loss.

We hypothesize three potential explanations for our findings (and acknowledge that others are possible). First, it is possible that the client base that reached out to the service provider (and thus was subject to randomization) was a specialized subset of unemployment claimants, a subset that did not actually need legal assistance. This theory suggests special attention to provider intake systems. Second, it is possible that the administrative adjudicatory system at issue, with its semi-inquisitorial style of judging, is pro se friendly. Third, it is possible that the subject matter in dispute in these cases is less legally, factually, or procedurally complex than in other settings.

We caution against both over- and under-generalization of these study results. We use these results as a springboard for a comprehensive review of the quantitative literature on the effect of representation in civil proceedings. We find that this literature provides virtually no credible information, excepting the results of two randomized evaluations occurring in different legal contexts and separated by over three decades. We conclude by advocating for, and describing challenges associated with, a large program of randomized evaluation of the provision of representation, particularly by legal services providers.”

We have assembled a terrific group of symposiasts, mixing clinicians, academic empiricists and practitioners. Besides Jim, Cassandra, Jaya and me,  the group includes twelve contributors, lauded in detail after the jump:

Read the rest of this post »

  March 23, 2011 at 11:23 am   Posted in: Behavioral Law and Economics, Empirical Analysis of Law, Law Practice, Law Rev (Yale), Law School, Law School (Scholarship), Symposium (What Difference Representation)   Print This Post   One Comment

UCLA Law Review Vol. 58, Issue 3 (February 2011)

posted by UCLA Law Review

Volume 58, Issue 3 (February 2011)

Articles

Good Faith and Law Evasion	Samuel W. Buell	611
Making Sovereigns Indispensable: Pimentel and the Evolution of Rule 19	Katherine Florey	667
The Need for a Research Culture in the Forensic Sciences	Jennifer L. Mnookin et al.	725
Commentary on The Need for a Research Culture in the Forensic Sciences	Joseph P. Bono	781
Commentary on The Need for a Research Culture in the Forensic Sciences	Judge Nancy Gertner	789
Commentary on The Need for a Research Culture in the Forensic Sciences	Pierre Margot	795

Comments

What’s Your Position? Amending the Bankruptcy Disclosure Rules to Keep Pace With Financial Innovation	Samuel M. Kidder	803
Defendant Class Actions and Patent Infringement Litigation	Matthew K. K. Sumida	843

  February 25, 2011 at 1:19 pm   Posted in: Bankruptcy, Civil Procedure, Constitutional Law, Courts, Criminal Law, Criminal Procedure, Current Events, Economic Analysis of Law, Empirical Analysis of Law, Evidence Law, History of Law, Indian Law, Intellectual Property, International & Comparative Law, Jurisprudence, Law and Humanities, Law and Inequality, Law and Psychology, Law Practice, Law Rev (UCLA), Psychology and Behavior, Race, Sociology of Law, Supreme Court   Print This Post   No Comments

The Representation Debate Continues

posted by Dave Hoffman

Jim Greiner and Cassandra Wolos Pattanayak have emailed me a reply to the Harvard Legal Aid Bureau’s comment on What Difference Representation. Since the topic has been the subject of several posts here, as well as some off-line communication from interested readers, I figured that I owed Greiner/Pattanayak a public space for reply.  It consists of a bit of introductory text, and a longer (9-page) paper.

“We recently became aware that HLAB President Rachel Lauter and HLAB Faculty Director David Grossman had written an email to the clinical listserve addressing our paper “What Difference Representation?”.  The email has been posted to various locations in the blogosphere.  Because the email expresses criticisms of the paper that we also have received from one or two other sources, we thought we would take the opportunity the email presented to clarify certain issues.  For example, President Lautner and Professor Grossman echo reactions we have received from another legal aid provider when they say that our study produced “only limited information,” and that more (and more useful) information would be available if we would just analyze the data properly.  We explain here that the analysis the email (and one or two other legal services providers) have advocated is statistically invalid, and that in any event the data required for it do not presently exist and cannot at this time be ethically collected.  As ought to be clear by now, we have the greatest respect for the students of HLAB, including President Lautner, and HLAB’s clinical faculty, including Professor Grossman.  We are using President Lautner and Professor Grossman’s email as a convenient foil representative of a few other comments we have received.

The substance of our response can be captured in the answers to two questions.

1.  Why study the effect of offers of HLAB representation?  All agree that the effect of actual use of representation is interesting, although as we will explain, perhaps less so than one might think at first.  But why study the effect of HLAB offers?

2.  Why not compare those who got offers from any source, not just HLAB, to those who did not get any such offers?}  This is what President Lauter, Professor Grossman, and a few others have suggested.  Why not make this comparison?

We also answer one final question:

3.  So how can we find out about the effect of offers from other service providers?”

To read the full response, click here.

  January 31, 2011 at 6:20 am   Posted in: Behavioral Law and Economics, Empirical Analysis of Law, Law School, Psychology and Behavior   Print This Post   One Comment

Harvard Clinic Responds to Greiner Study

posted by Dave Hoffman

  January 24, 2011 at 2:06 pm   Posted in: Behavioral Law and Economics, Civil Rights, Empirical Analysis of Law, Law School (Teaching)   Print This Post   No Comments

Experiments in Lawyering, Take Two

posted by Dave Hoffman

Ian Ayres, commenting on What Difference Representation, asks: “Does HLAB have a duty to stop offering representation or to change its modus operandi? Does it at least have an ethical duty to disclose the results of the study to prospective clients? Can other student legal service organizations ethically ignore the results of the study?”  I agree that the paper raises important questions – though I’m not sure the result generalizes to all “legal service organizations.”   LHAB  represents clients in employment benefit cases, where narrative framing & rough justice is likely to predominate.  This is the weakest case for the representation effect imaginable.  If we saw a similar effect in refugee clinics, or ones offering criminal defense services, it would be mindblowing.  But until we do, I think a little caution is warranted.

How could we encourage experimentation that would help us answer this important question?  The obvious place to start would be the ABA Accreditation Standards Committee, already engaged in a huge debate about both skills training & outcome measures.  If the Committee were so minded, it might link new requirements for more clinical education to rigorous ways to measure whether those new clinics and skill training opportunities produce better results for clients. By incorporating randomized control studies into clinic design, the ABA could thereby produce highly useful data for legal policymakers (which would certainly be otherwise unavailable), and would ensure that schools spend money on clinical education in ways that are socially beneficial.  Given that almost all law school clinics are incredibly oversubscribed, it ought to be quite easy to sell randomization in selection, unless there is an important pedagogical reason for selection that I’m missing.

I recognize that given the politics of clinical education, my proposal is a nonstarter. However, given that it is exceedingly difficult to study the effect of representation by looking only at observational data, randomized clinical selection might be our best chance to figure out when and how lawyers add value.  It would also allow the Bar & legal academy to advance two newly dominant goals for legal education, at very low cost.  What’s not to like?

  January 11, 2011 at 2:20 pm   Posted in: Empirical Analysis of Law   Print This Post   No Comments

• « Older Entries
• Newer Entries »