As others have commented, Larry always told you when he thought you were being an idiot.  When I presented one of my early empirical papers at an otherwise warm-and-friendly Canadian Law and Economics conference, Larry provided comments from the audience that had me wanting to go back to running fire drills at Cravath.  My god, how he schooled me!  But he was basically right, and it was business, not personal.  Some years later, he provided crucial encouragement on a new (better?) empirical paper.  Praise felt twice as good coming from him.  What [...]]]> Larry Ribstein, who died earlier this week, was a galvanic force as a scholar and blogger.  I join those who’ve expressed sadness and loss at his untimely passing.  I figured I’d add two comments.

As others have commented, Larry always told you when he thought you were being an idiot.  When I presented one of my early empirical papers at an otherwise warm-and-friendly Canadian Law and Economics conference, Larry provided comments from the audience that had me wanting to go back to running fire drills at Cravath.  My god, how he schooled me!  But he was basically right, and it was business, not personal.  Some years later, he provided crucial encouragement on a new (better?) empirical paper.  Praise felt twice as good coming from him.  What a teacher he must have been!

Second, I’ve recently read his book (coauthored with Erin O’Hara) The Law Market.  I think it’s simply amazing – provocative, and in some ways as mind-opening as Stuntz’s Collapse of American Criminal Justice.  Law and economics has lost a great and unique voice.

As promised, I’m filing a report from the Sixth Annual Empirical Studies Conference, held 11/4-11/5 at Northwestern Law School.  Several of the attendees at the Conference approached me and remarked on my posts from CELS V, IV, and III. That added pressure, coupled with missing half of the conference due to an unavoidable conflict, has delayed this post substantially.  Apologies!  Next time, I promise to attend from the opening ceremonies until they burn the natural law figure in effigy.  Next year’s conference is at Stanford.  I’ll make a similar offer to the one I’ve made in the past: if the organizing committee pays my way, I promise not only to blog [...]]]>

Northwestern's Stained Glass Windows Made Me Wonder Whether Some Kind of Regression Was Being Proposed

As promised, I’m filing a report from the Sixth Annual Empirical Studies Conference, held 11/4-11/5 at Northwestern Law School.  Several of the attendees at the Conference approached me and remarked on my posts from CELS V, IV, and III. That added pressure, coupled with missing half of the conference due to an unavoidable conflict, has delayed this post substantially.  Apologies!  Next time, I promise to attend from the opening ceremonies until they burn the natural law figure in effigy.  Next year’s conference is at Stanford.  I’ll make a similar offer to the one I’ve made in the past: if the organizing committee pays my way, I promise not only to blog the whole thing, but to praise you unstintingly.  Here’s an example: I didn’t observe a single technical or organization snafu at Northwestern this year.  Kudos to the organizing committee: Bernie Black, Shari Diamond, and Emerson Tiller.

What I saw

I arrived Friday night in time for the poster session.  A few impressions.  Yun-chien Chang’s Tenancy in ‘Anticommons’? A Theoretical and Empirical Analysis of Co-Ownership won “best poster,” but I was drawn to David Lovis-McMahon & N.J. Schweitzer’s Substantive Justice: How the Substantive Law Shapes Perceived Fairness.  Overall, the trend toward professionalization in poster display continues unabated.  Even Ted Eisenberg’s poster was glossy & evidenced some post-production work — Ted’s posters at past sessions were, famously, not as civilized. Gone are the days where you could throw some powerpoint slides onto a board and talk about them over a glass of wine!  That said, I’m skeptical about poster sessions generally.  I would love to hear differently from folks who were there.

On Saturday, bright eyed and caffeinated, I went to a Juries panel, where I got to see three pretty cool papers.  The first, by Mercer/Kadous, was about how juries are likely to react to precise/imprecise legal standards.  (For a previous version, see here.) Though the work was nominally about auditing standards, it seemed generalizable to other kinds of legal rules.  The basic conclusion was that imprecise standards increase the likelihood of plaintiff verdicts, but only when the underlying conduct is conservative but deviates from industry norms.  By contrast, if the underlying conduct is aggressive, jurors return fewer pro-plaintiff verdicts.  Unlike most such projects, the authors permitted a large number of mock juries to deliberate, which added a degree of external validity.  Similarly worth reading was Lee/Waters’ work on jury verdict reporters (bottom line: reporters aren’t systematically pro-plaintiff, as the CW suggests, but they are awfully noise measures of what juries are actually doing).  Finally, Hans/Reyna presented some very interesting work on the “gist” model of jury decisionmaking.

At 11:00, I had to skip a great paper by Daniel Klerman whose title was worth the price of admission alone – the Selection of Thirteenth-Century Disputes for Litigation.  Instead, I went to Law and Psychology III.  There, Kenworthey Bilz presented Crime, Tort, Anger, and Insult, a paper which studies how attribution & perceptions of dignitary loss mark a psychological boundary between crime and tort cases.  Bilz presented several neat experiments in service of her thesis, among them a priming survey- – people primed to think about crimes complete the word “ins-” as “insult,” while people primed to think about torts complete it as “insurance.”  (I think I’ve got that right – - the paper isn’t available online, and I’m drawing on two week old memories.)

At noon, Andrew Gelman gave a fantastic presentation on the visualization of empirical data.  The bottom line: wordles are silly and convey no important information.  Actually, Andrew didn’t say that.  I just thought that coming in.  What Andrew said was something more like “can’t people who produce visually interesting graphs and people who produce graphs that convey information get along?”

Finally, I was the discussant at an Experimental Panel, responding to Brooks/Stremitzer/Tontrup’s Framing Contracts:Why Loss Framing Increases Effort.  Attendees witnessed my ill-fated attempt to reverse the order of my presentation on the fly, leading me to neglect the bread in the praise sandwich.  This was a good teaching moment about academic norms. My substantive reaction to Framing Contracts is that it was hard to know how much the paper connected to real-world contracting behavior, since the kinds of decision tasks that the experimental subjects were asked to perform were stripped of the relational & reciprocal norms that characterize actual deals.

CELS: What I missed

The entire first day!  One of my papers with the cultural cognition project, They Saw a Protest, apparently came off well.  Of course, there was also tons of great stuff not written from within the expanding cultural cognition empire.  Here’s a selection: on lawyer optimism; on public housing, enforcement and race; on probable cause and hindsight judging; and several papers on Iqbal, none of which appear to be online.

What did you see & like?

According to John Summers and Michael Newman, we’ve forgotten that every case the Supreme Court takes implicitly also considers shadow cases from other circuits ruling on the same issue — that is, the Supreme Court doesn’t just “reverse” the circuit on direct appeal, it also affirms (or reverses) coordinate circuits while resolving a split.  Thus, both our numerator and our denominator have been wrong.  They’ve written up the results of this pretty interesting approach [...]]]> What is the meaning of an appellate court’s “reversal rate”?  Opinions vary.  (My view, expressed, succinctly, is “basically nothing.”) However conceived, we ought to at least be measuring reversal correctly.  But two lawyers at Hangley Aronchick, a Philadelphia law firm, think that scholars (and journalists) have conceptualized reversal in entirely the wrong way.

According to John Summers and Michael Newman, we’ve forgotten that every case the Supreme Court takes implicitly also considers shadow cases from other circuits ruling on the same issue — that is, the Supreme Court doesn’t just “reverse” the circuit on direct appeal, it also affirms (or reverses) coordinate circuits while resolving a split.  Thus, both our numerator and our denominator have been wrong.  They’ve written up the results of this pretty interesting approach to reversal in a paper you can find blurbed here.   Among the highlights: (1) reversal is less common that is commonly supposed; (2) the Court doesn’t predictably follow the majority of circuits; (3) there are patterns of concordance between circuits in analyzing issues; and (4) even under the new approach, the ninth circuit is still the least loyal agent of the Supreme Court.

I think that this method has real promise, and I bet that folks who are interested in judicial behavior will want to check it out.

My [...]]]> Earlier this month, right here on this very blog, Dave Hoffman pontificated about two of my favorite subjects: empirical legal studies and baseball. Primarily, Dave wondered about whether empirical legal research was facing might face the same problem as sabermatic baseball analysis: inaccessible complexity. I won’t rehash his argument because he did a very good job of explaining it in the original post. Although I completely agree with his conclusion that empirical legal studies should seek to be more accessible (which I always note at the end of my introduction of my empirical work), I disagree with his contention that empirical legal studies are facing might face widespread incomprehensibility due to growing complexity. Because I think it is a helpful analogy, I’ll borrow Dave’s example of advanced statistics in baseball.

My initial problem with Dave’s contention is that empirical legal studies have barely scratched the surface in some areas of research. In my own neck of the woods, law and courts, there has been an immense amount of study done primarily by political science and legal scholars. Yet, it wasn’t that long ago that scholars finally recognized that judges vote differently based upon who they are sitting with (panel effects). And despite strong empirical evidence that the same models don’t work outside of the Supreme Court, most scholarship has simply transplanted models of decisionmaking by the Justices to lower courts. My personal bugaboo outlined in my most recent article is that studies of lower courts, in particular federal Courts of Appeals, continue to treat judges as a monolithic group with no diversity except for the party of their appointing President. As an example of how dominant the belief of a homogeneity among judges is, consider this quote from a prominent scholar in a University of Chicago Law Review article:

“The jurisdictional competition model assumes that judges are homogeneous. Of course, they were not. Introducing heterogeneity would complicate the model without substantially altering the conclusions.”

This quote hardly does justice to the endemic nature of the belief that it is unnecessary to consider the significant personal variations that might exist among judges. It just happens that this particular author demonstrated his awareness of the issue by discussing the assumption whereas empirical legal scholars typically let the premise go unsaid in their judicial decisionmaking research. By not including possible variations in empirical legal studies of judicial behavior (beyond the appointing President’s party), the dominant law and courts research has yet to enter the realm of complexity comparable to advanced sabermatic research. Further, as law and courts is one of the most heavily researched areas using empirical methods, the simplistic assumptions of its models is indicative that other areas might be at even more basic levels of analysis.

I think the baseball analogy is particularly helpful in explaining my second objection to Dave’s reasoning. There are numerous concepts in sabermatic work that are beyond even sophisticated connoisseurs of such information. However, those advanced studies and ideas still find a way to inform statistical analysis in the mainstream (excluding Joe Morgan). I’m guessing that most people have no idea who Voros McCracken is, but he produced one of the most significant statistical findings in baseball history. McCracked discovered that, with few exceptions, a pitcher has little control over whether a ball in play is a hit or not. Instead, a pitcher can typically only control the three true outcomes: Walk, Strikeout, and Home Run. This insight supported by statistical analysis revolutionized the way we understand pitchers. Initially there was substantial resistance among baseball insiders. Had the management of the Diamondbacks, for example, listened to McCracken, there was no way they would have signed Russ Ortiz to the contract that every informed sabermatician knew was a disaster from day one. Eventually, McCracken’s ideas have infiltrated common understanding of educated baseball fans even though few probably know who McCracken is and even fewer could read and understand his study. As it turns out, even more complexity has been added to McCracken’s initial conclusions yielding even more insight into the relationship between pitching and hitting in baseball. More recent statistics such as WAR (Wins Above Replacement) and VORP (Value over Replacement Player) have started to be used by the mainstream press and baseball announcers. Yet, while people have come to understand the concepts measured by those stats, I doubt almost anyone could tell you how those stats are actually computed. Indeed, there is an ongoing fight among sabermaticians about which measurement of WAR is actually right (the difference is based upon which defensive metrics are used). For empirical legal studies to be important, not every study needs to be accessible. That isn’t to say that accessibility shouldn’t be a central goal. However, just like with sabermatics, the advanced studies ultimately can yield commonly held ideas.

Regardless of my quibbles with Dave’s post, I think this is an important discussion for scholars, empirical and otherwise, to have. And my agreements with his normative beliefs on the subject are more significant than the disagreements.

There’s only one problem with those projections: it turns out that “moving Medicaid recipients into managed care ‘did not lead to lower Medicaid spending during the 1991 to 2003 period,’” according to a report published by the National [...]]]> The Washington Post has featured two interesting pieces recently on Medicaid managed care. Christopher Weaver reported on a battle between providers and insurers in Texas. Noting that “federal health law calls for a huge expansion of the Medicaid program in 2014,” Weaver shows how eager insurers are to enroll poor individuals in their plans. Each enrollee would “yield on average $7 a month profit,” according to recent calculations. Cost-cutting legislators see potential fiscal gains, too, once the market starts working its magic.

There’s only one problem with those projections: it turns out that “moving Medicaid recipients into managed care ‘did not lead to lower Medicaid spending during the 1991 to 2003 period,’” according to a report published by the National Bureau of Economic Research this month. Sarah Kliff is surprised to find that this is “the first national look at whether Medicaid managed care has actually done a key thing that states want it to do.”

I share the frustration that we don’t have enough information on key policy issues. On the other hand, I can see why economists and other social scientists would be reluctant to pronounce on the efficacy of Medicaid managed care.

The NBER study suggested that Medicaid rates are often already “low enough that private insurers couldn’t negotiate a similar payment” rate. When the managed care providers did manage to reduce overall spending, they usually did so in states with above-average compensation rates for Medicaid providers. So how would we judge a provider which managed to reduce spending in a state with average or below average rates for providers? Probably not that highly if the strategy focused on delaying or denying access to needed care. Discussing past problems in managed care, one health law casebook notes that an “HMO was accused of putting its enrollment office on the third floor of a building without an elevator, and another tried to attract healthy elderly subscribers by holding square dances.”*

There are a wide range of insurer activities that can add or reduce costs to a system. But how do we assure that we are measuring the effects of an intervention correctly? Some laudable innovations, for example, may be costly upfront, but will end up saving many times the initial spend years down the road. “Meat ax” rationing of pills or procedures may reduce spending now, but worsen health conditions that eventually become extremely expensive. And when we consider quotidian decisions about denying or approving care, value judgments become even more complex. Accurate coding of whether a certain denial of coverage was actually appropriate can be very difficult. Empirical facts can be subject to widely divergent interpretations.

This is not to say that more studies on Medicaid managed care, and other health system innovations, should not be done. (It would be interesting to add up how much study spending was commissioned by the ACA (and even Dodd-Frank), as compared with policies like ELSI (which mandated a certain percentage of grants for the Human Genome Project to study its ethical, legal, and scientific implications).) I am just flagging the difficulties in obtaining “hard,” “objective” proof of the value of certain policies. There is usually plenty of room to massage the numbers.

X-Posted: Health Reform Watch.

*Hall, Bobinski, and Orentlicher’s Health Law, citing Mary Crossley, Discrimination against the Unhealthy in Health Insurance, 54 U. Kan. L. Rev. 73 (2005).

Photo Credit: Quikbeam.

Reading these two articles on the problems of complexity for sabermetrics, I wondered if the empirical legal studies community is coming soon to a similar point of crisis. The basic concern is that sabermetricians are devoting oodles of time to ever-more-complex formulae which add only a small amount of predictive power, but which make the discipline more remote from lay understanding, and thus less practically useful.   Basically: the jargonification of a field.  Substituting “law” into Graham MacAree‘s article on the failings of sabermetrics, we get the following dire warning:

“Proper [empirical legal analysis] is something that has to come from the top down ([law]-driven) rather than the bottom up (mathematics/data driven), and to lose sight of that causes a [...]]]>

Kenesaw Mountain Landis would have hated both sabermetrics and ELS.

Reading these two articles on the problems of complexity for sabermetrics, I wondered if the empirical legal studies community is coming soon to a similar point of crisis. The basic concern is that sabermetricians are devoting oodles of time to ever-more-complex formulae which add only a small amount of predictive power, but which make the discipline more remote from lay understanding, and thus less practically useful.   Basically: the jargonification of a field.  Substituting “law” into Graham MacAree‘s article on the failings of sabermetrics, we get the following dire warning:

“Proper [empirical legal analysis] is something that has to come from the top down ([law]-driven) rather than the bottom up (mathematics/data driven), and to lose sight of that causes a whole host of issues that are plaguing the field at present. Every single formula must be explainable without recourse to using ridiculous numbers. Every analystmust be open to thinking about the [law] in new ways. Every number, every graph in a [ELS] piece musttell a [legal] story*, because otherwise we’re no longer writing about the [legal system ] but indulging in blind number-crunching for its own sake. …

Surveying the field, I no longer believe that those essential precepts hold sway over the [ELS] community. Data analysis methods are being misapplied and sold to readers as the next big thing. Articles are being written for the sake of sharing irrelevant changes in irrelevant metrics. Certain personalities are so revered that their word is taken as gospel when fighting dogma was what brought them the respect they’re now given in the first place. [ELS] is in a sorry state.

How do we fix it? Well, the answer seems simple. [ELS] shouldn’t be so incomprehensible so as not to call up the smell of [a courtroom, or the careful drafting of the definition clauses in a contract, or the delicate tradeoffs involved in family court practice, or the importance of situation sense]. Statistics shouldn’t be sterile and clean and shiny and soulless. They shouldn’t just be about [Law]; they should invoke it. Otherwise, they run the risk of losing the language which makes them so special.”

Note: this is an entirely different  than Leiter’s 2010 odd critique that ELS work was largely mediocre.  The problem, rather, is that the trend is toward a focus on more complex and “accurate” models, often without the input of people with legal training, and insufficient attention to how such models will be explained to lawyers, judges and legal policymakers.  (See also all of Lee Epstein’s work.)

“This paper provides the first comprehensive assessment of the Federal Judicial Center’s long-anticipated study of motions to dismiss for failure to state a claim after Iqbal v. Ashcroft. Three primary assessments are made of the FJC’s study. First, there are reasons to be concerned that the study may be providing an incomplete picture of actual Rule 12(b)(6) activity. Even if [...]]]> Several months ago, the FJC put out a well-publicized study assessing the results of Twombly-Iqbal on motions practice.  It concluded that there was little reason, overall, for concern that the Supreme Court’s new pleadings jurisprudence had worked a revolutionary change down below.  Lonny Hoffman (Houston) has just released an important new paper which questions the methods and conclusions of the FJC’s work.  He pulls no punches:

“This paper provides the first comprehensive assessment of the Federal Judicial Center’s long-anticipated study of motions to dismiss for failure to state a claim after Iqbal v. Ashcroft. Three primary assessments are made of the FJC’s study. First, there are reasons to be concerned that the study may be providing an incomplete picture of actual Rule 12(b)(6) activity. Even if the failure to capture all relevant motion activity was a non-biased error, the inclusiveness problem is consequential. Because the study was designed to compare over time the filing and grant rate of Rule 12(b)(6) motions, the size of the effect of the Court’s cases turns on the amount of activity found. Second, even if concerns are set aside that the collected data may be incomplete, it misreads of the FJC’s findings to conclude that the Court’s decisions are having no effect on dismissal practice. The FJC found that after Iqbal, a plaintiff is twice as likely to face a motion to dismiss. This sizeable increase in rate of Rule 12(b)(6) motion activity represents a marked departure from the steady filing rate observed over the last several decades and means, among other consequences, added costs for plaintiffs who have to defend more frequently against these motions. The data regarding orders resolving dismissal motions even more dramatically shows the consequential impacts of the Court’s cases. There were more orders granting dismissal with and without leave to amend, and for every case category examined. Moreover, the data show that after Iqbal it was much more likely that a motion to dismiss would be granted with leave to amend (as compared to being denied) both overall and in the three largest case categories examined (Civil Rights, Financial Instruments and Other). Employment Discrimination, Contract and Torts all show a trend of increasing grant rates. In sum, in every case type studied there was a higher likelihood after Iqbal that a motion to dismiss would be granted. Third, because of inherent limitations in doing empirical work of this nature, the cases may be having effects that the FJC researchers were unable to detect. Comparing how many motions were filed and granted pre-Twombly to post-Iqbal cannot tell us whether the Court’s cases are deterring some claims from being brought, whether they have increased dismissals of complaints on factual sufficiency grounds, or how many meritorious cases have been dismissed as a result of the Court’s stricter pleading filter. Ultimately, perhaps the most important lesson to take away from this last assessment of the FJC’s report is that empirical study cannot resolve all of the policy questions that Twombly and Iqbal raise.”

I should disclose that I provided Lonny comments on an earlier draft, and overall I think he’s done an incredible (and generally very fair) job.  One thing to think about, as always when evaluating litigation data, is the degree to which we would expect to see any results at all given case selection effects.  That Lonny does observe such substantively significant changes notwithstanding selection tells us something about how dramatic the Twiqbal decisions really were.

This frustrates me, and if I could wave a magic wand (or controlled Google) I would create a national e-docketing system for [...]]]> Though many bemoan the expense and terrible functionality of PACER, the federal government’s electronic docketing system, it is vastly superior to existing state alternatives.  While some states have decent, and searchable, e-dockets, others do not, and it’s often quite hard to figure out the scope of the state databases.  The result is that a researcher (or a lawyer) who wants to study live dockets at the state level is faced with a host of known unknowns, making aggregate statistical inference basically impossible.  Even descriptive statistics about state courts are hard to verify.  It’s a black hole. (With some illumination provided by the BJS and other bodies.)

This frustrates me, and if I could wave a magic wand (or controlled Google) I would create a national e-docketing system for all state filings, permitting full-text searchers across states for comprehensive data – including searches of motions and orders – in both civil and criminal litigation.  The current state of the world, by contrast, directs much of the new empirical legal research to focus on federal cases and federal outcomes, because PACER provides access to the kinds of data that researchers need.  The problem, of course, is that PACER collects only Federal dockets, which aren’t representative of the kind or scope of litigation nationwide. Though of course studying dockets is vastly superior to studying opinions – if you want to know what judges are doing – we’re left still peering through a dark piece of glass.  Worse, I think, is that researchers end up focusing their energies on topics for which federal litigation is the dominant way of resolving legal claims.  Thus, there’s much more, and much better, docket-centered empirical work about securities law and federal civil rights statutes than there is about common law adjudication.

Our sadly patchwork court records system  doesn’t just hurt academics looking to illuminate doctrinal puzzles.  (The horror! Tenured professors can’t write more papers!)  It also means that lawyers and corporate officers may be forced to rely on anecdote and salience when deciding how to engage with the litigation system — a calculation that may lead such repeat players to develop a long-term strategy to exit the litigation system altogether.  If the state courts want to preserve their business, they need to innovate.  One way to do so would be to join forces in data collection, archival, and search.

(Image Source: Flicker)

“To Dave, we say that our enthusiasm for randomized studies is high, but perhaps not high enough to consider a duty to randomize among law school clinics or among legal services providers.  We provided an example in the paper of practice in which randomization was inappropriate because collecting outcomes might have exposed study subjects to deportation proceedings.  We also highlighted in the paper that in the case of a practice (including possibly a law school clinic) that focuses principally on systemic change, randomization of that practice is not constructive.  Instead, what should be done is a series of randomized studies of an alternative service provider’s practice in that same adjudicatory system; these alternative provider studies can help to assess whether the first provider’s efforts at systemic change have [...]]]> Jim and Cassandra write:

“To Dave, we say that our enthusiasm for randomized studies is high, but perhaps not high enough to consider a duty to randomize among law school clinics or among legal services providers.  We provided an example in the paper of practice in which randomization was inappropriate because collecting outcomes might have exposed study subjects to deportation proceedings.  We also highlighted in the paper that in the case of a practice (including possibly a law school clinic) that focuses principally on systemic change, randomization of that practice is not constructive.  Instead, what should be done is a series of randomized studies of an alternative service provider’s practice in that same adjudicatory system; these alternative provider studies can help to assess whether the first provider’s efforts at systemic change have been successful.”

I meant to cabin my argument to law school clinics.  And I do understand that there may be very rare cases where collecting outcomes will hurt clients (such as deportation).  But what about a clinic that focuses on “systemic change.” Let’s assume that subsidizing such a clinic would be a good thing for a law school to do (or, put it another way, we think it is a good idea for current law students to incur more debt so that society gets the benefit of the clinics’ social agitation).  Obviously, randomization of client outcomes would be a terrible fit for measuring the success of such a clinic.  It would be precisely the kind of lamppost/data problem that Brian Leiter thinks characterizes much empirical work.

But that doesn’t mean that randomization couldn’t be useful in measuring other kinds of clinic outcomes.  What about randomization in the allocation of law student “employees” to the clinic as a way to measure student satisfaction in the “learning outcomes“? Or randomization of intake and utilizing different client contact techniques as a way of measuring client satisfaction with their representation (or feelings about the legitimacy of the system?)  One thing that the commentators in this symposium have tried to emphasize is that winning & losing aren’t the only outputs of the market for indigent legal services.  Controlled study of the actors in the system needn’t be constrained in the way that Jim and Cassandra’s reply to my modest proposal to mandate randomization suggest.

Both Dave and Kevin offer  thoughts about the levelof enthusiasm legal empiricists, legal services providers, and clinicians should have for randomized studies.  We find ourselves in much but not total agreement with both.  To Kevin, we suggest that there is more at stake than just finding out whether legal assistance helps potential clients.  In an era of scarce legal resources, providers and funders have to make allocation decisions across legal practice areas (i.e., should we fund representation for SSI/SSDI appeals or for unemployment appeals or for summary eviction defense).  That requires more precise knowledge about how large representation [...]]]> We thank Kevin Quinn and David Hoffman for taking the time to comment in our paper.  Again, these are two authors whose work we have read and admired in the past.

Both Dave and Kevin offer  thoughts about the levelof enthusiasm legal empiricists, legal services providers, and clinicians should have for randomized studies.  We find ourselves in much but not total agreement with both.  To Kevin, we suggest that there is more at stake than just finding out whether legal assistance helps potential clients.  In an era of scarce legal resources, providers and funders have to make allocation decisions across legal practice areas (i.e., should we fund representation for SSI/SSDI appeals or for unemployment appeals or for summary eviction defense).  That requires more precise knowledge about how large representation (offer or actual use) effects are, how much bang for the buck.  Perhaps even more importantly, scarcity requires that we learn how to triage well; see Richard Zorza’s posts here and the numerous entries in his own blog on this subject.  That means studying the effects of limited interventions.  Randomized trials provide critical information on these questions, even if one agrees (as we do) that in some settings, asking whether representation (offer or actual use) helps clients is like asking whether parachutes are useful.

Thus, perhaps the parachute analogy is inapt, or better, it requires clarification:  we are in a world in which not all who could benefit from full-service parachutes can receive them.  Some will have to be provided with rickety parachutes, and some with little more than large blankets.  We all should try to change this situation as much as possible (thus the fervent hope we expressed in the paper that funding for legal services be increased).  But the oversubscription problem is simply enormous.  When there isn’t enough to go around, we need to know what we need to know to allocate well.  Meanwhile, randomized studies can also provide critical information on the pro se accessibility of an adjudicatory system, which can lay the groundwork for reform.

To Dave, we say that our enthusiasm for randomized studies is high, but perhaps not high enough to consider a duty to randomize among law school clinics or among legal services providers.  We provided an example in the paper of practice in which randomization was inappropriate because collecting outcomes might have exposed study subjects to deportation proceedings.  We also highlighted in the paper that in the case of a practice (including possibly a law school clinic) that focuses principally on systemic change, randomization of that practice is not constructive.  Instead, what should be done is a series of randomized studies of an alternative service provider’s practice in that same adjudicatory system; these alternative provider studies can help to assess whether the first provider’s efforts at systemic change have been successful.

Our great thanks to both Kevin and Dave for writing, and (obviously) to Dave (and Jaya) for organizing this symposium.

I’m a legal services attorney in Boston. My employer, Massachusetts Law Reform Institute (MLRI), has as one of its primary tasks to connect the state’s field programs, where individual client representation occurs, with larger political bodies, including legislatures and administrative agencies, where the systemic changes affecting our clients most often take place. (The legal services programs in many states include organizations comparable to MLRI; we are sometimes known by the somewhat infelicitous name “backup centers.”) Among the programs with which MLRI is in communication is the Harvard Legal Assistance Bureau, and I would take this moment to acknowledge the high regard in which I and my colleagues regard their work.

The substantive area [...]]]> Thanks for the invitation to participate in this interesting and provocative symposium.

I’m a legal services attorney in Boston. My employer, Massachusetts Law Reform Institute (MLRI), has as one of its primary tasks to connect the state’s field programs, where individual client representation occurs, with larger political bodies, including legislatures and administrative agencies, where the systemic changes affecting our clients most often take place. (The legal services programs in many states include organizations comparable to MLRI; we are sometimes known by the somewhat infelicitous name “backup centers.”) Among the programs with which MLRI is in communication is the Harvard Legal Assistance Bureau, and I would take this moment to acknowledge the high regard in which I and my colleagues regard their work.

The substantive area of my work is employment law. It is no surprise that during the past three years of our country’s Great Recession, the importance of the unemployment insurance system for our clients has increased enormously and, consequently, it has occupied a greater portion of my time than might otherwise have been the case.

I’m not a statistician nor do I work in a field program representing individual clients, so my comments will not address in any detail the validity of the HLAB study or the conclusions that may properly be drawn from it. As one member of the community of Massachusetts legal services attorneys, however, I have an obvious interest in the way the study portrays us: we are variously described as self-protective, emotional, distrustful of being evaluated, and reluctant to the point of perverseness in participating in randomized studies of the kind the authors wish to conduct. Our resistance in this regard has itself already been the subject of comment here. Happily, it is not often that one looks into what seems to be a mirror and sees the personage looking back wearing a black hat and a snarl. But when it does happen, it’s hard to look away without some effort at clarification. So I will devote my contribution to the symposium to the topic of the perceived reluctance of the legal services community to cooperate in randomized trials. It goes without saying, but the following thoughts are those of only one member of a larger community.

My understanding is that in the HLAB study, no significant case evaluation occurred prior to randomization. Many of us in legal services view with trepidation the idea of ceding control over case selection to the randomization process. Others have more sanguine views, either because they assume that randomization is already taking place or that it ought to be. For example, in his comments from a few months ago, Dave Hoffman was working under the assumption that to randomize client selection would not change an agency’s representation practices at all, and on that basis, he criticized resistance to randomized control trials as “trying to prevent research from happening.”

The authors of the study are enthusiastic about randomization not only because of its scientific value in statistical research but also because it can help to solve one of the thorniest problems facing legal services programs – the scarcity of resources as compared to the demand. As long as the demand for legal assistance outstrips the supply, Professor Greiner has said, randomization – a roll of the dice or the flip of a coin — is an easy and appropriate way to decide who gets representation and who does not.

I believe it’s erroneous to assume that randomization would not change representation practices, at least in the area of legal services in which I work. I also acknowledge that it is possible, at least theoretically, for all the cases in a randomized control trial to have met the provider’s standards for representation. This would provide some measure of reassurance. However, in one area of law, immigration asylum cases, the authors have concluded that time constraints make such an effort unworkable.

Moreover, to many of us allowing the attorney-client relationship to be the product of random selection rather than conscious choice would mean not only that clients with meritorious claims are being denied representation (the familiar problem of scarcity), but also that clients with unmeritorious claims are receiving representation in the furtherance of scientific experimentation. This latter outcome would almost certainly raise issues of ethics and professional responsibility. To take just one example, a lawyer whose client was assigned blindly rather than selected after at least some case evaluation could easily find himself or herself risking a lack of candor to the tribunal in order to afford the zealous representation the law requires or withdrawing from representation altogether. Neither of those results, it seems to me, would further the interests of justice or the efficient use of legal resources.

Of course we in legal services do not find it easy to make decisions about which clients to represent. Demand swamps supply, making triage (even choosing principles for triage) difficult. It is similarly difficult to allocate resources between individual representation and systemic advocacy. Among the balancing acts in which we in legal services are perpetually engaged is to attempt to meet both these important goals. Shall we spend our resources advocating for a systemic change that is necessarily speculative but potentially of great significance, or shall we focus on the representation of an individual client, a matter of less sweeping significance, but one with a frequently higher probability of success and one that might also shine a useful light on the proper direction of our systemic work? Notwithstanding the difficulties and uncertainties, it seems to me that it is our obligation to choose our clients and our priorities and not to consign the job to a coin flip, even in the face of overwhelming demand. The fact that we will make some wrong choices does not relieve us of the burden of making them.

As is the case with the study’s authors, I regard the improvement of the legal services delivery system as part of our systemic advocacy. If randomized control trials can inform us about what works and what does not and can do so without interfering with our professional responsibility as lawyers, I would welcome them.

My comment supports the article’s emphasis on caution and not overgeneralizing. My focus is on the article’s Question 2: Did an offer of HLAB representation increase the probability that the claimant would prevail? My analysis of the simple frequencies (I have not delved into the regressions and ignore weights) suggests that HLAB attorneys should view the results as modest, but inconclusive, evidence that an offer of representation improves outcomes.

Based on Table 1, page 24, there are 129 No offer observations and 78 Offer observations. Ignoring weights, which I think are said not to make a huge difference, page 26 reports that .76 of claimants who received an offer prevailed [...]]]> Congratulations to the authors on an excellent study that promotes and explores the importance of random assignment.

My comment supports the article’s emphasis on caution and not overgeneralizing. My focus is on the article’s Question 2: Did an offer of HLAB representation increase the probability that the claimant would prevail? My analysis of the simple frequencies (I have not delved into the regressions and ignore weights) suggests that HLAB attorneys should view the results as modest, but inconclusive, evidence that an offer of representation improves outcomes.

Based on Table 1, page 24, there are 129 No offer observations and 78 Offer observations. Ignoring weights, which I think are said not to make a huge difference, page 26 reports that .76 of claimants who received an offer prevailed in their first level appeals, and that .72 of claimants who did not receive an offer prevailed in their first level appeal.

So, those who were offered representation fared better; one measure of which is they did .04/.72 x 100, or 5.6% better. Given the high background (no offer condition) rate of prevailing, the maximum improvement (to 1.00 success rate) is .28/.72 x 100 or 38.9%.  Another measure could be the proportionate reduction in defeat. The no offer group was “defeated” 28% of the time. The offer group was defeated 24% of the time.  The reduction in defeat is .04/.28 x 100 is 14.3%. This measure has the sometimes attractive feature that it can range from 0% to 100%. So by this measure the offeree group did 14% better than the non-offeree group, a modest improvement for the offer condition.

A concern expressed in the paper is that the result is not statistically significant. This raises the question: given the sample size, how likely was it that a statistically significant effect would be detected? Assessing this requires hypothesizing what size effect of an offer would be of societal interest.  Suppose we say that lawyers should do about 10% better and move the win rate from .72 for non offerees to .80 for offerees.  This is an 11.1% improvement by the first measure and a 28.6% improvement by the second measure.  Both strike me as socially meaningful but others might specify different numbers.

We can now pose the question: given the sample size and the effect of specified size, what is the probability of observing a statistically significant effect if one exists?  I use the following Stata command to explore the statistical power of the study:

sampsi .72 .80,n1(129) n2(78), which yields the following output:

Estimated power for two-sample comparison of proportions

Test Ho: p1 = p2, where p1 is the proportion in population 1 and p2 is the proportion in population 2

Assumptions:

alpha =   0.0500  (two-sided)

p1 =   0.7200

p2 =   0.8000

sample size n1 = 129

n2 = 78

n2/n1 = 0.60

Estimated power:

power =   0.1936

A power of 0.19 is too low to conclude that the study was large enough to detect an effect of the specified size at a statistically significant level. If one concluded that an offer of representation did not make a significant difference from this study, there is a good chance the conclusion would be incorrect. To achieve power of about 0.70, one would need a sample four times as large as that in the study. If one thought that smaller effects were meaningful, the sample would be even more undersized.

I think my analysis so far underestimates the benefit of an offer by HLAB attorneys.  Perhaps we can take .72 as a reasonable lower bound on success. Even folks without an offer succeeded at that rate.  The realistic upper bound on success is likely not 1.00.  Some cases simply cannot be won, even by the best lawyer in the world. Perhaps not more than 90% of cases are ever winnable, with the real winnable rate likely somewhere between .8 and .9.  If the winnable rate was .8, then the offer got clients halfway there, from .72 to .76. If the real rate was higher, the offer was less effective but not trivial in size.  At .9, the offer got the clients 22% closer to the ideal. The study just was not large enough to detect much of an effect at a statistically significant level.

So while I agree that the study provides no significant evidence that an offer increases success, my analysis (obviously incomplete) suggests that the study provides no persuasive evidence that an offer does not increase success. The study is inconclusive on this issue because of sample size.

HLAB lawyers should not feel that they have to explain away these results; the results modestly, but inconclusively, support the positive effect of an offer because they are in the right direction in a small study.

In my line of work, I have seen many efforts in the political realm to shut down civil legal services for the poor, and have continually worked to combat such efforts.  In 1996, when the Gingrich Congress barred federally funded legal services lawyers from bringing class actions on behalf of the poor, I left Legal Services for the Elderly in order to finish a lawsuit on behalf of widows and widowers who were suing to compel the United States Treasury to fix its practices for replacing stolen Social Security payments.  When I later moved to the Brennan Center for Justice, I helped [...]]]> David Udell is the Executive Director of the National Center for Access to Justice and a Visiting Professor from Practice at Cardozo Law School.

In my line of work, I have seen many efforts in the political realm to shut down civil legal services for the poor, and have continually worked to combat such efforts.  In 1996, when the Gingrich Congress barred federally funded legal services lawyers from bringing class actions on behalf of the poor, I left Legal Services for the Elderly in order to finish a lawsuit on behalf of widows and widowers who were suing to compel the United States Treasury to fix its practices for replacing stolen Social Security payments.  When I later moved to the Brennan Center for Justice, I helped bring a lawsuit against the rules that barred legal services lawyers from participating in such class actions, I filed another lawsuit against similar rules that barred law school clinic students from bringing environmental justice cases in Louisiana, and I built a Justice Program at the Brennan Center dedicated to countering such attacks on the poor and on their lawyers.

In their March 3, 2011 draft report, What Difference Representation? Offers, Actual Use, and the Need for Randomization (“the Study”), authors D. James Greiner & Cassandra Wolos Pattanyak are right about the importance of developing a solid evidence base – one founded on methodologies that include randomization – to establish what works in ensuring access to justice for people with civil legal cases. They are right again that in the absence of such evidence, both the legal aid community and its critics are accustomed to relying on less solid data.  And they are smart to “caution against both over- and under-generalization of these study results.”  But, unfortunately, the bare exhortation to avoid over- and under-generalization is not sufficient in the highly politicized context of legal services.

While the authors obviously do not have any obligation to arrive at a particular result, they can be expected to recognize a need to avoid statements that have a high probability to mislead, especially in light of the likely inability of much of the Study’s audience to understand the authors’ methodology and findings.  In fact, because of the Study’s novelty and appearance in a non-scientific journal, it will be relied on to analyze situations where it doesn’t apply, and by people who have no background in social science research, plus it will be given disproportionate weight because so few comparable studies exist to judge it against.  It is these factors, in combination with the politicization of legal services, that make it crucial that the authors’ assertions, particularly in the sections most likely to be seen by lay readers (the title and the abstract), do not extend beyond what the findings justify.

So, it is a cause of concern that the Study leads with dramatic headlines and minimizes essential caveats – most significantly, omitting the authors’ own important acknowledgement that the data could not support any useful empirical conclusion about the effect of actual representation (as distinct from an offer of it).  Thus, the title still leads with the phrase “What Difference Representation?,” while the abstract declares the findings “unexpected” and then goes on to state:

• “a service provider’s offer of representation to a claimant had no statistically significant effect on the claimant’s probability of a victory.”
• “the offer of representation inflicted a harm upon such claimants [. . .] with no discernible increase in the probability of a favorable outcome.”
• “within the limits of statistical uncertainty, these claimants would have been better off without the offer of representation.”

By leading with the question “What Difference Representation,” and by employing alarming phrases such as “no … effect,” “inflicted harm,” and “better off without an offer of representation,” the title and abstract guide readers to conclusions that reach beyond the underlying data.  The abstract offers caveats, but only in the form of opaque technical phrases such as “statistically significant,” “no discernible increase in the probability,” and “statistical uncertainty.”  This is an approach that invites misuse.

I do not exaggerate the risk.  An overview of the politics surrounding the federally funded Legal Services Corporation (LSC)), published just two weeks ago in the National Law Journal, describes recurring efforts to defund the nation’s flagship legal services institution.  One piece carries the title, “For LSC, a 30-year funding rollercoaster; Throughout most of its history, the agency has been a political football, periodically the target of massive cuts.”  Another leads with, “Looking for allies in Congress, and finding few.”  Indeed, for decades, we have seen the legal services community compelled to respond to allegations that on investigation prove trumped up or spurious.  A recent example is Disappointing Reporting on Legal Services, in which the Legal Aid Society of the District of Columbia undertakes to rebut an “exposé” circulated on the internet that falsely sought to tar all legal services programs based on a single instance involving a single employee’s misconduct.

But, more specifically, in a Brief submitted this past month to the United States Supreme Court, opponents of legal services cited the “What Difference” study for the proposition that legal representation doesn’t matter, urging “A recent randomized, controlled Harvard study of simple, nonjury litigation found no significant difference in success rates between litigants who were offered legal representation and those who were not.”   In its haste to attack legal representation, the Brief omits to mention that the Study:  i) contains no empirically useful finding regarding the efficacy of actual representation (as distinct from offers of representation), ii) examined administrative advocacy, not court litigation, and, iii) evaluated law students, not lawyers.  Nor does the Brief mention that (as described in greater detail below):  i) the control group for the Study included many people (apparently 49% of the group) who ultimately  received actual representation from “other service providers,” ii) those “other service providers” presumably possessed greater experience and greater training than the law students, and iii) the Study included cases without regard to whether representation was expected to make a difference in their outcomes.  Finally, the Brief omits to mention the Study’s explicit admission that “It would be a mistake to over-generalize the results of our study to conclude that offering free legal assistance is not worth the cost or time, or even that offers of representation make no difference in Massachusetts first-level appeals.”  Study at 47.

But, the Study’s presentation isn’t likely to confuse only the enemies of legal services.  Even as esteemed a thought leader as Ian Ayres omits mention of some the Study’s limitations in an essay he published this past winter, Iatrogenic Legal Assistance, in the on-line forum, Freakonomics.  Describing the Study’s primary finding as “The claimants who were offered representation were no more (or less) likely to win their administrative appeal,” Ayres (who flagged several of the Study’s limitations, including a concern about the small size of the pool of subjects) does not include the authors’ acknowledgment that the Study failed to make any empirically useful finding that actual representation (as distinct from offers of representation) has any effect on a claimant’s probability of a win.  He also makes no mention of the fact that the “other service providers” who represented members of the control group should be presumed to possess deeper experience and training than the law students.  Nor does he mention that the Study included cases without regard to whether representation would be expected to make a difference in their outcomes.  Like the authors of the Supreme Court Brief, Ayres also does not mention the Study’s acknowledgment, that “It would be a mistake to over-generalize the results of our Study to conclude that offering free legal assistance is not worth the cost or time, or even that offers of representation make no difference in Massachusetts first-level appeals.”

Although many of the Study’s limitations were omitted from the Supreme Court Brief and from the Ayres essay, and although virtually all of them are omitted from the Study’s abstract, for the most part they are articulated in the body of the Study:

1. The “mixed control group” problem – Contrary to what is implied by the abstract, the Study compared outcomes obtained by members of a “treatment group,” who received offers of representation from HLAB law students, to outcomes obtained by members of a “control group,” 49% of whom obtained offers of representation (and, actual representation) elsewhere (from advocates with presumably greater experience and greater training, see discussion below), and 51% of whom obtained no offers (and, no representation).  The fact that 49% of the members of the control group received representation potentially upwardly biases how well the control group did (as if in a medical trial, ensuring that 49% of the control group took the same medicine administered to the treatment group), thereby making the impact of HLAB offers of representation more difficult to discern in relation to the full control group.  Indeed, some members of the unrepresented 51% portion of the control group may also have received certain limited forms of legal help, such as “legal advice” or “brief assistance,” further upwardly biasing the performance of the control group.  The Study acknowledges the mixed control group issue (see, e.g., pp. 11, 41), but rejects, as “implausible,” the theory that it prevented  detection of the impact of the HLAB offers of representation (see p. 45).  In fact, the authors’ assertion of implausibility appears unwarranted in light of the “experience  gap” problem (discussed in greater detail below).  But, regardless of whether the authors are persuaded (or not persuaded) that the presence of represented persons in the control group upwardly skewed the results, the abstract should let readers know the facts. To prevent readers from being misled about what was studied, the authors should simply modify the abstract to make clear that:  “the HLAB law student service provider’s offer of representation to a claimant had no statistically significant effect on the claimant’s probability of a victory when compared to a control group in which 49% of the members received representation from other service providers and in which the remainder of the members may have received other forms of assistance.”
2. The experience gap problem – The abstract omits all mention of the experience gap that exists between the HLAB students and the “other service providers” who represented members of the control group.  Although the Study explicitly rejects, as “implausible,” the notion that the HLAB “lawyering” was “low quality,” (see p. 45), “low quality” isn’t the relevant issue.  Rather, the Study fails to acknowledge that the “other service providers” who represented 49% of the members of the control group presumably possessed greater experience and greater training than the HLAB students (some of whom were handling their first case), and that this discrepancy with respect to experience and training may have upwardly biased the performance of the control group, with respect to both the possibility of a clamant obtaining a favorable outcome and the speed with which a favorable outcome is obtained.  This experience gap may thus be expected to conceal the effectiveness of the HLAB students’ performance while highlighting any delay caused by the HLAB students’ performance. To prevent readers from being misled about what was studied, the authors should modify the abstract to make clear that “the HLAB law student service provider’s offer of representation to a claimant had no statistically significant effect on the claimant’s probability of a victory in a study in which a portion of the control group (49%) received representation from other service providers who should be presumed to possess greater experience and training than the HLAB law students, and in which the remainder of the members may have received other forms of assistance.”
3. The “screening for merit” problem – Another limitation that is missing  from the abstract is that subjects were selected for inclusion in the Study without regard to whether advocacy would be expected to make a difference in the probability of their obtaining a victory. The authors describe this problem as worthy of further inquiry (which they are pursuing) and explicitly acknowledge that:  “One might object, however, that [the current study’s] design cannot capture the effect of representation because one of the tasks attorneys, particularly legal aid attorneys, perform is to choose which cases will benefit from representation, and the randomizer prevents them from exercising their judgment in this manner.”  Study at 72.  To prevent readers from being misled about what was studied, the authors should modify the abstract to make clear that:  “Subjects were included in the Study without investigation as to whether their cases could benefit from representation by an advocate.”
4. The “what is statistically significant” problem – The abstract relies on technical concepts of statistical significance, statistical uncertainty, and discernibility, but contains no definition of these concepts, thereby giving no clue to the lay reader to counteract the abstract’s direct message that a service provider’s offer of representation is ineffective, harmful, and necessary to avoid.  It is therefore interesting to see, deeper in the Study, the following clarification:

This finding [of “no statistically significant effect on the probability that a claimant would prevail”] does not mean that we know that the HLAB offer had no positive effect on a claimant’s probability of success; we can say, however, that the [sic] any such effect is unlikely to be very large (or the data probably would have shown it).

Study, at 8; see also Study at 45 (repeating clarification). Moreover, the Study explicitly acknowledges that the data are “useless” with respect to the question of whether HLAB “actual representations” increase a claimant’s probability of a victory (as distinct from HLAB “offers”).  See Study, at 43.  The Study also explicitly acknowledges, as noted above, that “It would be a mistake to overgeneralize the results of our study to conclude that offering free legal assistance is not worth the cost or time, or even that offers of representation make no difference in Massachusetts first-level appeals.”  Study at 47.  To prevent readers from being misled about the Study’s findings, the authors should modify the abstract to  make clear that:  “The Study does not find ‘that the HLAB offer had no positive effect’ or ‘that offers of representation make no difference,’ nor does it contain any useful findings regarding the possible effect of actual representation.”

As I hope is evident, I direct my comments primarily to issues concerning the accuracy of the presentation of the authors’ findings in the title and in the abstract, and leave issues concerning the accuracy of the findings themselves to the empirical experts in this on-line symposium.  I commend the authors for tackling very important questions, highlighting randomization methodology, acknowledging limitations on their findings, and urging readers neither to over-or under-generalize their findings.  But in light of the politicization of legal services, and as the Brief and the Ayres column make plain, readers will tend to overlook the Study’s limitations, including those acknowledged by the authors themselves in the body of the Study.  Of course the authors are not entirely accountable for choices others may make about how to use their Study.  But, nor would it be responsible for the authors to decline to take easy corrective steps to ensure that their title and abstract describe the Study for what it is rather than for what it is not.  One subject is beyond dispute:  “What Difference Representation?” poses a more challenging problem than “What Difference Presentation?”

It is important with any study; however, to know what it concludes and what it does not.  For instance, and most importantly, the study concedes on page 43 that it could draw no conclusions about the effect on outcome for claimants actually receiving representation, as opposed [...]]]> I am the Executive Director of Greater Boston Legal Services, the primary provider of civil legal services to poor people in the greater Boston area.  My program and I have a great stake in assuring that our limited resources are used where they can be most effective.  Indeed we are participating with Professor Greiner in a study of the impact of our staff attorneys’ representation in defense of eviction cases.  My comments refer to the draft dated February 12, 2011.

It is important with any study; however, to know what it concludes and what it does not.  For instance, and most importantly, the study concedes on page 43 that it could draw no conclusions about the effect on outcome for claimants actually receiving representation, as opposed to just an offer of representation.  Thus, this study should be recognized for what it is: a limited analysis of the somewhat abstract concept of “offering” assistance.  Indeed, the study wisely cautions against drawing any conclusions from the study about the usefulness of free legal assistance or even about the usefulness of offers of representation in unemployment cases in general (page 47).

I feel some changes are necessary to avoid much confusion about (and misuse of) this study’s conclusions (or lack thereof) as to the effect of representation itself, as opposed to just the offer.  For instance, given that this study’s principal conclusions are about an offer of representation and not actual representation, a more accurate title to this study would be, “What Difference an Offer of Representation?”  And the very first sentence of the Introduction on page 5 currently reads, “Particularly with respect to low-income clients in civil cases, how much of a difference does legal representation make?”  It is only a footnote that explains the study looks at offers as well as effect, and much later in the study (page 32) that no conclusions were reached at all as to the effect of representation.  Similarly, the conclusion (“Where Do We Go From Here?”) states, “the present study primarily concerned representation effects on legal outcomes affecting the potential client’s pecuniary interests.”

I am concerned also that the results reported in the study with respect to offers of representation by HLAB are misleading at best and of little utility at worst.  This is because nearly half of the control group were represented by counsel and, more significantly, probably that many and perhaps more in the control group got an offer of free representation from my program or another providing free legal services in unemployment cases.  To make an analogy to the medical world, suppose there was a Pfizer drug trial where 50% of Pfizer’s control group were offered the exact same medication from Merck.  Wouldn’t that cast serious doubt on the outcome of the study?  There is no mention of this 49% in either the abstract or introduction which unfortunately are all many readers will read.

The article purports to study the impact of an “offer” of representation rather than the impact of actual representation.  One implication of studying offers is that the control group i.e. the group that did not get an offer, may, and probably will, contain some people who will end up being represented because they will have found other means to obtain representation.  The article refers to them as “Go-Getters” (text accompanying footnotes 133 to 137).

The problem in the study with the control group is that during the study period in the same geographical area several other organizations including my own were also making offers of free representation to those in the control group.  This was built into program design.  “If the randomization was not to offer, the student attorney so informed the claimant by telephone and provided her with names and telephone numbers of other legal services provides in the area who might take her case.”  (text accompanying footnotes 103 to 104)  In the study 49% of those in the control group ended up being represented.  (text accompanying footnotes 134-136).

The article characterizes those who in the control group who obtained representation as “Go-Getters” and with no empirical evidence the authors say:

Our instincts are that the process of finding an attorney once turned down by HLAB required effort, energy articulateness, and persistence from a claimant.  (text accompanying footnotes 136-137).

This could be plausible for those in the control who found a private attorney.  But in Boston, there were other organizations, primarily my own, also offering free representation.  And everyone in the control group was given our phone number and phone numbers of other free providers.

Participants got into the study by making a phone call to HLAB.  (text accompanying footnotes 101-102)  HLAB is simply one of several organizations listed in the information provided by the Massachusetts Division of Unemployment Assistance to all unemployment claimants participating in hearings concerning their eligibility for unemployment insurance benefits.  So this was a group with the ability and desire to make a phone call in order to get representation.  All that was needed “to find an attorney once turned down by HLAB” was to make a second phone call to another organization listed by the Division or provided by the HLAB student.  This hardly seems to require a greatly different amount of “effort, energy and articulateness and persistence.”

Our staff reports that in at least some cases, the HLAB student attorneys, concerned that persons in the control group who especially needed assistance would end up without representation, actually sent GBLS the names and phone numbers of the persons randomized out.  Our staff then called and in each such instance both offered and provided them representation.  So, in at least some cases, those in the control group got second offers of free representation with no extra effort on their part.  For those making phone calls in the summer, which were apparently a very small part of the group studied, our program had a special line set up to represent people in UI cases, so all that was needed was  one phone call.

Most callers seeking representation, would be screened by our sister organization LARC (the Legal Advocacy and Resource Center).  In a slightly different context the article discusses the difficulties in getting through to LARC (text accompanying footnotes 241-245) To be sure there are times when it is hard to get through to LARC.  At other times, there is no significant delay.  LARC annually handles 12,000 phone calls a year, and last year 1000 of them concerned employment matters.  So it is equally plausible to say that that for those in the control group, it was not significantly harder to get through to LARC than for them to have made their initial phone call to HLAB.  This is particularly so since LARC is open after working hours while HLAB is only open during working hours and during the summers and school vacation has even more limited hours.  Moreover, as described above, individuals are provided with the telephone numbers of a range of free providers by the Division of Unemployment Assistance.  At the least, there is no evidence that folks we represent who got through the LARC line are different from those in the study group.

In continuing to try to minimize the 49% figure, the study states, “The second hypothesis we do not credit is the possibility that that the control group had too many claimants who ended up finding representation so that there is an insufficient contrast…  [between the two groups].”  (text accompanying footnotes 149-150).

But the study does not set out what difference there might have been the between represented and unrepresented.  If the represented in the control group got better outcomes than the group as a whole, then removing them from the group would result in the control group showing worse outcomes.  And if, as I suggest, the represented group were not “go getters” but simply people who did get an offer after either doing nothing or making one more phone call, then an offer of representation from HLAB or another free service does make a difference.

Nor does the article study the final “outcome” of any of the cases measured.  In the treatment group, HLAB’s victory in cases ensured that the claimant obtained unemployment benefits.  If the claimant lost HLAB would continue to represent these claimants and might end with a favorable outcome.  Moreover, even if claimants “win” a hearing, if there are other “issues” needing to be adjudicated, such as immigration status, whether or not the claimant is able and available for work, etc. further representation is often needed to secure benefits.  In contrast, for the control group, we have no data on what happened after the initial win or loss at the hearing for the represented or unrepresented group.  If the study purports ultimately to measure what difference representation makes, the final outcome for the claimant is important to know.

To measure truly the impact of an offer of free representation, I suggest the study needs to remove from the control group those who also had an offer of free representation or do some statistical taking into account of those in the control group who also had an offer of free representation.  (This is different from removing everyone in the control group who got representation which the article with some reason rejects (text accompanying footnotes 137-138)).  For those who got a private attorney (I suspect few, if any) the go-getter hypothesis is at least plausible.)  Without removing the impact of those in the control group who had an offer for free representation, the conclusion of the article are not warranted.

The possibility that the offer of representation made a difference is strengthened by the fact that the win rate in the control group was higher than the state average.  When claimants in the control group appealed they won 65% of the time compared to the state average of 47%.  When claimants in the control defended they won 83% of the time compared to the state average of 75% (text accompanying footnote 150)  The article  gives three hypotheses to explain this(text accompanying footnotes 150-155) but never suggests what to most lawyers would be the obvious conclusion, 49% of the were represented by counsel.  I do not have the data, but it seems unlikely that in the state as whole close to 75% of the claimants in the system are represented (49% of the control group and virtually all of the studied group).

Why is this so important to me?  I believe that the title and abstract, as well as the control group issue have the possibility of making the study and the article very misleading.  If this were simply a case of a purely academic study then I would be more than happy to leave it to the academics and use my time to provide legal services.  Further studies might clarify things and get us all in agreement.  But further studies are way in the distance.  Even our eviction defense study with Profess Greiner is months away from being finished.  Meanwhile, despite all the careful caveats in the article, it is being cited by others for very broad propositions.  In the Turner case now before the U.S. Supreme Court this article is cited for the proposition that lawyers are unlikely to make a significant difference in civil contempt hearings.

If the article is misleading, or leads to a misreading, it could do real harm to people who do need assistance of counsel for some basic human needs.

I am delighted to announce that Concurring Opinions will be hosting a symposium next Monday and Tuesday on What Difference Representation? Offers, Actual Use, and the Need for Randomization, the forthcoming Yale Law Journal article by Jim Greiner and Cassandra Wolos Pattanayak.  [Update: You can read all posts in the symposium by clicking on this link.]  As you may recall, What Difference has already caused quite a stir in the clinical and legal aid communities. Given our shared interest in questions of empirical methodology, and Jaya’s background in clinical legal services, we decided that bringing that debate to CoOp would be an excellent use of our time and energy. Here’s the (revised) abstract) – though you should download the [...]]]>

Should Law School Clinics Select Clients by Roulette?

I am delighted to announce that Concurring Opinions will be hosting a symposium next Monday and Tuesday on What Difference Representation? Offers, Actual Use, and the Need for Randomization, the forthcoming Yale Law Journal article by Jim Greiner and Cassandra Wolos Pattanayak.  [Update: You can read all posts in the symposium by clicking on this link.]  As you may recall, What Difference has already caused quite a stir in the clinical and legal aid communities. Given our shared interest in questions of empirical methodology, and Jaya’s background in clinical legal services, we decided that bringing that debate to CoOp would be an excellent use of our time and energy. Here’s the (revised) abstract) – though you should download the article if you haven’t already:

“We report the results of the first in a series of randomized control trials designed to measure the effect of an offer of, and the actual use of, legal representation. The results are unexpected. In the context of administrative litigation to determine eligibility for unemployment benefits, a service provider’s offer of representation to a claimant had no statistically significant effect on the claimant’s probability of a victory, but the offer caused a delay in the proceeding. Because a substantial percentage of the provider’s client base consisted of claimants who were initially denied benefits but who would later have that initial denial reversed as a result of the litigation, the offer of representation inflicted a harm upon such claimants in the form of an additional waiting time for benefits to begin, this with no discernible increase in the probability of a favorable outcome. In other words, within the limits of statistical uncertainty, these claimants would have been better off without the offer of representation. The size of the delay (around two weeks, depending on how measured) was not large in absolute terms, and would have been negligible in many other legal settings, but was relevant in the context of this particular administrative and legal framework, one in which speed has remained a special concern for decades. Moreover, in a small number of cases with a certain profile, the delay caused the unemployment system to continue paying benefits erroneously for a longer period of time, potentially imposing costs on the financing of the unemployment system. We were also able to verify a delay effect due to the actual use of (as opposed to an offer of) representation; we could come to no firm conclusion on the effect of actual use of representation on win/loss.

We hypothesize three potential explanations for our findings (and acknowledge that others are possible). First, it is possible that the client base that reached out to the service provider (and thus was subject to randomization) was a specialized subset of unemployment claimants, a subset that did not actually need legal assistance. This theory suggests special attention to provider intake systems. Second, it is possible that the administrative adjudicatory system at issue, with its semi-inquisitorial style of judging, is pro se friendly. Third, it is possible that the subject matter in dispute in these cases is less legally, factually, or procedurally complex than in other settings.

We caution against both over- and under-generalization of these study results. We use these results as a springboard for a comprehensive review of the quantitative literature on the effect of representation in civil proceedings. We find that this literature provides virtually no credible information, excepting the results of two randomized evaluations occurring in different legal contexts and separated by over three decades. We conclude by advocating for, and describing challenges associated with, a large program of randomized evaluation of the provision of representation, particularly by legal services providers.”

We have assembled a terrific group of symposiasts, mixing clinicians, academic empiricists and practitioners. Besides Jim, Cassandra, Jaya and me,  the group includes twelve contributors, lauded in detail after the jump:

Jeanne Charn is Director of the Bellow Sacks Access to Civil Legal Services Project and Senior Lecturer in Law at Harvard Law School, teaching clinical courses on civil advocacy, delivery of legal services, professional responsibility and housing law and policy.  In 2006, she stepped down after 27 years as Director of the WilmerHale Legal Services Center at Harvard Law School.  During law school, Jeanne was a student practitioner at Community Legal Assistance Office (CLAO), one of the first O.E.O funded legal service projects. Upon graduation from law school in 1970, Jeanne was a staff attorney at Massachusetts Law Reform Institute representing statewide and local public housing tenant groups and providing training and support for legal services in the state.  In 1973, Jeanne was appointed Assistant Dean for Clinical Programs at Harvard Law School. She arranged for and monitored the educational quality of all course related student placements and worked with Professor Gary Bellow to develop Harvard’s clinical program. In 1978, Gary and Jeanne conceived of a Harvard Law School supported “teaching law office” similar to the teaching hospital in medicine. The predecessor of the WimerHale Legal Services Center opened in 1979.  Jeanne received her B.A. from the University of Michigan in 1967, and her J.D. from Harvard Law School in 1970.

Ted Eisenberg is a Professor of Law at Cornell Law School,  where he has emerged in recent years as one of the foremost authorities on the use of empirical analysis in legal scholarship. After his graduation from University of Pennsylvania Law School, Eisenberg clerked for both the District of Columbia Circuit of the U.S. Court of Appeals, and Chief Justice Earl Warren of the U.S. Supreme Court. After three years in private practice, Professor Eisenberg began teaching at UCLA. A groundbreaking scholar in the areas of bankruptcy, civil rights, and the death penalty, Eisenberg has used innovative statistical methodology to shed light on such diverse subjects as punitive damages, victim impact evidence, capital juries, bias for and against litigants, and chances of success on appeal. He is the founder of the Journal of Empirical Legal Studies and a Fellow of the American Academy of Arts and Sciences. He currently teaches bankruptcy and debtor-creditor law, constitutional law, and federal income taxation.

Steve Eppler-Epstein is the Executive Director of Connecticut Legal Services (CLS), whose core mission is to provide comprehensive civil legal help to low-income people throughout most of Connecticut, partnering with other members of the legal aid network to cover the state.  Steve has worked with CLS since 1984, when he joined them as a staff attorney.  His work has included individual and class-action client service on domestic violence and public benefit issues; legislative advocacy; and community legal education and writing.  In 1995 Mr. Eppler-Epstein was hired to be the Deputy Director of Connecticut Legal Services, and in 2007 he was selected to be the Executive Director.  As a leader in legal services, Mr. Eppler-Epstein’s priorities are keeping program priorities fresh and relevant in light of changes in the client population; empowering staff to explore change and to pursue client service through the most effective means available; and enhancing the public understanding of CLS’ mission of justice so as to maximize the funding resources available to serve low-income people in crisis.  In addition to his work at CLS, Mr. Eppler-Epstein is the Chair of the Board of the Boston-based Center for Legal Aid Education, where he has been very active in the development and deployment of CLAE’s Leadership Institute.  He has served on the Board of Directors of the Connecticut Coalition Against Domestic Violence, and is co-Chair of the Advisory Council to the Judicial Branch Office of Victim Services.  His recognitions include the Governor’s Victim Services Award, the Connecticut Bar Foundation’s Legal Services Leadership Award, and the Connecticut Bar Association’s Charles J. Parker Legal Services award.

Michael Heise is a Professor of Law at Cornell Law School, where he specializes in empirical legal scholarship and bridging empirical methodologies, legal theory, and policy analysis.  His empirical work focuses on a wide array of public and private law topics.  He is the author of numerous articles and has co-edited the Journal of Empirical Legal Studies since 2005.  Michael is also a Founding Director of the Society for Empirical Legal Studies and a contributing editor of the ELS Blog.  Prior to joining the Cornell Law School faculty Michael taught at Case Western and Indiana University.  He is a graduate of Stanford (A.B.), Chicago (J.D.), and Northwestern (Ph.D.).

Andrew D. Martin is Professor of Political Science and Professor of Law at Washington University in St. Louis. He is the Chair of the Political Science Department in Arts & Sciences, and serves as the founding Director of the Center for Empirical Research in the Law in the School of Law. Professor Martin is a Resident Fellow of the Center in Political Economy, and is a core faculty member of the Center for Applied Statistics.  Professor Martin’s expertise is in the study of judicial decisionmaking, with special emphasis on the Supreme Court and the lower federal courts. He also works extensively in the field of political methodology and applied statistics. He has published in a number of prominent law reviews and leading social science and applied statistics journals, and is a frequent presenter at conferences and workshops throughout the country. His research has been supported by the National Science Foundation, the MacArthur Foundation, and the National Institutes of Health.  Professor Martin is the principal investigator of the The Judicial Elections Data Initiative, a study of litigation processes in employment discrimination suits initiated by the EEOC, and a cross-national study that examines decisionmaking in constitutional courts around the globe. He is also a collaborator on The Supreme Court Database project, and a contributor to The Discography. With his collaborator Kevin M. Quinn (UC Berkeley), Professor Martin developed the Martin-Quinn Scores that are widely used to measure ideology on the U.S. Supreme Court.  Professor Martin teaches courses in the law school on judicial decisionmaking and on social science and statistics for lawyers, in addition to graduate and undergraduate courses in political methodology in Arts & Sciences. He also regularly offers workshops on social science research methods for judges, prosecutors, and legal academics.

Margaret Monsell joined the Massachusetts Law Reform Institute in January 1999.  She practices employment law in areas including unemployment insurance, the earned income tax credit program and low-wage worker protections, and she is a co-author of the Unemployment Advocacy Guide. Prior to joining MLRI, she worked as a law clerk at the Massachusetts Appeals Court and the Massachusetts Supreme Judicial Court, as an Assistant Attorney General in the state Attorney General’s office, and as General Counsel for the Senate Committee on Ways and Means in the Massachusetts Legislature. She received a Ph.D. in Theological and Religious Studies from Boston University in 1988, and she graduated with honors from Boston College Law School in 1990.

Kevin Quinn is a Professor of Law at the UC Berkeley School of Law. Prior to  joining the Boalt faculty in 2009 he was associate professor of Government at Harvard University. Professor Quinn holds a Ph.D in Political Science from Washington University in St. Louis. He has written extensively on judicial decision making and statistical methodology. His research has been supported by the National Science Foundation and has appeared in, among other outlets, the Columbia Law Review, the Stanford Law Review, the University of Pennsylvania Law Review, the American Journal of Political Science, the Quarterly Journal of Political Science, the Journal of the American Statistical Association, and the Journal of the Royal Statistical Society Series A. Professor Quinn is a three time winner of the Gosnell Prize for excellence in political methodology. He is currently an Associate Editor for the Journal of the American Statistical Association.

Bob Sable has served as the Executive Director of Greater Boston Legal Service since 1991, and he has spent more than 40 years as a legal services lawyer.  He has won many awards for his service, including the Massachusetts Bar Association’s Legal Services Award and the Boston Bar Association’s John G. Brooks Legal Services Award.  Some would say that Bob Sable’s proudest moment occurred in 1994, when with the help of the Boston Bar Association and the Volunteer Lawyers Project, he gave up more than $2.1 million in federal funding for GBLS funding rather than accept new Legal Services Corporation rules that would have restricted the types of representation and people who could be represented by GBLS.  After giving up the LSC funding, GBLS grew from a budget of $6 million to $14 million expanding its base of support to include everyone from individuals, corporations, law firms foundations, state and federal government to the United Nations enabling GBLS to serve more than 15,000 individuals and families as well as continuing to bring major cases.  A graduate of Harvard College, Bob Sable served as a Peace Corps Volunteer in Liberia before going on to Yale Law School, where he was Chair of the Student Legal Services Corporation.

Rebecca L. Sandefur is a senior research fellow at the American Bar Foundation, which she joined in September 2010 to lead the Foundation’s new access to justice research initiative. Her research has two strands: one focuses on inequality and civil justice and the other focuses on lawyers’ work and careers. A nationally and internationally recognized expert on legal professions, access to justice and public experience with civil justice problems, she is a frequent speaker at scholarly conferences and meetings of practitioners. Her scholarship has appeared in numerous edited volumes and law reviews as well as outlets such as American Sociological Review, Annual Review of Sociology, and Law and Society Review. She is the editor of the multidisciplinary volume Access to Justice (Emerald/JAI Press, 2009) and an author of Urban Lawyers: The New Social Structure of the Bar (with John P. Heinz, Robert L. Nelson and Edward O. Laumann, University of Chicago Press, 2005). She has served in elected governance positions with the American Sociological Association, the Law and Society Association, and the Pacific Sociological Association, as well as on the editorial boards of the American Journal of Sociology, Law and Social Inquiry and Law and Society Review. Her public service includes work on the Right to Counsel Committee of the California Access to Justice Commission and on the Research Advisory Board of the Civil Right to Counsel Leadership and Support Initiative. Before joining the American Bar Foundation, Sandefur served on the faculty of Stanford University for nine years after receiving her PhD in sociology from the University of Chicago. In addition to being Senior Research Social Scientist at the ABF, she is Consulting Assistant Professor of Sociology at Stanford University.

Jeffrey Selbin is a Clinical Visiting Professor of Law at Yale Law School and a Clinical Professor of Law at the University of California, Berkeley, School of Law (Boalt Hall). He is also Faculty Director of the East Bay Community Law Center, Boalt Hall’s community-based poverty law clinic. He founded EBCLC’s HIV/AIDS Law Project in 1990 and served as the organization’s Executive Director from 2002 through 2006. His research interests include clinical education, anti-poverty lawyering, and community lawyering, with an emphasis on evidence-based approaches. He is a graduate of the University of Michigan, L’Institut d’Etudes Politiques, and Harvard Law School.

David Udell is Executive Director of the National Center for Access to Justice, is also a Visiting Professor from Practice at Cardozo Law School. He was founding director of the Justice Program of the Brennan Center for Justice from September 1997 through August 2010, and earlier served as a Senior Attorney at Legal Services for the Elderly and as a managing attorney at MFY Legal Services. He has served in leadership roles in the national civil right to counsel movement and the national indigent defense reform movement, and has coordinated national advocacy initiatives to help strengthen the Legal Services Corporation. Mr. Udell has taught as an adjunct professor at New York University School of Law and at Fordham Law School, and has served on the Pro Bono and Legal Services Committee of the Association of the Bar of the City of New York. He is currently a member of the Advisory Board to the Justice Center of the New York County Lawyers’ Association. He is a 1982 graduate of New York University School of Law.

Richard Zorza is an attorney and independent consultant who has worked for the past fifteen years on issues of access to justice.  He is the coordinator of the national Self Represented Litigation Network, see www.selfhelpsupport.org, and has acted as a consultant to the Harvard Law School Bellow-Sacks Project on the Future of Access to Civil Justice, www.bellowsacks.org, and works in support of the national LawHelp network of access to justice websites, www.lawhelp.org.His book, The Self-Help Friendly Court: Designed from the Ground Up to Work for People Without Lawyers, was published by the National Center for State Courts in 2002.   His article The Disconnect Between the Requirements of Judicial Neutrality and Those of the Appearance of Neutrality when Parties Appear Pro Se: Causes, Solutions, Recommendations, and Implications, 17 Georgetown Journal of Legal Ethics, 423 (2004) is widely used to define the structure of thought on the topic.  He coordinated the National Judicial Conference on Self-Represented Litigation held at Harvard Law School in November of 2007, the launching conference of the Court Leadership Package on Self Represented Litigation, in the fall of 2008, and a national confernce on Public Libraries and Access to Justice in January of 2010.  He is the recipient of the 2008 American Judicature Society Kate Sampson Access to Justice Award. He lives in Washington DC, and is in partnership with his wife Joan.  Additional information and publications are available on his website, www.zorza.net. He blogs at www.accesstojustice.net.

Volume 58, Issue 3 (February 2011)

Articles

Comments

“We recently became aware that HLAB President Rachel Lauter and HLAB Faculty Director David Grossman had written an email to the clinical listserve addressing our paper “What Difference Representation?”.  The email has been posted to various locations in the blogosphere.  Because the email expresses criticisms of the paper that we also have received from one or [...]]]> Jim Greiner and Cassandra Wolos Pattanayak have emailed me a reply to the Harvard Legal Aid Bureau’s comment on What Difference Representation. Since the topic has been the subject of several posts here, as well as some off-line communication from interested readers, I figured that I owed Greiner/Pattanayak a public space for reply.  It consists of a bit of introductory text, and a longer (9-page) paper.

“We recently became aware that HLAB President Rachel Lauter and HLAB Faculty Director David Grossman had written an email to the clinical listserve addressing our paper “What Difference Representation?”.  The email has been posted to various locations in the blogosphere.  Because the email expresses criticisms of the paper that we also have received from one or two other sources, we thought we would take the opportunity the email presented to clarify certain issues.  For example, President Lautner and Professor Grossman echo reactions we have received from another legal aid provider when they say that our study produced “only limited information,” and that more (and more useful) information would be available if we would just analyze the data properly.  We explain here that the analysis the email (and one or two other legal services providers) have advocated is statistically invalid, and that in any event the data required for it do not presently exist and cannot at this time be ethically collected.  As ought to be clear by now, we have the greatest respect for the students of HLAB, including President Lautner, and HLAB’s clinical faculty, including Professor Grossman.  We are using President Lautner and Professor Grossman’s email as a convenient foil representative of a few other comments we have received.

The substance of our response can be captured in the answers to two questions.

1.  Why study the effect of offers of HLAB representation?  All agree that the effect of actual use of representation is interesting, although as we will explain, perhaps less so than one might think at first.  But why study the effect of HLAB offers?

2.  Why not compare those who got offers from any source, not just HLAB, to those who did not get any such offers?}  This is what President Lauter, Professor Grossman, and a few others have suggested.  Why not make this comparison?

We also answer one final question:

3.  So how can we find out about the effect of offers from other service providers?”

To read the full response, click here.