Randomization Uber Alles?
posted by Dave Hoffman
Jim and Cassandra write:
“To Dave, we say that our enthusiasm for randomized studies is high, but perhaps not high enough to consider a duty to randomize among law school clinics or among legal services providers. We provided an example in the paper of practice in which randomization was inappropriate because collecting outcomes might have exposed study subjects to deportation proceedings. We also highlighted in the paper that in the case of a practice (including possibly a law school clinic) that focuses principally on systemic change, randomization of that practice is not constructive. Instead, what should be done is a series of randomized studies of an alternative service provider’s practice in that same adjudicatory system; these alternative provider studies can help to assess whether the first provider’s efforts at systemic change have been successful.”
I meant to cabin my argument to law school clinics. And I do understand that there may be very rare cases where collecting outcomes will hurt clients (such as deportation). But what about a clinic that focuses on “systemic change.” Let’s assume that subsidizing such a clinic would be a good thing for a law school to do (or, put it another way, we think it is a good idea for current law students to incur more debt so that society gets the benefit of the clinics’ social agitation). Obviously, randomization of client outcomes would be a terrible fit for measuring the success of such a clinic. It would be precisely the kind of lamppost/data problem that Brian Leiter thinks characterizes much empirical work.
But that doesn’t mean that randomization couldn’t be useful in measuring other kinds of clinic outcomes. What about randomization in the allocation of law student “employees” to the clinic as a way to measure student satisfaction in the “learning outcomes“? Or randomization of intake and utilizing different client contact techniques as a way of measuring client satisfaction with their representation (or feelings about the legitimacy of the system?) One thing that the commentators in this symposium have tried to emphasize is that winning & losing aren’t the only outputs of the market for indigent legal services. Controlled study of the actors in the system needn’t be constrained in the way that Jim and Cassandra’s reply to my modest proposal to mandate randomization suggest.
March 29, 2011 at 1:23 pm
Posted in: Empirical Analysis of Law, Law Rev (Yale), Law School (Scholarship), Law School (Teaching), Symposium (What Difference Representation)
Print This Post
Responses (1)
Jim Greiner - March 29, 2011 at 2:56 pm
Hi, Dave, thanks for writing again!
I see now we may have been addressing different subjects. Several aspects of agreement: Cassandra and I very much agree that non-pecuniary outcomes (outcomes other than win/loss and delay) are important, and we tried to speak about that in the paper; surveys can be a key component of gathering information of this kind. We agree that randomized evaluation of educational programs is a good idea in principle, subject to all of the usual concerns about such things. We also agree that randomizing methods of attempting client contact can be useful, again depending on the usual concerns and details. One thing this proposal highlights is that some legal services provides depend to some extent on client-initiated intake systems, with outreach to the community aimed at reaching an entire community (not to specific persons). Thus, figuring out what to randomize in outreach and intake could be a challenge (again, a challenge worth confronting, but a challenge).
Speaking for myself only at this point, with respect to randomizing allocation of law student “employees,” it depends on context. For example, if the student clinic is contemplating two different models of doing its work, and it will eventually choose one or the other as its operating model for all students, a randomized evaluation might make a great deal of sense.
In contrast, if the student clinic has two different models of doing its work, and it plans on maintaining both, randomizing students into one or the other might be counterproductive. I’m thinking here of HLAB’s practice: students usually specialize in either housing or family law. Randomizing students to housing or family law might make little sense because we WANT a selection effect here; that is, we may want students to choose the area of law of greatest interest to them, because we may hypothesize that by doing so we will get students in the family law unit that are more excited about (and who will work harder in) the family law unit.
I could be wrong, but my guess is that we’re not disagreeing here, just making sure that we’re talking about the same ideas.
Jim
Leave a Reply