« Drafting a Group Blog Operating Agreement | Main | Regulating Private Military Companies »
October 10, 2007
Bar Passage & Accreditation: The "Neutral" Case Against Standards
Back in August, the ABA withdrew proposed interpretive standard 301-6, which would have de-accredited schools that didn't graduate students who passed their state bar at certain rates:
Under the first option, a school would have to show that in three or more of the most recent five years, in the jurisdiction in which the largest proportion of the school’s graduates take the bar exam for the first time, they pass the exam above, at or no more than 10 points below the first-time bar passage rates for graduates of ABA-approved law schools taking the bar examination in that jurisdiction during the relevant year. For schools from which more than 20 percent of graduates take their first bar examination in a jurisdiction other than the primary one, the schools also would be required to demonstrate that at least 70 percent of those students passed their bar examination over the two most recent bar exams.The major critiques I saw of 301-6 focused on its alleged discriminatory effects:"all of the five ABA accredited law schools with the highest African-American enrollment (Howard, Southern, Texas Southern, North Carolina Central, and District of Columbia) would fail to meet the proposed interpretation."I recently saw an interesting paper by Gary Rosin titled Benchmarking the Bar: No Unity in Difference Scores that seems to provide a race-neutral argument against the standard. From the abstract:Schools unable to satisfy the first alternative still could comply by demonstrating that 80 percent of all their graduates who take a bar examination anywhere in the country pass a bar examination within three sittings of the exam within three years of graduation.
Under ABA proposed Interpretation 301-6, the primary benchmark used to measure the adequacy of a law-school's academic program would be the amount by which is “local” Bar passage rate for first-takers differs from the overall passage rate for all first-takers from ABA-approved law schools. The study used generalized linear modeling as a method to compare Bar “difference scores” of ABA-approved law-schools in two states, New York and California. The study found that Bar difference scores in California were significantly more sensitive to changes in law-school relative LSAT scores than were Bar difference scores in New York. Bar difference scores - subtracting the “local” overall ABA Bar passage rate - do not fully adjust for variations in state grading practices, especially differences in minimum passing scores (“cut scores”) .That is, because of state-to-state variation in slope of the bar passage curve, a standard that uses that curve as a predominant factor in accreditation decisions will have disparate effects. This seemed like an neat finding, but I wondered whether it is possible that the ABA (if it has to be the agency doing this) could correct for this slope problem using a weighting technique of some kind. I asked Gary, and he has kindly permitted me to quote his answer, after the jump.
"The most certain way to handle it would be to calculate a "reference" LSAT difference score that corresponds to the national Bar difference score that is adopted. You would then use that reference to calculate each state Bar difference score.This is an interesting problem, which deserves more discussion. (There was a good, but technical, comment thread on the ELS blog about Gary's earlier paper.) I imagine that the Bar will be back at some point soon with a new proposed standard that will ameliorate the politically hot diversity objections. But Gary's objection strikes me as deeper, and potentially more fatal to the project of using Bar passage as a good accreditation measure. What do you think?
1. Calculate an LSAT-only national model (see the earlier paper, Unpacking the Bar).
2. Use that model to calculate the LSATs that correspond to (i) the national average for ABA first-takers and the BPR that gives the adopted lowest acceptable Bar difference score.
3. Use the resulting LSAT difference score as the standard of reference.
4. Calculate an LSAT + cut score national model.
5. For each state, use the second model to calculate the LSAT that corresponds to the average for ABA first-takers in that state.
6. Subtract the reference LSAT difference score from that "average" LSAT to find the lower LSAT bound.
7. Use the second model to find the Bar passage rate that corresponds to that LSAT. The difference between that BPR and the state ABA average would be the lower bound for the Bar difference score in that State.
The reason that you can't just use the slopes is that the slopes fall at an increasing rate as LSAT falls. So what you want is the slope of the chord between the reference (LSAT, Bar) pairs. Because the slopes don't change a constant rate, you can't depend on the slope of the reference chord.
Another problem is the size of the 95% confidence intervals for a law school's BPR, which is
+/- 1.96 * SQRT[Bar*(1-Bar)/N] (N is the number of the school's
candidates)
Again, the size of the confidence interval increases both as BPR falls towards 50%, and as N falls. For the schools in the study, the school confidence intervals ranged from +/- 2.7% to +/- 15.1%. That's more than enough to swamp a 10% difference score!"
Posted by Dave Hoffman at October 10, 2007 06:26 PM
Trackback Pings
TrackBack URL for this entry:
http://www.concurringopinions.com/movabletype/mt-tb.cgi/2564.
Comments
Rest assured that 301-6 will rise in some form at the Section Council meeting in February 2008. The Department of Education is leaning on the ABA to adopt credible outcome measures.
Apart from the debate on structuring an outcome measure based on Bar passage, there is a more fundamental question: Is "No Child Left Behind" thinking appropriate to the education of professionals?
More to come...
Gary
Posted by: Gary Rosin at October 11, 2007 01:46 PM
another moronic accreditation idea from the ABA - funny that this one follows on the heels of their proposed "equal opportunity and diversity" standard.
The ABA should just get - or better yet, be pushed - out of the accreditation game.
Posted by: Maryland Conservatarian at October 12, 2007 11:15 AM
