Concurring Opinions » Charles Sullivan

Shepardizing the Academy

Charles Sullivan — Tue, 05 Sep 2006 18:36:41 +0000

Having received the guest bogger’s dreaded “here’s your hat, what’s your hurry” from Dan Solove, I thought I’d sign off (and make way for my colleague Rachel Godsil who will undoubtedlly be far more re interesting than I), with a final entry on easing the plight of the scholar.

A perennial complaint of the legal scholar is the difficulty of keeping up with the literature. (Admittedly, the non-academic world, especially the part of it that moves heavy things, is not likely to be sympathetic to our travails, but we’re speaking within the club here).

One time-honored solution, of course, is to re-define one’s field into smaller and smaller fragments, thus excluding increasingly larger amounts of material from that about which one must know. This has its limitations, however. I don’t mean logical limitations because, like particle physics, any field can apparently be reduced to progressively smaller parts. The limits are mostly loss of credibility among colleagues and students when a supposed expert really doesn’t know much about the next quark over.

Anyhow, the problem, as I see it, that we don’t have a simple device to help us decide what’s worth reading. Our cousins in the bench and bar have such a mechanism for the tools of their trade –Shepards for Lexis and whatever West calls its imitation. But there’s no similar labor-saving device for scholarship. Thiink of how much easier our lives would be with some version of this for our articles:

Of course, one can jury-rig available tools to come up with some approximation of a Shepards, but present technology is too limited. It’s easy to see how often a particular article has been cited, but not so easy to see if it’s been string-cited, actually discussed at any length, and ultimately approved or disapproved by other scholars. Various rankings, of course, periodically attempt to redress this, but often do so only for the upper tier. SSRN downloads are a measure, but maybe only of how interesting the abstract (as opposed to the article) actually is, and then there’s the pesky problem of how to assess success once an article has been published and scholars are (presumably) accessing it on Lexis or Westlaw or (gasp!) even in hardcopy.

One partial solution is so obvious it’s amazing no one has thought of it before: a Shepards for scholarship. (OK, someone probably has, but the closest I’ve seen is Michael Madison, who suggests a theoretically better but practically more complicated method of post-publication review by scholars “tagging” others’ work. (Thanks for the citation to my colleague Frank Pasquale who thinks a lot about “Information Overload Externalities”.)

Naysayers will at once object that Shepardizing scholarship would lack the crisp tools of case law. It’s true that it is rare for a scholar to “overrule” himself and, of course, a scholar cannot be “reversed” by anyone else. More’s the pity. But a moment’s thought l reveals that much of the work done by Shepards is not limited to critical moments in a precedent’s life (or death). Rather, much of Shepards is devoted to tracking who cited whom, and, using what I assume are algorithms rather than human judgment, determining whether a particular authority has been “questioned” or “disapproved.” And I’m pretty confident that some algorithm often explains the ultimate summary signal that a case is bad law (Lexis uses a stop sign) or dubious (Lexis uses a question mark).

That’s all I want: some system designed to tell me who cited what, when, and how favorably.

Now, I admit, this is not a panacea. There are those who find signals to be misleading, and I admit to having been taken aback when I discovered that Brown v. Board of Education was assessed by Lexis as.

Who knew? Maybe the Court has shifted further to the right than I’d noticed.

But all tools have their limitations, and a Scholarly Shapards (or maybe a version of this on SSRN) would go a long way to filling a gap. Not to mention the fun of developing the correct algorithms to decode the academic language by which scholars express their true opinions of prior work.

The biggest problem with this proposal is the obvious — it would be of little use in deciding what new scholarship to read since, by definition, it hasn’t been around long enough to trigger reactions from other scholars. Truly, there’s no rest for the weary.

Are We Writing Only For Ourselves?

Charles Sullivan — Thu, 31 Aug 2006 15:56:57 +0000

In 1992, Judge Harry Edwards wrote an article in Michigan, The Growing Disjuncture Between Legal Education and the Legal Profession, which was, as the title suggests, an effort to redirect legal scholarship from what Edwards perceived as a too-abstract and theoretical path towards work that would be of more immediate use to h judges and lawyers. I think it fair to say that, while legal scholarship, has since evolved in a variety of directions, one of them has not been more practical scholarship along the lines Judge Edwards urged.

His piece has been much discussed, but also essentially ignored as legal scholarship has moved futher in the direction Edwards decried (I’m speaking too broadly here, I know, since terrific doctrinal scholarship continues to be produced and an increasingly empirical literature might be the most “practical” of anything that appears in the law reviews). But the median journal article, so to speak, at least in the leading law reviews, is one it is hard to imagine a lawyer or court citing, and not much easier to imagine your average policy-maker even understanding.

But this is old news. As I start this academic year, iit occurred to me that not only is legal scholarship no longer written for judges and attorneys but we in the legal academy seem far less interested than in past years in even acquainting future judges, lawers, legislators, and policy makers with our scholarship. Casebooks are increasing ignoring the literature that is the stuff of academic discussions of the topic in question, and, when the scholarship is cited it is typically in tip-of-the-hat fashion.

This might be a kind of left-handed compliment to Edwards, who was also concerned about whether legal education was failing to educate students for actually practicing law: “The law student who merely takes a variety of pure theory courses, and learns that ‘practitioners [a]re sell outs,’ will be woefully unprepared for legal practice. That student will lack the basic doctrinal skills: the capacity to analyze, interpret and apply cases, statutes, and other legal texts. More generally, the student will not understand how to practice as a professional.”

I guess my thesis is that legal education has responded to Edwards by taking his critiques of the educational process seriously, but not by altering in any meaningful way the fundamental goals of legal scholarship. Thus, the disjuncture that Edwards was concerned about to a full-fledged breach. We not only do not write for the bench and bar — we also do not write for our students.

Some might say that the move towards simpler, less sophisticated casebooks, is due to pressures by publishers, who want shorter, more student-friendly books. But, of course, students don’t buy casebooks (well, they do, literally speaking, but they are a captive audience for the professors who “require” them), so it’s likely that the publishers are reacting to the market which is, in turn, rejecting (as far as the classroom is concerned) our own work. Casebooks have always been viewed as “the lowest form of scholarship,” but current pressures threaten to push them down even further — to the highest form of study aids.

All of which leads to the title of this piece — “are we writing only for ourselves?” And, if so, why?

Digital Scholarly Integrity

Charles Sullivan — Sun, 20 Aug 2006 03:32:45 +0000

I recently received a phone call from an attorney — a piece she had published some time ago had an embarrassing mistake in its title. Needless to say, she was looking for a new position at the time and was concerned that a prospective employer would see the title and send her application to the circular file. While I was less worried than she that due diligence by employers extended this far, I too would have stewed at such a mistake hanging over my head forever.

Not to worry. While the published version will continue to exist (and embarrass) in wherever inaccessible and unaccessed places it now resides, the digital versions in Lexis and Westlaw have been corrected.

It would be hard to quibble with the correction in question — truly a typographical error — and both Lexis and Westlaw required authorization from the journal in question. Neither responded just to the importuning of the author of the piece.

But it made me wonder about both the security of our digital scholarship and the scholarly ethics of making changes, at least more substantive changes than this.

As to digital integrity, law reviews are increasingly not read in their traditional form, and many libraries are not even carrying many journals they previously stocked. Lexis and Westlaw have become the gold standard — and I am excluding SSRN intentionally. There is often a substantial difference between the version(s) of a piece posted on SSRN and the published article, making it dangerous to rely too heavily on the SSRN version. Indeed, one of my colleagues spent some time responding to a SSRN piece only to discover that the printed version had removed the offending section.

Thus, what Lexis and Westlaw reproduce is, for all practical purposes, what we write, and the extent to which the two databases honor requests for changes by journals and/or authors raises some interesting questions — ones to which I do not have an answer. While few would object to correction of typographical errors (a recent article I saw included “insert emdash” in the text, twice), what is a typo and what is a substantive change can obviously be a close call. Further, there is obviously value to freezing scholarly output — warts and all. But making some changes is not only important to avoid author (or journal or home institution) embarrassment, but might also be critical when a problem reaches the level of libel or plagiarism. So one issue is the precise policies of Lexis and Westlaw and the extent to which the scholarly community is comfortable with them.

Then there is the scholarly ethics question. Assuming the databases will allow a given change, what kinds of changes do we think are in bounds, and when? Although I am probably the only one who has noticed most of the gaffes in my past work, there are a fair number of things I would like to change — if I could. History is said to be written by the winners, but it might be rewritten by those with the incentive to do so.

Pushing, or Perhaps Ripping, the Arbitration Envelope

Charles Sullivan — Tue, 15 Aug 2006 14:20:54 +0000

The New Jersey Supreme Court has just issued an opinion on arbitration that, while perhaps sensible in result, seems to turn the usual paradigms upside down. Instead of deciding whether an arbitration clause was unconscionable and should be stricken (or at least whether the unconscionable parts should be severed), the Court decided (1) it didn’t know what the clause meant, (2) that question was for the arbitrator, but (3) that if the arbitrator decided certain ways, that would be unconscionable. The effect, of course, was to interpret the agreement it wasn’t interpreting, at least by ruling out certain interpretations.

Given the pervasiveness of arbitration in both consumer and employment contexts, this is an important decision. How it fits into more conventional notions of the relationship between courts and arbitrators is anyone’s guess, but I’ll hazard a few.

The case is Delta Funding v. Harris, and I encountered it because one of my colleagues, Bahar Azmy in Seton Hall’s Center for Social Justice, submitted an amicus brief. The underlying dispute involved the usual predatory lending claim, with a twist: the arbitration clause exempted foreclosure proceedings. The result was that Ms. Alberta Harris, “a seventy-eight-year-old woman with only a sixth grade education and little financial sophistication,” was forced to defend a foreclosure proceeding in state court but if she wished affirmative relief (such as under the Consumer Fraud Act) or or to sue third parties, had to resort to arbitration.

The case reached the NJ Supreme Court by certification from the Third Circuit when Harris appealed from the federal district court’s order that the non-foreclosure claims be abitrated (which resulted in the state court dismissing those claims in the foreclosure proceeding). The Third Circuit asked the NJ Supemes (as the latter reformulated the question), “Is the arbitration agreement at issue, or any provision thereof unconscionable under New Jersey law, and, if so, should such provision or provisions be severed.”

To which NJ responded: we don’t know. Or more precisely: there are ambiguities that it is the job of the arbitrator to decide, and “To the extent that the unconscionability of those provisions ultimately turns on how the arbitrator resolves the ambiguities, we are unable to determine whether the provisions are, in fact unconscionable.” However, the Court then went on to explain in chapter and verse how certain provisions could be applied unconscionably, for example, allocation of the costs of the hearing.

Such an answer might not be so problematic within one court system (where the case could be sent down to a lower court for review consistent with the appellate opinion), but with the court/arbitration intersection, it raises real questions that the Third Circuit will have to struggle with in the first instance.

After all, the defendant wanted the arbitration clause struck down so it proceed in state court. If it were unconscionable (and not severable) that would be the indicated relief. But we now know it might or might not be unconscionable, so (maybe) the Third Circuit should affirm the district court order compelling arbitration of Harriss’s affirmative claims against Delta.

But if it does, and the case goes to arbitration, what is the effect of the NJ Supreme Court’s decision on the arbitrator? (One might also ask about the collateral estoppel effects, if any, of any decision in the foreclosure proceedings on the award, but that takes us too far afield). The Federal Arbitration Act does not pemit a very generous review of arbitral awards (to do so would convert arbitration into another system of lower courts), but it does allow limited review and, as interpreted by the courts, one basis for rejecting an award is “manifest disregard” of the law by the arbitrator. So, should the arbitrator interpret the agreement contrary to the Delta decision, presumably it can be set aside.

This, however, suggests an end-run around the notion that the arbitrator decides what a contract means — the enforcing court announces what the contract doesn’t mean, thereby cutting down substantially on the wiggle room the arbitrator would otherwise have. Plus, to the extent the NJ Court was focusing on issues specific to arbitration (such as cost allocation), it also results in state-mandated rules specific to arbitration, which the United States Supreme Court has rejected on Supremacy Clause grounds.

Given the pervasiveness of arbitration these days, Delta may be the best we can hope for, but whether it respects the boundaries the Supreme Court has tried to establish between court and arbitrator is another question.

It is probably too much to hope that the Third Circuit will decide that an arbitration clause that gives the arbitrator so wide latitude to impose unconscionable terms in the arbitration is thereby unconscionable and should not be enforced.

Mel Gibson and the IAT

Charles Sullivan — Tue, 08 Aug 2006 16:15:08 +0000

My corner of the academic cabbage patch is consumed these days with whether intent to discriminate includes “unconscious bias,” one of many terms for a phenomenon that is helpfully taxonomized by my colleague Mark Poirier in his article Using Stereotyping and Cognitive Bias Evidence to Prove Gender Discrimination: Is Cognitive Bias at Work a Dangerous Condition on Land?, 7 EMPLOYEE RTS. & EMP. POL’Y J. 459 (2003). Research in this area includes the Implicit Association Test, which you can take from the comfort of your computer to determine your implicit attitudes towards a wide variety of traditional discrimination subjects (race, sex, sexual orientation), and some not so traditional (Presidents). Needless to say, there is a great controversy as to whether this test measures what it purports to measure, and, if so, whether such attitudes are likely to affect conduct in real world situations, and, if so, whether the law can or should do anything about it. Amy Wax savaged the IAT in a co-authored commentary in the Wall Street Journal last December.

Now along comes Mel Gibson and pushes the issue into the limelight. Sort of….

You’ve undoubtedly seen coverage of Mel Gibson’s antisemetic remarks while being busted for DWI. While the story is too well-known to need citation here, I was struck by a comment from Rabbi Abraham H. Foxman, national director of the Anti-Defamation League: ”If it’s true what’s reported, frequently hatred, bigotry and prejudice, which is controlled, explodes at moments of stress and crisis.” He went on ”Liquor loosens the tongue of what’s in the mind and in the heart, and in his mind and in his heart is his conspiracy theory about Jews and hatred of Jews.”

This is, of course, not the first time Mr. Gibson has been accused of anti-Semitism. On this occasion, however, not only did he provide some pretty strong admissions of hostile beliefs as to Jews by the language he used (what Justice O’Connor, with a fine disregard for evidence theory, used to call “direct evidence”) but his condition validated, for Rabbi Foxman, and many others, the depth and emotional salience of these beliefs.

“In vino veritas” predates Mel Gibson (actually, it may predate the events retold in his Passion), but exactly what that old saw means isn’t clear. The Times’ Week in Review summed up the controversy with the headline “Is It the Drunk or the the Drink Doing the Talking?”

One possibility is the obvious — Mel has always known his true beliefs but has concealed them from the public because of the negative reaction they are sure to invoke. When he’s drunk, the gatekeeper is asleep at the wheel, and the true Mel emerges.

But another possibility is that liquor may unleash feelings of which the imbiber is not consciously aware. One of Gibson’s apologies reported that he is “in the process of understanding where those vicious words came from during that drunken display.”

Then there’s a third possibility — some of his friends, including Jewish friends, seem to suggest that it wasn’t him doing the talking at all — it was the drink. Whatever that means.

While I (and no doubt most others) am skeptical about Gibson’s denial of self-knowledge, the whole implicit attitude project suggests that it is not as far-fetched as it might seem. Dr. Kevin Corocoran, identified in the article as “a psychological researcher studying alcohol’s effect, stated that Gibson “may not fully believe” what he said.

I’m not suggesting that researchers move from the computer to the barroom to conduct their studies, nor am I arguing that Professor Wax is necessarily wrong about the limitations (or legal utility) of such research. My point is more limited – we pretty much all believe in strong, if concealed, motivations that, as Foxman argues, are more likely to see the light of day only in moments of stress and crisis, and maybe intoxication. “Concealed” might mean fully known to the person but concealed from the world. But it might alternatively mean motivations that are in some sense concealed from the individual himself — if not the underlying cognitive beliefs at the least the emotional force that Gibson’s statements revealed.

As the courts struggle with the role, if any, for research such as the IAT, they should at least recognize the commonsense basis for such work.

More on Ranking Law Reviews

Charles Sullivan — Thu, 03 Aug 2006 15:43:46 +0000

The Sullivan Scale

I’m a big fan of data mining (not the NSA variety, but the kind you do when you’re cleaning up your office), and there, nestled next to an article on rankings that I had lied to myself about responding to some day, was a pile of rejection letters from my Spring submission. As I was throwing them away, I noticed that several tried to ease the pain of rejection by informing me that I was just one of many who were also not quite good enough (many of these letters also solicited me to try again, as they had done the last 19 times).

Eureka, I thought, the perfect ranking system: ranking law reviews by number of submissions. One clear advantage of this method is that it does not necessarily reproduce the current hierarchies that dominate the other rankings.

Rank/ Law Journal/ # of Submissions

1 Stanford/ 3000

2 Ohio State/ 2000

2 Iowa/ 2000

2 Virginia/ 2000

2 Cornell/ 2000

6 Texas/ 1500

7 California/ 1200-1500

8 Tulane/ 1000

8 Rutgers-Camden/ 1000

10 Davis/ 800

While Stanford has the highest rating, thus providing some confirmation for other rankings, there are some obvious omissions from the top tier. Harvard, Yale, Columbia, for example – not there! Admittedly, there are a number of other absences; indeed, almost all law reviews are missing. This does pose some problems for the Sullivan Scale, but not insurmountable ones. And notice that the Scale, limited though it may be, has provided some useful information: Ohio State is ranked higher than Virginia, Texas, and California, not something most other ranking have uncovered.

There are two problems with my data. The first was that I didn’t submit to all the law reviews, not even all the 200 or so primary journals. It would be fair for Great-Review-at-Fourth-Tier-School to complain about my data collection. Obviously, I can resolve this with my next submission, at really only a very small cost. ExpressO will allow me to saturate the known universe for less than $1000. The second problem is, in my mind, a nonstarter: most reviews that rejected me didn’t tell me how many other articles they rejected. True, but, I ask you, whose fault is that? Besides as the Scale gains traction, this problem will solve itself. Everyone will feel obligated to provide me with the data I need, if only I am prepared to risk rejection.

I see two objections to the whole enterprise, one theoretical, the other cynical. The theoretical objection would ask why submissions are a good marker of quality. The answer seems plain: we are the academy, we know best.

The cynical objection would question reliability of data or at least manipulability. For example, what’s with California’s 1200 to 1500? Can’t they count out there? And it’s one thing for Stanford to claim 3000 submissions (a suspiciously round number in any event) when Yale is not on the board, but it might be tempted to report Yale + x the next time around. Similarly, a journal trying to improve its rankings might resort to tactics to increase submissions.

In an earlier era, the reliability objection would be serious, but in an age of ExpressO, there is some independent check on reliability (indeed, maybe I should just use ExpressO submissions, if they’d let me have it — that would avoid the heartbreak of collecting data through rejections).

As for encouraging submission to improve a journal’s rankings, the most effective way to increase submissions is to offer a greater prospect of actually being published by adding slots or reducing placements by the usual suspects. I ask you, what’s wrong with that?

Ranking Law Reviews as the August Window Opens

Charles Sullivan — Tue, 01 Aug 2006 13:23:28 +0000

It’s not so clear whether the recent spate of scholarship on ranking law reviews is a reaction to an emerging phenomenon or oblivious to it. Rankings of law reviews long predate the recent efforts and, indeed, long predate the US News & World Report ranking of law schools. However, judging from my younger colleagues, US News is a perfect proxy for law review ranks. (To that extent, I take an even stronger position than Professor Alfred Brophy’s recent posting on SSRN, which, after engaging in considerably more empirical research than I, finds a .86 correlation between citations to a school’s main review and USNews peer review ranking).

In short, I believe that, when engaged in window shopping during the Spring and summer submission seasons., both the submission strategy and the expedite strategy are pretty much dictated by where a particular review’s law school falls on the US News scale. There is almost no concern with whether a review is better (or worse) than its home law school, nor any effort to consult studies such as Professor Brophy’s.

To the extent that this view pervades legal academia, and I am confident it does, other rankings of law reviews are exercises in futility: if suppliers of “content” (sometimes still called scholarship), follow the USNews hierarchy, law reviews will have their choice of articles pretty much in that order. Unless the student editors screw up pretty badly (and maybe even if they do), the “best” scholarship will continue to be published in the “best” journals, and the “best” journals will continue to be those associated with the “best” schools, all as determined by US News.

Precisely why this has occurred is unclear. US News is the 800 pound gorilla of legal education, so maybe it is not so surprising that it ends up ranking things it doesn’t even purport to rank. In that sense, maybe “black hole” is the more appropriate metaphor. But legal scholars should make more informed judgments about law review quality than do 22 year olds as to graduate schools to attend, so it remains a bit odd thatwe don’t spend more time deciding what their assessments are.

I think the reasons are fourfold. First, scholarship is more specialized these days, maybe hyperspecialized. How can we judge the quality of a law review when we can only understand an article or two per volume?

Second, we used to actually see law reviews, or at least the covers. I could tell what Harvard or Baylor published at a glance. Now we get much of our information in more focused SSRN or CILP or Lexis/Westlaw searches and usually have no idea what other articles were published in the issue.

Third, the current crop of young scholars came to adulthood under the spell of USNews. At some deep level, they believe in the rankings as more-or-less accurate with respect to schools and expect that law reviews will follow suit. When challenged, they will point out that, even if that’s not completely accurate, bounded rationality justifies looking to USNews rather than sorting through competing rankings.

Fourth, and probably most important, why pick # 53 over #41 (I promise, I didn’t look to see who these schools actually are!) just because it’s a better review qua review when friends at other school, colleagues, tenure committees and (God help us) University Provosts won’t understand the “fall” in prestige?

In sum, and with apologies to those who have put in so much effort in this enterprise, we should give up ranking reviews. (Well, actually, there may be a niche for rankers — we seem to be confused as to how to compare a secondary journal from Elite School with a primary journal from Lower Tier school. While it’s my sense that Elite School Secondary prevails over Tier 3 or 4 Primary, it’s possible that ranking studies may solve this puzzle.)

For those who disagree and want to continue the enterprise, my next blog will offer a cost-effective way to go about doing it, and trot out a new ranking, which, with all due modesty, I describe as the Sullivan Scale.