Rating Academic Reputation
posted by Daniel Solove
There’s lot of talk recently about how to rate an academic’s reputation. As scholars, we’ve devoted extensive thought and discussion to the issue. Some ingenious techniques we’ve devised:
1. Count citations to a scholar’s work. Of course, for the reasons Brian Leiter documents, citation counts aren’t an indication that a particular article is any good. Jack Balkin and Sanford Levinson have a hilarious discussion of the foibles of citation counts in their article, How to Win Cites and Influence People, 71 Chi.-Kent L. Rev. 843 (1996). They write, for example:
If you want to understand how to write the sort of articles that will get cited incessantly, you first must understand why people cite articles. The most obvious reason to cite something is as authority for a legal proposition one is arguing for. But much legal scholarship is cited to represent an idea, a movement, or a memorable phrase associated with one of them. Thus, many citations to legal scholarship, and particularly to the most canonical pieces of legal scholarship, are citations to what the article symbolizes rather than acknowledgements of the truth of what the article says. The most-cited articles are less influences than icons; they are like colors of paint conveniently dabbed on the canvas because they are familiar and are easily accessible. . . .
Citations, we think, are often just such a form of public relations or impression management. They are a way of displaying information the citing author wants to convey about him-or herself, while concealing other information that would interfere with his or her desired “performance” as a legal scholar. From a public relations perspective, it may be quite irrelevant what you’ve actually read—or, much less what you’ve genuinely grappled with—, as long as you can successfully create the impression that you are the sort of person who is familiar with the cited work. Just as the insecure dinner-party host can walk into the wine shop and ask for “the wines most often bought by classy people,” the insecure legal academic—and is this not a redundancy?—can rarely go wrong by associating him-or herself, even if only through footnotes, with the articles published by classy people in classy law reviews. Citations are thus an essential part of the rhetoric of the law review article. They are not merely forms of evidence or proof; they also fall under the category of what rhetoricians call the “ethical appeal”—the demonstration, necessary in every persuasive speech, that the speaker is the sort of person who can be trusted and believed.
2. Count SSRN downloads. Gordon Smith of the Conglomerate has a post about how the SSRN system can be gamed. But gaming only matters if SSRN downloads tell you something meaningful about a scholar’s articles. The problem is that SSRN downloads aren’t any indication of article quality — they’re an indication of popularity among surfers of the Web. Post an article about Paris Hilton’s porn video and you’ll be one of the most downloaded scholars. [And by mentioning this, I'm ironically gaming our blog's Site Meter stats as I hope to get many Google hits for Paris Hilton's video.] Or write an article about computer hacking and hope to get a Slashdot link, which will bring thousands of downloads. Just because an article is downloaded a lot doesn’t mean that it is respected or that it is any good.
3. Look to the article’s law review placement. Another technique for determining whether an article is any good is to see what law review it is published in. But many crappy articles get placed well and many good articles get placed badly. Nate Oman’s astute post aptly suggests we quit blaming the law reviews and start looking at ourselves.
Perhaps we should try something else: We could actually read the articles and determine for ourselves if they are any good. This would, of course, mean actually reading the articles. But why would we want to do something that tedious when checking SSRN and citation counts is much more quick and fun?
November 13, 2005 at 4:14 pm
Posted in: Law School, Law School (Scholarship)
Print This Post
Responses (4)
John Armstrong - November 13, 2005 at 4:52 pm
Actually reading the articles would be determining an academic’s quality, not his reputation. The two can be wildly different, though I’ll decline to give a list of who I think makes up the difference in my field until I’ve gotten tenure somewhere.
Daniel J. Solove - November 13, 2005 at 5:03 pm
John — Fair point. There is indeed a distinction between quality and reputation. Reputation is broader than quality — it encompasses a scholar’s quality as well as her popularity/visibility.
Quality is the most important component of reputation in my opinion. Thus, when I speak normatively about how we ought to evaluate another scholar’s reputation, I believe that we should do so by giving a very strong weight to quality. Not that visiblity/popularity aren’t important — but they should matter far less in my opinion.
As metrics of quality, citations, SSRN downloads, and article placements are not very good. Thus, we’re using metrics that measure the least important dimensions of a scholar’s reputation, thus giving short shrift to the most important reputational component — quality.
John Armstrong - November 14, 2005 at 6:02 pm
I agree that quality “should” be the dominant factor in reputation, but in the real world it just ain’t so (at least in my experience). Reputation is “the beliefs and opinions that are generally held about someone” (OED). It’s almost synonymous with popularity and visibility. I’m willing to believe that SSRNs, citations, and the like are an accurate measure of reputation even if they don’t really measure quality.
As an example, when I write a paper that mentions a certain basic component in knot theory, I must cite Kurt Reidemeister’s 1934 paper, even though in practice the results are not exactly those I’m using and there have been numerous refinements and better proofs that have come along in the interim. That paper isn’t the highest quality one that’s on-point, but it’s the most popular and visible one and if I don’t cite it my paper would never get past a referee.
Daniel J. Solove - November 14, 2005 at 9:16 pm
John — The fact that one is popular and visible just means that many people do, in fact, hold “beliefs and opinions” about that person. But reputation does not just involve the quantity of people who hold beliefs or opinions — rather, it also involves what these beliefs and opinions are. Thus, the indicators you mention go to how widely known a scholar might be, but not to the reputation, which also involves the content of the beliefs and opinions. If Scholar X is being cited because he’s the paradigm of a moron, then is this a reputation worth having?
Leave a Reply