Article on Ranking and Citation
posted by Dave Hoffman
I just saw that Chee Chow (San Diego State, Accounting) et al. have posted How Well Can Publication of an Article in a Top Accounting Journal Be Used as a Proxy for its Contribution? on SSRN. From the abstract:
The study investigates the appropriateness of using publication of an article in a top (specifically, top three) accounting journal as a proxy for its quality, as reflected by its impact on others’ research. This investigation is motivated by an apparent increase in pressures to publish in top journals, with attendant effects on the allocation of faculty and institutional resources and more broadly, the health of accounting knowledge advancement.Following related prior studies, SSCI citation counts are used as the primary metric for gauging impact. Over eight-year event windows for articles published in nine accounting journals in 1992, 1994 and 1996, articles published in the journals most often considered to be the top three (Journal of Accounting and Economics, Journal of Accounting Research, and The Accounting Review) do tend to be cited significantly more often than ones published in the other journals, though some of the non-top-three journals are not far behind. Far more important, across three different criteria for placing articles into top vs. non-top categories, there were substantial errors from using a top-three journal ranking as a proxy for quality. Many top accounting articles are published in non-top-three journals, at the same time that substantial numbers of non-top articles make it into the top three. Citations from a Google-based Internet search exhibited somewhat different patterns, but did not change these fundamental results. Together, these findings strongly support the need to evaluate each article on its own merit, rather than abdicating this responsibility by using journal ranking as a key proxy for quality.
The results of this study aren’t all that surprising, but it does seem circular to use citations as a quality proxy, in an article that recommends evaluating pieces on “merit”. The authors acknowledge this oddness, especially at pgs. 7, 18-19. Their defense of the measure is that qualitative measures of merit across many articles is a project whose “enormous scope” renders it impractical. And they remind us that institutions might not care as much about quality as a naive academic might think. It is, indeed, “possible” that schools might seek to promote placement in “top” journals because of their “circulation, visibility, and perceived statute,” not quality signalling. (18)
Possible?
May 2, 2007 at 11:21 am
Posted in: Law School (Scholarship)
Print This Post
Responses (1)
Frank - May 2, 2007 at 1:31 pm
Another issue, elucidated toward the James R. Hackney’s Under Cover of Science: American Legal-Economic Theory and the Quest for OBjectivity, is to question whether there is a measure of “quality” that is independent of ideological commitments. A tour de force case for a more egalitarian society may be deemed trivial or irrelevant by those pursuing other objectives.
Leave a Reply