Improving the Grant System with Prizes
posted by Michael Abramowicz
In a front-page article in yesterday’s New York Times, Gina Kolata argues that the system of awarding grants for cancer research unduly favors research projects that make incremental advances over projects that have a smaller probability of achieving more fundamental breakthroughs by challenging established dogmas. This particular problem is part of a broader problem: Decisions on grant funding are not made using cost-benefit analysis or any systematic methodology for assessing which projects are the most promising. And that in turn is part of a broader problem still: Granting agencies don’t have much incentive for identifying the procedures that are likely to lead to the socially best allocation of research dollars.
At a bare minimum, grant-granting institutions ought to generate probability distributions of different levels of benefit for alternative proposed projects. I suspect that resistance to such an approach stems from recognition that any subjective estimates of both probabilities and benefits are likely to be somewhat arbitrary. How can even an expert scientist know that there is either a 1% or a 5% chance that an experiment testing an unorthodox claim will be successful? And that difficulty pales in comparison to the challenge of assessing the benefits of experiments. We might be able to estimate the benefits of a cure for cancer, by estimating the effects of a cure on quality-adjusted life years, but it is difficult to assess how far toward that goal any particular successful experiment will bring us. The task is made still more complicated by the fact that some experiments will be valuable not because they confirm either the experimenters’ or skeptics’ views, but because they produce some entirely serendipitous discoveries.
My view is that grant decisions will be better if we force scientists making assessments to give their best subjective estimates, ultimately producing a probability distribution of different possible benefit levels, even if such numbers are inherently subjective. It seems unlikely that intuitive decisionmaking will produce better results than more rigorous approaches. Scientists may worry that quantification would discourage investments in basic research relative to more applied research. The reverse seems likely to be true. The more foundational the research, the greater the potential benefits to which it may contribute, and this factor seems likely to outweigh the fact that any single highly theoretical experiment may provide only a small bit of progress. Whether I’m right or wrong about this, allocation decisions ideally should be based on rigorous analysis of this question, or at least on moderately developed back-of-the-envelope calculations, rather than on pure intuition.
One objection is that any system that the government or indeed any bureaucracy develops for making more mathematically rigorous assessment of grants may be flawed by ignoring important criteria that scientists may take into account implicitly. But it need not be government that is charged with making these estimates. An alternative to the grant system would flip government’s role to ex post evaluation of benefits and costs. Twenty-five years from now, it should be much easier for scientists to assess the relative benefit of experiments conducted today. Instead of grants, the government could place grant money into a prize fund, let it accumulate interest, and distribute the money later. This approach would give private parties, akin to venture capitalists, incentives to anticipate the benefits of research. At the least, such parties should be less risk averse than the grant agencies that Kolata describes.
This may seem too radical a change from our existing system of scientific funding. But it is possible to integrate a modest version of this system within the existing grant system. For example, we might set aside just 10% of current grant money for a prize fund. Private parties would be required to auction their rights to any prize to independent third parties, conditional on the grants being approved. The grant agency might then consider the results of the auctions, in addition to any information they ordinarily would consider. At the least, this could help provide the grantors cover for approving low-probability, high-benefit projects. Moreover, the practices of the third parties-What kind of models do they use? What kind of disclosure do they expect from grant applicants?-might help us identify how we could improve the government’s own procedures. Whether or not the auction participants do a better job than the government (and with relatively small stakes, they might not), the types of projects they select with their own money on the line could help inform the government about what its decisionmakers’ biases might be.
June 29, 2009 at 12:33 pm
Posted in: Administrative Law
Print This Post
Responses (5)
Tom Lopy - June 29, 2009 at 3:08 pm
Redefinition of grants and awards wouldn’t be easy, you would be fighting the massively popular and successful firms that are making incremental progress, a massive roadblock.
Check out my website at http://newfiction.com if you like science and mystery.
Tom
A.J. Sutter - June 29, 2009 at 5:09 pm
Since you’re alarmed at the non-use of CBA, why should scientists be “force[d]” to estimate benefits only, but not costs?
More importantly, your rhetoric makes a dangerous slide when you start talking about “probability distributions of different levels of benefit” and “mathematically rigorous”. How does your approach justify either of these?
To say “probability distribution” suggests a frequentist point of view, based on observation of repeated trials. But actually all you’ll have is a distibution of subjective estimates. Then to characterize this approach as “mathematically rigorous” is accurate only in a very narrow sense, the same as saying that 1 unicorn + 1 unicorn = 2 unicorns.
You seem to be insisting on the use of fantasy numbers, in a manner that makes it easier to ignore their subjective orgins. This is ontogeny repeating phylogeny (or “…second time as farce”) in its exact parallel to the Army Corps of Engineers’ invention of cost-benefit analysis as a way to mask professional (or politically-influenced) judgment behind a veneer of objectivity. What’s objective or “rigorous” is the procedure, not the data or the judgment.
I’m deeply agnostic about the use of auctions in this context, but at least they seem to be more transparent about the subjective origins of the numbers arrived at. But maybe my optimism that they won’t be reified comes only from my not being an economist.
Steven M. Bellovin - June 29, 2009 at 7:54 pm
I think the Kolata’s article was dead on. Furthermore, I know that the problems she describes are present in my field (cybersecurity); from what I’ve read, they’re present in many other fields as well. I’ll take a step further — I’ll assert that they’re inherent in most (and possibly all) of our much-vaunted peer review process. Peer review, whether for grant funding or research paper publication, is an inherently conservative process. It does a very good job of weeding out bad ideas; it’s very much worse at understanding the merits of unusual ones.
(I should add by way of introduction that I’m not an attorney; I’m a computer scientist specializing in network security and assorted related public policy questions. I’ve served on and chaired many program committees that evaluate papers for publication; I seek grants; I’ve served on peer review committees that evaluate grant proposals. The only part of this process I haven’t personally experienced is that of the program manager — the government official who actually hands out the money.)
On paper, the grant process is fine. Proposals outline the new idea and the proposed research program, assess the impact if the work is successful, and (of course) request a particular budget. Reviewers (who are screened for conflicts of interest) evaluate the proposals, look at the impact and the cost, rank them, etc. So what could go wrong?
The problem is that unconventional ideas are usually wrong. There’s a reason the ideas are unconventional; in a mature discipline, there’s a long history of things that do work. Very many other paths were explored in the past and then abandoned as unfruitful. (To be sure, these dead ends are rarely described in the literature; there are very few papers submitted, let alone published, that say “here’s a mistake I made”. That in itself is a pity.) The challenge, then, is to pick out the ideas that just might be correct — and to be willing to take a chance on being wrong. (Even there, the process is stacked against the unconventional. You’re required, in grant applications, to list results from previous grants. Who wants to write “I tried this strange idea and got nowhere?” How often would you risk it?)
There’s also a sociological component at play. In a typical review committee, it usually takes only one loud voice to kill a submission. *Any* new idea will have flaws — but too often, reviewers will demand that the new approach to some old problem solve every other related problem that the field has worked through over the last 20 or 30 years. I’ll make up an example: suppose that someone had a radically new, promising approach to fighting spam, but adopting it would (remember that this is hypothetical) cause Internet congestion of a type that was solved 20 years ago. Is the research worth pursuing? What happens, too often, is that someone will say “but they completely ignore the congestion control problem!”, and point out — correctly — that compatibility with today’s congestion control mechanisms has been an absolute requirement for deployment of anything new. Should a new research idea be held to that standard? Too often, this happens.
The result is defensiveness. People “propose” things they know will work, because they’ve already done much of the work. You’re simply not allowed to have a significant gap in your solution; the reviewers *will* spot it, and give the money to another proposal that may have lower potential impact, but doesn’t have any lacunae.
Other fields have experienced this as well. In Lee Smolin’s book “The Trouble with Physics”, he talks of the dichotomy between “seers” and “craftspeople”. (See also my short essay of that title at http://www.cs.columbia.edu/~smb/papers/04336288.pdf) The seers propose the radical new ideas; the craftspeople fill in the blanks. But it’s hard, especially when peer review committees are staffed by senior people. The noted science fiction author Arthur Clarke had it right; his “first law” states “When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.” I should note that when Smolin wrote an article about this situation and had it rejected, he accused the journal editor of being part of the problem. The answer was actually more dismaying: the editor showed him so many other articles about it that he was contributing nothing new.
Unfortunately, I don’t know what to do about this. I try to remind myself, when evaluating papers and proposals, to watch for such a conservative approach. When I see new approach, high impact, high flaw papers, I try to ask that the problems be acknowledged and left for future papers, rather than asking authors to solve world hunger along with everything else. I’ve seen program committees with “chair prerogative” — the chair can unilaterally accept one or two papers that the committee has rejected, presumably if he or she thinks that they fit this profile. But I have yet to see a good structural solution. (In that vein, I’m trying hard not to reject out of hand the cost-benefit analysis idea, with or without auctions. As I noted, grant proposals do have to discuss the impact, albeit generally not quantitatively — but a quantitative assessment has to include a very big unknowable: what is the probability that the new idea, even if successful, will be successfully marketed? Market failures are very real. In my field, what Microsoft decides often makes or breaks an idea, but they’re (rightly) concerned with maximizing their total profit.)
And you’ll forgive me if I reuse most of this essay for a posting on my own blog…
Kevin Outterson - June 30, 2009 at 10:29 am
Quite a few people are working on these prize-type proposals, including Michael’s Perfecting Patent Prizes, 56 Vand. L. Rev. 115 (2003) and my own work on post-patent buyouts http://ssrn.com/abstract=873402 and open access licensing http://content.healthaffairs.org/cgi/content/abstract/27/1/130.
The most prominent current proposals are from Love & Hubbard ( J. Love and T. Hubbard, “The Big Idea: Prizes to Stimulate R&D for New Medicines,” Chicago-Kent Law Review 82, no. 3 (2007): 1519-1554) and Pogge & Hollis (A. Hollis, “The Health Impact Fund: A Useful Supplement to the Patent System?” Public Health Ethics 1, no. 2 (2008): 124-133; Pogge, supra note 7; T. J. Hubbard and J. Love, “A New Trade Framework for Global Healthcare R&D,” PLoS Biology 2, no. 2 (2007): 147-150).
Prize options are certainly worthy of much more sustained attention.
Aidan Hollis - July 5, 2009 at 10:03 pm
Michael,
You have wonderful ideas… And I think that much of the interesting work in this area starts with your 2003 paper.
Universities already perform the function you are proposing, by competing for academics whose work is thought to be valuable. That is, scientists (and others) with the best ideas do receive recognition and salary offers which account, at least in part, for the probability that their work will be recognized by Nobel and other prizes. So I think you are really just recommending the system that we already have.
I am not sure, however, that the mechanisms for hiring are replicated by granting agencies. Does the NIH look at who was being recruited by the top medical schools? Is there much learning? Or is it more likely that the top medical schools have people who are well equipped to judge who are the best young researchers?
Leave a Reply