Universities already perform the function you are proposing, by competing for academics whose work is thought to be valuable. That is, scientists (and others) with the best ideas do receive recognition and salary offers which account, at least in part, for the probability that their work will be recognized by Nobel and other prizes. So I think you are really just recommending the system that we already have.

I am not sure, however, that the mechanisms for hiring are replicated by granting agencies. Does the NIH look at who was being recruited by the top medical schools? Is there much learning? Or is it more likely that the top medical schools have people who are well equipped to judge who are the best young researchers?

The most prominent current proposals are from Love & Hubbard ( J. Love and T. Hubbard, “The Big Idea: Prizes to Stimulate R&D for New Medicines,” Chicago-Kent Law Review 82, no. 3 (2007): 1519-1554) and Pogge & Hollis (A. Hollis, “The Health Impact Fund: A Useful Supplement to the Patent System?” Public Health Ethics 1, no. 2 (2008): 124-133; Pogge, supra note 7; T. J. Hubbard and J. Love, “A New Trade Framework for Global Healthcare R&D,” PLoS Biology 2, no. 2 (2007): 147-150).

Prize options are certainly worthy of much more sustained attention.

(I should add by way of introduction that I’m not an attorney; I’m a computer scientist specializing in network security and assorted related public policy questions. I’ve served on and chaired many program committees that evaluate papers for publication; I seek grants; I’ve served on peer review committees that evaluate grant proposals. The only part of this process I haven’t personally experienced is that of the program manager — the government official who actually hands out the money.)

On paper, the grant process is fine. Proposals outline the new idea and the proposed research program, assess the impact if the work is successful, and (of course) request a particular budget. Reviewers (who are screened for conflicts of interest) evaluate the proposals, look at the impact and the cost, rank them, etc. So what could go wrong?

The problem is that unconventional ideas are usually wrong. There’s a reason the ideas are unconventional; in a mature discipline, there’s a long history of things that do work. Very many other paths were explored in the past and then abandoned as unfruitful. (To be sure, these dead ends are rarely described in the literature; there are very few papers submitted, let alone published, that say “here’s a mistake I made”. That in itself is a pity.) The challenge, then, is to pick out the ideas that just might be correct — and to be willing to take a chance on being wrong. (Even there, the process is stacked against the unconventional. You’re required, in grant applications, to list results from previous grants. Who wants to write “I tried this strange idea and got nowhere?” How often would you risk it?)

There’s also a sociological component at play. In a typical review committee, it usually takes only one loud voice to kill a submission. *Any* new idea will have flaws — but too often, reviewers will demand that the new approach to some old problem solve every other related problem that the field has worked through over the last 20 or 30 years. I’ll make up an example: suppose that someone had a radically new, promising approach to fighting spam, but adopting it would (remember that this is hypothetical) cause Internet congestion of a type that was solved 20 years ago. Is the research worth pursuing? What happens, too often, is that someone will say “but they completely ignore the congestion control problem!”, and point out — correctly — that compatibility with today’s congestion control mechanisms has been an absolute requirement for deployment of anything new. Should a new research idea be held to that standard? Too often, this happens.

The result is defensiveness. People “propose” things they know will work, because they’ve already done much of the work. You’re simply not allowed to have a significant gap in your solution; the reviewers *will* spot it, and give the money to another proposal that may have lower potential impact, but doesn’t have any lacunae.

Other fields have experienced this as well. In Lee Smolin’s book “The Trouble with Physics”, he talks of the dichotomy between “seers” and “craftspeople”. (See also my short essay of that title at http://www.cs.columbia.edu/~smb/papers/04336288.pdf) The seers propose the radical new ideas; the craftspeople fill in the blanks. But it’s hard, especially when peer review committees are staffed by senior people. The noted science fiction author Arthur Clarke had it right; his “first law” states “When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.” I should note that when Smolin wrote an article about this situation and had it rejected, he accused the journal editor of being part of the problem. The answer was actually more dismaying: the editor showed him so many other articles about it that he was contributing nothing new.

Unfortunately, I don’t know what to do about this. I try to remind myself, when evaluating papers and proposals, to watch for such a conservative approach. When I see new approach, high impact, high flaw papers, I try to ask that the problems be acknowledged and left for future papers, rather than asking authors to solve world hunger along with everything else. I’ve seen program committees with “chair prerogative” — the chair can unilaterally accept one or two papers that the committee has rejected, presumably if he or she thinks that they fit this profile. But I have yet to see a good structural solution. (In that vein, I’m trying hard not to reject out of hand the cost-benefit analysis idea, with or without auctions. As I noted, grant proposals do have to discuss the impact, albeit generally not quantitatively — but a quantitative assessment has to include a very big unknowable: what is the probability that the new idea, even if successful, will be successfully marketed? Market failures are very real. In my field, what Microsoft decides often makes or breaks an idea, but they’re (rightly) concerned with maximizing their total profit.)

And you’ll forgive me if I reuse most of this essay for a posting on my own blog…

More importantly, your rhetoric makes a dangerous slide when you start talking about “probability distributions of different levels of benefit” and “mathematically rigorous”. How does your approach justify either of these?

To say “probability distribution” suggests a frequentist point of view, based on observation of repeated trials. But actually all you’ll have is a distibution of subjective estimates. Then to characterize this approach as “mathematically rigorous” is accurate only in a very narrow sense, the same as saying that 1 unicorn + 1 unicorn = 2 unicorns.

You seem to be insisting on the use of fantasy numbers, in a manner that makes it easier to ignore their subjective orgins. This is ontogeny repeating phylogeny (or “…second time as farce”) in its exact parallel to the Army Corps of Engineers’ invention of cost-benefit analysis as a way to mask professional (or politically-influenced) judgment behind a veneer of objectivity. What’s objective or “rigorous” is the procedure, not the data or the judgment.

I’m deeply agnostic about the use of auctions in this context, but at least they seem to be more transparent about the subjective origins of the numbers arrived at. But maybe my optimism that they won’t be reified comes only from my not being an economist.

Tom