Site Meter

Author: Michael Abramowicz


Improving the Grant System with Prizes

In a front-page article in yesterday’s New York Times, Gina Kolata argues that the system of awarding grants for cancer research unduly favors research projects that make incremental advances over projects that have a smaller probability of achieving more fundamental breakthroughs by challenging established dogmas. This particular problem is part of a broader problem: Decisions on grant funding are not made using cost-benefit analysis or any systematic methodology for assessing which projects are the most promising. And that in turn is part of a broader problem still: Granting agencies don’t have much incentive for identifying the procedures that are likely to lead to the socially best allocation of research dollars.

At a bare minimum, grant-granting institutions ought to generate probability distributions of different levels of benefit for alternative proposed projects. I suspect that resistance to such an approach stems from recognition that any subjective estimates of both probabilities and benefits are likely to be somewhat arbitrary. How can even an expert scientist know that there is either a 1% or a 5% chance that an experiment testing an unorthodox claim will be successful? And that difficulty pales in comparison to the challenge of assessing the benefits of experiments. We might be able to estimate the benefits of a cure for cancer, by estimating the effects of a cure on quality-adjusted life years, but it is difficult to assess how far toward that goal any particular successful experiment will bring us. The task is made still more complicated by the fact that some experiments will be valuable not because they confirm either the experimenters’ or skeptics’ views, but because they produce some entirely serendipitous discoveries.

My view is that grant decisions will be better if we force scientists making assessments to give their best subjective estimates, ultimately producing a probability distribution of different possible benefit levels, even if such numbers are inherently subjective. It seems unlikely that intuitive decisionmaking will produce better results than more rigorous approaches. Scientists may worry that quantification would discourage investments in basic research relative to more applied research. The reverse seems likely to be true. The more foundational the research, the greater the potential benefits to which it may contribute, and this factor seems likely to outweigh the fact that any single highly theoretical experiment may provide only a small bit of progress. Whether I’m right or wrong about this, allocation decisions ideally should be based on rigorous analysis of this question, or at least on moderately developed back-of-the-envelope calculations, rather than on pure intuition.

One objection is that any system that the government or indeed any bureaucracy develops for making more mathematically rigorous assessment of grants may be flawed by ignoring important criteria that scientists may take into account implicitly. But it need not be government that is charged with making these estimates. An alternative to the grant system would flip government’s role to ex post evaluation of benefits and costs. Twenty-five years from now, it should be much easier for scientists to assess the relative benefit of experiments conducted today. Instead of grants, the government could place grant money into a prize fund, let it accumulate interest, and distribute the money later. This approach would give private parties, akin to venture capitalists, incentives to anticipate the benefits of research. At the least, such parties should be less risk averse than the grant agencies that Kolata describes.

This may seem too radical a change from our existing system of scientific funding. But it is possible to integrate a modest version of this system within the existing grant system. For example, we might set aside just 10% of current grant money for a prize fund. Private parties would be required to auction their rights to any prize to independent third parties, conditional on the grants being approved. The grant agency might then consider the results of the auctions, in addition to any information they ordinarily would consider. At the least, this could help provide the grantors cover for approving low-probability, high-benefit projects. Moreover, the practices of the third parties-What kind of models do they use? What kind of disclosure do they expect from grant applicants?-might help us identify how we could improve the government’s own procedures. Whether or not the auction participants do a better job than the government (and with relatively small stakes, they might not), the types of projects they select with their own money on the line could help inform the government about what its decisionmakers’ biases might be.


Testing the Public Option

One of the most contentious issues in the current health reform debate is whether a reform bill should include a “public option,” allowing individuals and/or businesses to choose the United States as their insurer. Paul Krugman, who believes that government can deliver health care more cost-effectively than the private sector, insists that providing an additional option can’t do any harm and could do a lot of good. Greg Mankiw counters that if taxpayer funds are used to cover shortfalls, now or in the future, then the government option has a built-in advantage that could lead to universal coverage. Because in the absence of taxpayer support, the United States would essentially just be a nonprofit health plan, Mankiw argues that the public option can’t do any good and could do a lot of harm.

Mankiw likes challenging Krugman to bets, so let me propose one that could serve as a compromise on the public option issue: an experimental test of the public option. Here’s how it would work: Legislation would create a public option for a test period, perhaps three or five years. Some number of randomly selected individuals and businesses (a large number for statistical comparison purposes), but would be given some up-front cash incentive if they agreed to be randomized into either the public plan for the test period or into their choice of private plan.

An agency independent of the health agency (with, ideally, politically balanced membership) would distribute detailed health surveys to participants, particularly at the end of the period, and conduct a comparison assessment. The purpose of the surveys would be to compare the public option with all private plans (not to compare private plans with each other, especially because this would increase incentives to manipulate data).

The agency’s task then would be to compare benefits and costs. Benefits can be measured by assessing the health status of the participants. (See chapter 3 of this excellent (and free online) book for a detailed study of health impact measurement.) Ideally, the legislation would provide some indication of a scale along which individual health would be measured, and an indication of how to convert these health measurements into dollar values. The surveys and measurement would be costly, but these measurements could produce useful information quite apart from resolving the bet, by producing publicly available data, with appropriate privacy protections, that could be quite useful for research purposes. One might also factor the variance of health status by income group into the measure of total benefit, so that an approach that produces more equitable health outcomes is counted as producing greater benefit.

Meanwhile, the costs should include the sum of all private money and government subsidies spent on providing health care for both the public option and the private option, respectively. (If a participant in either plan ends up switching to Medicaid or Medicare, which would be separate from the public option, we would count these expenses to the plan to which the person was originally assigned, to reduce the danger of bias from nonrandom attrition or crossover.) The cost measure would also take into account the time costs to participants in medical visits, filling out forms, and the like.

The legislation should specify changes that would automatically occur based on the results of the experiment. If either the public or private plan produces better health at lower costs, the comparison is simple, but if one turns out to provide better outcomes for a greater cost, the legislation could include a formula determining whether the additional cost would be justified. If the public plan performed better according to the specification, then there would automatically be some expansion in the public plan at the end of the test period, for example by increasing subsidies for the public plan. If the private plan had a lower ratio, then there would be a contraction or elimination of the public option. A more nuanced approach might calculate cost/benefit ratios for different groups, based on either demographic characteristics or initial health status, and automatically result in more fine-grained modifications of the reforms.

The experiment would thus be self-executing in the sense of automatically changing the policy baseline. Of course, it would be possible to argue about the appropriate interpretation of the results even after the experiment. But it’s useful to force legislators to craft ex ante measurements and procedures for assessing and automatically changing policies. This saves the legislature from the enactment costs of subsequent changes and forces the legislature and the public to focus on what it is actually trying to achieve. Moreover, the possibility of such a public experiment should make health reform more palatable to both those who are optimistic and those who are pessimistic about the public option. As long as each side genuinely believes what it is arguing, then each side should expect the experiment ultimately to produce a reform more to its liking.


Semantic search and government support of AI

The last few weeks have seen the introduction of several search engines that boast “semantic search” capabilities. That is, they do not just look for keywords using statistical formulas, but seek to find meaning in the search phrases and in at least some content on web. The most publicized of these include Wolfram-Alpha, Google Squared, and Bing, (Microsoft’s new search engine, whose semantic search abilities are limited to specific domains, like shopping and travel).

Today these sites are extremely limited in their ability to parse user queries. For example, one of Wolfram-Alpha’s sample queries is “gdp spain/italy”, showing that it can combine and compare different types of data. But small changes stump Wolfram-Alpha. For example, it can produce a graph of “gdp per capita spain,” but not a graph of “gdp per capita growth spain” or “growth in gdp per capita spain.” Maybe there is some way to do this, but it’s not a clever interpreter.

Meanwhile, Google Squared is fun, but not terribly useful. Type, “Mets players,” and receive an interesting list: some historic greats — David Cone, Nolan Ryan, Darryl Strawberry, and Howard Johnson — as well as some strange choices, and an amusing mistake, a link to a concert by the other Kenny Rogers (a type of mistake that can occur only with semantic search). The most entertaining feature is that you can suggest some items in a series, and it will try to find more. For example, it didn’t know of any “steroid users,” but once I suggested Barry Bonds, Roger Clemens, Rafael Palmeiro, Manny Ramirez, and Alex Rodriguez, it helpfully added Sammy Sosa and Derek Jeter (who knew!) to the grid.

Overall, these sites both showcase some achievements of artificial intelligence, while also highlighting how far we are from truly revolutionary products. The existence of such products might seem to support conclusions that there is little need for active government promotion or subsidy of artificial intelligence research, but I draw the opposite conclusion. Artificial intelligence is a technology that today produces modestly useful products, but if it were possible to create a more powerful general reasoning system that could be a technology with tremendous social value. Many lesser artificial intelligence goals, such as robust computer vision, also would be extremely valuable.

But because these goals are unlikely within the patent term of any new technologies (assuming that software even remains patentable!), private incentives for such technology development are suboptimal. Thus, if further technology development is cost/benefit justified, then there is a case for governmental support of basic research in artificial intelligence. My own intuition is that there is sufficient warrant for optimism about artificial intelligence in the next 20 to 40 years that existing levels of government spending are too low. Admittedly, projections of future technology development are speculative, so it’s hard to be sure. One thing that one can be more sure about: the government doesn’t do a good job thinking systematically about what social returns we can expect from investments in different types of basic research, from bio to energy to nanotech. Funding priorities are mostly a result of politics, which corresponds only loosely to actual need.


The Unmentioned Issue in Bilski

The Supreme Court this morning granted certiorari in Bilski v. Doll (No. 08-964). The case presents an opportunity for the Court to consider issues of patentable subject matter, particularly whether business methods and financial patents should be patentable. I’ll probably have more to say about this issue soon, because I am currently working on an article tentatively entitled “Reconceptualizing Business Method Patents,” which argues that business method patents have a theoretically sound place in patent law.

But for now I thought I’d address an issue that to my knowledge has not received attention in the Bilski briefing, and that the Supreme Court did not mention in the question presented: whether the Board of Patent Appeals, whose decision the Federal Circuit was reviewing below, properly had jurisdiction. Two years ago, my frequent coauthor John Duffy (writing solo) argued that panels of the Board including judges appointed after March 29, 2000, were unconstitutionally constituted. These judges were appointed by the Director of the Patent and Trademark Office, who serves under the Secretary of Commerce. Under the Appointments Clause, the authority to appoint inferior officers may be vested “in the President alone, in the Courts of Law, or in the Heads of Departments.” Arguing that the PTO director is not a head of a department and that the administrative patent judges are inferior officers, Duffy concluded that these judges are unconstitutional.

The arguments were sufficiently persuasive that Congress amended the statute, so that the Secretary of Commerce can appoint patent judges. The problem that this leaves is the question of retroactivity. The statute addresses this issue by allowing the Secretary to make retroactive appointments (which he did), and as a fallback, by providing, “It shall be a defense to a challenge to the appointment of an administrative patent judge on the basis of the judge’s having been originally appointed by the Director that the administrative patent judge so appointed was acting as a de facto officer.” Meanwhile, the Federal Circuit has so far managed to sidestep the issue, by holding that the issue is forfeited if not raised before the Board of Patent Appeals itself.

There seems to me a strong argument for the Court to vacate the decision on the basis of this problem. Even though the issue does not appear to have been raised in Bilski, the Board panel appears to have been unconstitutionally constituted, and it is of course the duty of the Court to inquire into its subject matter jurisdiction. It seems doubtful that the Court would approve of retroactive appointments, and the applicability of the de facto officer doctrine is questionable. In Ryder v. United States, the Court refused to apply the de facto officer doctrine, reasoning that there was not merely a misapplication of an appointments statute, but an unconstitutional appointments statute.

Conceivably, the Court might agree with the Federal Circuit’s view that the issue has been procedurally defaulted. As the Federal Circuit has noted, when the Supreme Court has agreed to hear appointments challenges despite waiver issues, it has indicated that it was doing so in its discretion, and it has never held that waiver is generally excused. But I do not know of any case (though I have not researched this fully) in which the Supreme Court has granted certiorari on other issues, confirmed that it really does have discretion to ignore a known issue whether judicial decisionmakers were unconstitutionally appointed, and indicated that it was exercising its discretion to ignore the potential infirmity. The Supreme Court’s past characterization (possibly in dicta) of the procedural default issue as nonjurisdictional seems in some tension with the venerable Capron v. Van Noorden, which held that defects of a lower court’s subject matter jurisdiction could be raised for the first time at the Supreme Court.

It seems to me that in any event, the Court has an obligation at least to inquire into its subject matter jurisdiction, for example by holding definitively that it can exercise jurisdiction over a case in which Board members may have been unconstitutionally appointed if the issue was defaulted. It might have been more prudent to wait for another case in which to grant certiorari. (A separate reason for this is that there is a strong argument that the patent at issue in Bilski was obvious, but this is not relevant to the subject matter jurisdiction issue.)


Gamble on Gambling

Keith Jacks Gamble has written a super paper on the Super Bowl. After each play in the Super Bowl, he noted the “bid’ and “ask” prices for a share on Tradesports that would pay off $10 if and only if the Colts won the Super Bowl. He used the midpoint of those prices to derive probability estimates, making it possible to assess, at least based on the market’s wisdom, how each play made more or less likely a Colts victory.

Perhaps someday sports pages will routinely post graphs of trading prices to summarize the games of the day before — especially if annotated with plays (as in this graph of the Super Bowl on Midas Oracle), it’s an excellent way to get a quick overview of what happened and what mattered.

I’m a bit more skeptical, however, of one use that Gamble makes of the data. He uses the prices to assess individual player performances. For example, Devin Hester is given 10.25 percentage points for his opening kick return for a touchdown. With this methodology, Gamble concludes that Bears QB Rex Grossman contributed 36.5 percentage points to the Colts winning.

Certainly, Grossman had a very bad day. But there are at least two reasons to be skeptical of this approach as a general way of assessing player contributions: (1) The simultaneity problem. Football is a team sport, and it is difficult to disaggregate all the players’ contributions. My hunch has always been that behind a great quarterback is a great offensive line. (2) The anticipation problem. Estimates of a particular player’s ability is already impounded into market prices. If Tom Brady were leading a last minute drive, a market might assume that he’ll probably be successful because he’s Tom Brady, thus understating the extent of his contribution. A better approach may be to give credits based on how different events contribute to winning in general. (See the Protrade markets, for examples of this approach.)

Gamble is also doing some very interesting serious research about how bounded rationality in financial markets can cause cascade behavior. I’d be curious to know also how seemingly random factors affect trading patterns — in the Super Bowl, did trading volume or direction change depending on how funny the ads were?

For those who have no idea what the relation of this to law, you’ll have to wait until my book, Predictocracy: Market Mechanisms for Public and Private Decisionmaking, comes out in the fall. In the meantime, thanks very much to Dan Solove for inviting me to guest blog! I enjoyed my time here, and very much appreciated all of the thoughtful comments on my posts.


Recovering from collapse

Robin Hanson has written a characteristically interesting article about the possibility of human extinction. And with characteristic understatement, Hanson notes that “[a] disaster large enough to kill off humanity … should be of special concern.” Indeed. Hanson’s point, of course, is that wiping out all of humanity is much worse than wiping out almost all of humanity.

Hanson appears to worry about extinction in part because, he observes, disasters sometimes follow a power law distribution in the destruction that they cause. This suggests that in expected value terms, we should perhaps worry as much or more about disasters that wipe out humanity as about disasters on the scope observed in past human history.

Extrapolating from some assumptions, Hanson suggests maybe there is a one in three million chance per year of an event that would kill everyone (perhaps not instantaneously, but in a gradual collapse, as the failure of some social systems lead to the failure of others). My inclination is to agree with Hanson that the danger of extinction is sufficiently severe that it’s worth worrying about. But I wouldn’t rely too much on extending a power law distribution beyond previously observed events. That may actually understate the danger of extinction, which seems to me to be very low but a lot higher than one in three million. In part, this is because modern technology creates many scenarios for catastrophe (nuclear war, superviruses, nanotech gray goo, the possibility that the Knicks could keep Isiah Thomas after this year) that could not have occurred hundreds of years ago.

One argument that Hanson makes is that it might be useful to establish refuges to ensure that if a disaster occurred, at least some small number of people (perhaps 100) would survive. Eventually, these people could return to a hunter-gatherer lifestyle, eventually develop the capacity to communicate innovations, and then within maybe a mere twenty thousand years, a blip of cosmic time, returns to where we are now. While there is value in such an approach, government policy might not make a big difference. If some calamity is strong enough to prevent the survivalists from surviving, it would seem hard to believe that government-produced sanctuaries would do much better. It seems a narrow class of extinction events that would kill even those who, from the perspective of most observers, have absurdly exaggerated estimates of the probability of catastrophe while sparing a government project.


Other experts needed to cool things down

Now that an international group of climate scientists predicts confidently that global warming will occur, Scientific American’s David Biello concludes that “the science of climate change may partially undergo a shift of its own, moving from trying to prove it is a problem … to figuring out ways to fix it.”

But there is at least one other step that should be taken too, and that is to try to achieve some economic consensus on the best measures to take. There remains serious debate among economists looking at the costs and benefits of various global warming mitigation strategies. (See, for example, this review by one economist of another’s work.) We should be just as wary about the danger of junk economics as we should be about the danger of junk climate science. Economists should take the assumptions of climate scientists as a given, and give bottom line assessments of the costs and benefits of different strategies. My instinct is that major policy changes would be recommended, but if it turns out that the costs of mitigation are greater than the benefits, we should accept that too.

Ideally, we should convene other expert groups as well — such as experts on agriculture, oceans, and on and on –so that their numbers too can be plugged into economists’ calculations. Some important questions, unfortunately, don’t necessarily suggest expert groups uniquely qualified to answer them. For example, we should probably predict the political dislocations from agitation due to global warming, but it’s hard to know whose models are best suited to making such a forecast.

Other individual questions probably need many different types of experts. We should seriously consider geoengineering strategies such as pumping sulfates into the stratosphere, even though such consideration might well result in a conclusion that these strategies would not be worth undertaking. Each of these postulated strategies probably demands various groups of expert assessors, including climatologists, economists, materials scientists, and even they may have an almost hopeless task given the difficulty of predicting the feasibility of technological solutions many years from now.

Of course, even if we can figure out the best strategy incorporating all relevant perspectives, we then need to figure out which second-best strategies are worth taking given political constraints, such as countries that won’t cooperate with an effort to reduce carbon emissions. In the end, we can throw all the experts in the world at this problem, and we should, but the ultimate decisions will be made by our very imperfect political systems, so rational responses are probably too much to hope for.


Prizes, pieces, and property rights

David Leonhardt’s Economix column today suggests that prizes may be a useful way of stimulating innovation. His primary example is’s million dolar prize for any one that can improve its movie suggestion algorithm by 10%. The current best team is at 6.75 percent. Netflix may be better positioned than most companies to be able to offer prizes that will provide the winners reputational benefits that may exceed the value of the prize itself, but Leonhardt may still be right that prizes are an often overlooked means of accomplishing corporate goals.

The prize is an example of “crowdsourcing.” The web site Innocentive contains many such crowdsourced offers. If, for example, you can develop a pressure sensitive adhesive for re-sealing flexible bags for salty snacks (the adhesive must not adhere to potato chips or hands reaching in to take out chips), you can make $50,000 (plus the eternal gratitude of me, a Doritos connoisseur). An advantage of this approach over conventional sourcing is that the project sponsor does not need to assess the quality of those who may work on the project.

Why do we not see more crowdsourcing via prizes? See my comments after the jump.

Read More


Market mysteries: The case of Extra Innings

Major League Baseball is reportedly entering a deal that would shift its Extra Innings product, which has been available to up to 75 million customers, to being available exclusively on DirecTV, which currently has only 15 million subscribers, for the next seven years. My primary reaction to this has been genuine sadness. Watching baseball games is my number one hobby, and my house can’t get DirecTV signals because of nearby trees. It did occur to me that if I chopped down my neighbors’ trees, I would probably do a year in jail, which would leave me six years to enjoy the games. More likely, I’ll have to find a new hobby besides watching baseball. Other alternative approaches to following the Mets — going to a sports bar, watching on my laptop — just won’t cut it.

But I’m also intellectually puzzled. How is it possible that it ends up being more profitable for MLB to sell Extra Innings as an exclusive franchise? Even putting aside possible loss of fans and thus revenue on other products (such as tickets), I would have guessed that whatever MLB could have received in nonexclusive deals for 75 million customers would be greater than what MLB could receive in an exclusive deal for 15 million customers. Obviously, that guess would have been wrong. What explains this?

A partial explanation is that the subscriber base for DirecTV is not fixed. If all cable Extra Innings subscribers could be expected to just switch over to DirecTV, then the initial subscriber populations would be irrelevant to the revenue calculation. But many people won’t — either because they (like me) can’t get satellite, or because they have some preference for cable over satellite. So, on reasonable assumptions, the Extra Innings subscriber base will be much lower in the future — and yet DirecTV seems to be able to pay more than everyone combined in a nonexclusive arrangement.

The answer to this market mystery probably has to do with branding. DirecTV expects to have a hipper brand by virtue of its exclusive deals on MLB Extra Innings and NFL Sunday Ticket. The exclusive contract thus sends a signal to consumers. I suppose that this could be an efficient result if consumers somehow underappreciate the virtues of DirecTV, or if consumers who still buy Extra Innings will value it more because others don’t have it. But I’m more inclined to think that the property rule protection that MLB has for its copyrighted shows leads to an inefficient result here, even if one that genuinely benefits MLB and DirecTV.

I generally believe in property rights, but this deal is creating a personal crisis for me that is making me challenge my views. Should the law in some way seek to discourage such deals?


Eat your broccoli and win a Nobel

Matthew D. Rablen and Andrew J. Oswald have written a very interesting paper comparing the life spans of Nobel Prize winners and individuals who were nominated but didn’t win. (Hat tip to the Economist.) They conclude that winning a Nobel confers about one or two years of extra longevity relative to being merely nominated for one.

The paper is admirably careful. For example, it exploits variation in the amount of purchasing power provided by the Nobel, factoring in cases in which someone wins only a portion of a Nobel, as a way of teasing out the possibility that extra longevity is a result of the money provided by a Nobel. In the end, the paper certainly seems to boost the conclusion that status is important for longevity, and more broadly, that status is something separate from money that people care greatly about.

Nonetheless, there is reason to be skeptical about results like these. Maybe the population of Nobel winners differs from the population of Nobel nominees. Perhaps the Nobel winners had better health than the also-rans, and this allowed them to do extra work that led to the Nobel, or maybe they had more or fewer children, or maybe they were smarter, or maybe they had better home lives, or just had better genes. The problem, of course, is it’s not feasible to control for all of these things, plus the many that one might not think of.

Here’s a suggestion for a study: How does winning a seat in Congress (or any other legislative body) affect longevity? An advantage of this study is that one might be able to exploit random variables that affect the probability of winning independent of the characteristics of the candidate, such as recent economic growth rates at the time of the election. In the second stage of the regression, the dependent variable would be the expected probability of victory given existing conditions exogenous to the candidate. I realize that it’s hard to assemble the data, though.