Search Neutrality as Disclosure and Auditing
posted by Frank Pasquale
Search neutrality is on the rise in Europe, and on the ropes in the US (or at least should be, according to James Grimmelmann). We barely have net neutrality here, and the tech press bridles at the thought of a sclerotic DC agency regulating god-like Googlers. I want to question its conventional wisdom, by proving how modest the “search neutrality” agenda is now, and how well it fits with classic ideals of neutrality in law.
There are many reasons to think that Google will continue to dominate the general purpose search field. Sure, searchers and advertisers can access a vibrant field of also-rans. But most users will always want a shot at Google for serious searching and advertising, just as a mobile internet connection is no substitute for a high bandwidth one for many important purposes.
Given these parallels, I’ve compared principles of broadband non-discrimination and search non-discrimination. But virtually every time the term “search neutrality” comes up in conversation, people tend to want to end the argument by saying “there is no one best way to order search results—editorial discretion is built into the process of ranking sites.” (See, for example, Clay Shirky’s response to my position in this documentary.) To critics, a neutral search engine would have to perform the (impossible) task of ranking every site according to some Platonic ideal of merit.
But on my account of neutrality, a neutral search engine must merely avoid certain suspect behaviors, including:
1) Stealth marketing (secretly taking cash or other consideration in exchange for elevating the profile of sites in organic search results)
2) De-indexing without notice and explanation (removing legal, non-spam sites from the index after they have been included in the search engine’s corpus, and failing to give some explanation to the removed site as to why it was removed)**
I think my concept of neutrality is much closer to the way the term is normally used in political philosophy (where, say, a “neutral state” is one that does not favor any particular conception of the good, rather than one that accords exactly the right amount of respect to each conception of the good). Neutrality is a very broad term, and the obvious differences between the technical operation of physical infrastructure and search engines should not stop us from applying certain broad principles to each entity.
Neutral State, Neutral Search Engine
What do we mean when we talk about search neutrality? Opponents of net neutrality have called the term too vague, identifying “31 flavors” of neutrality to support any ideological commitments under the sun. But Michael Powell’s “four freedoms” have proven relatively uncontroversial. I think a similar consensus will coalesce around stealth marketing and de-indexing. A brief review of how one insightful political philosopher conceptualized neutrality may help get us there.
In his “The Ideal of a Neutral State” (in this collection), Peter Jones helps us understand key controversies in conceptualizing neutrality.* His central insight is a recognition of the familiar, “ordinary language” sense in which neutrality is used. The “question of neutrality often only arises in a conflict; a commitment to neutrality indicates a willingness to help or hinder parties to an equal degree.”
By examining a list of examples of such conflicts, Jones demonstrates that “being neutral” can take different forms in different contexts. For instance, a nation may have to refrain from helping or hindering either side in an international conflict if it is to remain neutral; a judge may have to ensure that there are strict guidelines for the presentation of evidence if he is to be seen as a neutral arbiter of a fair decisionmaking process. Based on examples like these, Jones argues that there are two senses of neutrality, negative and positive. Whereas negative neutrality can be achieved simply through inaction, positive neutrality “is a matter of establishing certain conditions of equality among individuals, conditions which ‘neutralise’ certain factors that might otherwise enable one individual to fare better than another.”
Does a search engine have a duty of “positive neutrality?” No, but much of what it does amounts to a similar effort. If some scheming company starts “link farms” to make its sites more visible, it should be punished. It’s gotten an unfair advantage. We can, in general, count on the search engine to promote its users’ interests by detecting and deterring that kind of advantage-seeking.
But there are many other situations where a search engine’s interests may coincide more with those of its best advertisers, rather than its users’. As Brin & Page stated, “we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.” We’ve been in this situation repeatedly in the communications context, and the neutral state has come up with a solution: require both conduits and content providers to disclose whether they are raising the profile of those who pay them. Stealth marketing is unfair to consumers and to competitors of the stealth marketer. The state realizes that money already confers an enormous advantage in the battle for mindshare, and requires, at the very least, that such advantage be disclosed when it is bought. It is right to neutralize certain factors (such as sub rosa payments) that might enable one source to fare better than another.
Opaque Search Technology is a Widespread Problem
Search engines are only one of many intermediaries that use opaque search technology. Moreover, to the extent search engines become some magical category of zero-regulation, expect other entities to incorporate search technology to obtain the same advantages that search engines have. The regulatory arbitrage game is just too easy to play.
Given these looming problems, some entity should be able to audit the systems used by any dominant intermediary to find out if stealth marketing and de-indexing without notice is happening. I don’t necessarily care if the entity is public or private—we just need some group to be able to “look under the hood.” And if anyone says “it’s just too complex to explain or understand,” consider the kind of black box future that position portends.
Looking ahead, I don’t actually think this disclosure remedy will do much to change consumer or advertiser behavior. Dominant search engines, online retailers, device makers, and social networks have enormous advantages over rivals now, and as Tim Wu has shown, a “Cycle” of early cutthroat competition has repeatedly congealed into oligopoly in the communications and media fields. But even ex post disclosure will at least allow future historians, academics, and policymakers to understand how our information environment was shaped.
In the long term, we’ll probably need to have some publicly funded alternative to dominant internet intermediaries, just as we have a Postal Service to serve those who can’t afford FedEx, or Medicare to serve those who are too old or disabled to get reasonably priced private insurance, or arts subsidies to help forms of expression that the market will never underwrite. Disclosure helps us understand how urgent the need is for an alternative.
*Peter Jones, “The Ideal of the Neutral State” in Robert E. Goodin and Andrew Reeve (eds.), Liberal Neutrality (Routledge, 1989).
**My work in the article “Asterisk Revisited” has also led me to believe that a trademark holder should appear somewhere on the top results page when its trademark, without more, is a search query. But I don’t believe current law requires that, and I don’t want to clutter this post with what would certainly be a complex discussion of trademark policy.
Image Credit: Steve Rhodes.
February 19, 2011 at 11:16 am
Posted in: Cyberlaw, Google & Search Engines, Intellectual Property, Philosophy of Social Science
Print This Post
Responses (11)
James Grimmelmann - February 19, 2011 at 11:45 am
A few thoughts.
First, your search neutrality agenda may be modest, but others’ is not. Consumer Watchdog disapproves of the integration of other Google results, such as local search and maps, into the main web search engine. Adam Raff’s NYT op-ed called for search results to be “comprehensive, impartial and based solely on relevance.” Dawn Nunziato has argued that Google should be required to accept keyword ads from political candidates. If you believe that your narrower concept of “neutrality” is the right one, then I take it you’ll join me in saying publicly that these other proposals are bad ideas?
Second, your proposal really just consists of transparency. You’re opposed to stealth marketing and secret deindexing, but take away the “stealth” and the “secret” and it would seem you’re fine with marketing and deindexing. If so, why call this “neutrality?”
Third, your invocation of positive neutrality begs the crucial normative question: which factors should be neutralized for? You have a view that content farms are deviations from a baseline of true equality, but where are you getting that intuition? Contrast that with my preferred theory of the normative goals of search: relevance for users rather than equality among sites. Relevance gets you to the same result — content farms are bad — without needing to specify a contested, undefined, impossible baseline.
I’d like to talk to you about why the idea that search engines should strive for a kind of neutrality or objectivity in their methodology has such a persistent grip on the imagination. I have some thoughts along those lines that may provide a way of rescuing at least some of those ideals (while also recognizing their dangerous undercurrents), but I don’t think that Jones’s approach really gets at the concept we’re looking for.
Frank Pasquale - February 19, 2011 at 12:55 pm
@James:
a) As for the proposals, some off-the-cuff thoughts:
1) “Consumer Watchdog disapproves of the integration of other Google results, such as local search and maps, into the main web search engine:” Bad idea re local search and maps, especially” b/c they are so good. But I would hope that the company might follow Alex Halavais’s advice in the comment on this post:
http://www.concurringopinions.com/archives/2010/07/the-decline-of-media-studies-and-privacy-in-a-search-engine-society.html
“placing that instrumentality in the hands of the user.”
2) “Adam Raff’s NYT op-ed called for search results to be “comprehensive, impartial and based solely on relevance:”” No, if an SE wants to do narrow, partial results, I’m fine with that, as long as it’s disclosed.
3) “Dawn Nunziato has argued that Google should be required to accept keyword ads from political candidates.” I would not go that far now, but if surveys found that people were getting, say, 50% of their knowledge about local elections from Google, I’d be far more sympathetic.
b) As for “why call this ‘neutrality?’ I think people could have said much the same about Jerry Kang’s deontological concerns in his Race.net Neutrality article, or Paul Ohm’s effort to include privacy in the net neutrality debate. Neutrality is an essentially contested concept. When Kang and Ohm brought up their concerns, I’m sure lots of antitrust-oriented people were very frustrated that they could not continue the debate in a purely economic, consumer-welfarist mode.
There are dominant players at many layers of the web, and some responsibilities coalesce once they’ve attained a certain critical mass of power. I could just say “be transparent,” but one also has to have a theory of *why* transparency is necessary. Neutrality is a good term to encompass the ensemble of concerns surrounding stealth marketing, wrongful de-indexing, qualified transparency, and auditing.
c) as for “begging the normative question:” I’m pointing to a long history of stealth marketing law, reaffirmed as recently as last year in FTC guidelines for bloggers’ disclosure of swag from the makers of products they review.
d) as for the “grip on the imagination:” I recently heard a discussion of IBM’s Watson, where a promoter of the technology bragged that “computers will be solving problems in their own ways, in ways that no one can understand. They’ve found their own way to solve the problem.” I think that’s really great if the problem is finding a cure for cancer or 100 mpg engine. I don’t think it’s so great if the problem is deciding which books are most relevant to a given query, or which people get credit, or who is on the TSA watchlist.
The call for objectivity and transparency arises because people are justly cautious about injecting techniques for managing and controlling the natural world into the social world.
James Grimmelmann - February 19, 2011 at 1:11 pm
Re: (b), Kang and Ohm help us understand what values are at stake in the net neutrality debate. Their pieces are not primarily or even incidentally, about what “neutrality” itself in “net neutrality” consists of. That has, at least to a first and second order approximation, purely technical definitions. Whether enforcing that particular technical standard of nondiscrimination at the network, link, and physical layers is a good idea is a difficult question that a broader perspective helps us with. In contrast, calling setalth marketing, “wrongful” de-indexing (as though anyone had a meaningful theory of which de-indexing is “wrongful”), transparency, and auditing “neutrality” hitches these concerns to each other in a way that obscures, rather than illuminates.
Re: (c), stealth marketing is quite far from “a matter of establishing certain conditions of equality among individuals, conditions which ‘neutralise’ certain factors that might otherwise enable one individual to fare better than another.” Jones’s theory, as you describe it, is about substantively correcting for certain kinds of inequality. The stealth marketing concern is not about equality or neutrality among advertisers; it is about consumer autonomy.
prometheefeu - February 19, 2011 at 10:06 pm
All of those bad behaviors would degrade search results and give an advantage to competitors. So competition does provide actual incentives to Google. Regulation is unnecessary. It’s not like isps that have huge starting costs and are a natural monopoly.
Adam Raff - February 20, 2011 at 8:57 am
As a non-lawyer, I will probably regret even dipping my toe here, but I wanted to point out that our search neutrality agenda seems pretty modest and achievable to me. As an advocate of vertical search, you shouldn’t be surprised to hear that I don’t have a problem with a search engine narrowing its results along disclosed lines either. As we said last year:
“Search neutrality is not a call for Government-mandated ‘standardised algorithms’. Foundem defined search neutrality as the principle that search engine results should be comprehensive, impartial, and based solely on relevance. Clearly, no two search engines will produce the same search results; nor should they. But any genuine pursuit of the most relevant results must, by definition, preclude any form of arbitrary discrimination. Google’s Universal Search and its increasingly heavy-handed penalty algorithms are both clear examples of arbitrary discrimination.” SearchNeutralty.org, November 2010.
Again, I’m no lawyer, but I suggest that the word “relevance” in our definition ought to preclude many of the more pedantic arguments: clearly Spam is only relevant if you are looking for Spam, and a specialist search engine for real estate wouldn’t be doing its job if it didn’t set out to exclude cake recipes.
It might be helpful to provide some insight into why we founded SearchNeutrality.org.
In September 2009 we were in Brussels meeting with policy makers and regulators (in a process that ultimately led to us filing our European antitrust complaint against Google). During these meetings it became clear that network neutrality was high on the digital agenda in the US and Europe, but that the potential risks of search engine bias were not on the radar of politicians or regulators.
The European Internet Foundation (EIF), for example, had recently published its Digital World in 2025 report. This report, laying out the EIF’s predictions for the digital challenges of the next fifteen years, failed to mention search engines as a source of any concern. This, despite the increasingly vital role played by search engines in steering traffic and revenues through the global digital economy, and the fact that a single US corporation has an 85% global market share of this crucial channel to market.
The penny dropped that by framing our concerns under the, highly appropriate, label “search neutrality”, we could tap into the considerable educational effort that had been invested in the very similar concerns of network neutrality over the years.
I registered the domain name searchneutrality.org on the train back from Brussels.
Search Neutrality isn’t an academic principle to us – it is a framework for addressing one of the greatest threats to diversity and healthy competition the Internet has ever faced.
In Foundem’s EC Complaint, filed in February 2010, we argue that Google is leveraging its dominance of horizontal search into adjacent markets in ways that stifle innovation, suppress competition, and erode consumer choice. Foundem’s Complaint is essentially in two halves. The first concerns Google’s increasing use of exclusionary automated penalties, which can remove legitimate sites from Google’s natural search results, irrespective of relevance. The second concerns Google’s Universal Search mechanism, which is transforming Google’s ostensibly neutral search results into an immensely powerful marketing channel for its own services.
Seth Finkelstein - February 20, 2011 at 12:47 pm
FYI, just now:
http://www.theregister.co.uk/2011/02/18/google_on_manual_search_penalties/
Google opens curtain on ‘manual’ search penalties
Anonymous Coward - February 20, 2011 at 3:23 pm
It was always my understanding that “search neutrality” was originally retaliation on behalf of AT&T and other ISPs to Google’s strong advocacy of network neutrality. Hence why they called it “search neutrality” instead of search transparency or the like, and why the target is always Google rather than e.g. Microsoft despite that Microsoft’s Bing behaves in much the same ways and is the default search engine in Microsoft’s dominant Windows operating system and Internet Explorer web browser.
It seems now that some people are discussing the idea seriously. I don’t mean to suggest that everyone currently advocating it is astroturfing, or even that the idea is completely without merit (although the antitrust complaint seems rather over the top), but it makes me wonder about how these kind of topics come to be the subject of public scrutiny. Would we be having this discussion if the issue had not been pressed by ISPs?
Really what concerns me is the message it sends to corporations that act in the public interest. If advocating the public interest on an issue like network neutrality can so easily result in retaliation in a form that leads to an antitrust complaint, it makes corporations far less likely to advocate in the public interest in the future, even where the public interest approximately aligns with their own. Which is plainly undesirable when the public has so few powerful advocates. Perhaps some kind of sanctions for this sort of retaliation are in order — but how can it even be detected? It isn’t as though we have existing high-accuracy methods of distinguishing well-researched propaganda from legitimate public concerns.
I suppose the best answer may be for academics and policy makers to remain on their toes as to whether an issue is popular because it has public support or because it has corporate support, and make sure that when the answer is the latter it receives exceptional scrutiny to ensure that silver-tongued lobbyists are not setting the agenda.
Frank Pasquale - February 20, 2011 at 3:50 pm
@Anon Coward: I have appreciated many of your contributions. But I must dispute certain implications of this comment.
My first work on search engine bias was published in 2006, before “search neutrality” became an issue. Nissenbaum and Introna were worried in 1999. I also predicted in 2008 what we are seeing today: more moves toward vertical integration or other deals between SE’s and carriers. All the players here are more “frenemies” than life-or-death antagonists.
I’ve written tens of thousands of words of articles and blog posts on this topic, and I haven’t gotten a penny from any of the corporate players you suggest are “astroturfing” this. I’m not a paid consultant to anyone. I’ve criticized the ISPs more harshly than I’ve criticized Google. I am a critic of all unaccountable intermediary power, and consider myself a completely independent voice here.
Finally, as shown in the links in this and other blog posts, the big issue is secret ranking algorithms generally. To frame this as “Googlephobia” or some sort of insider DC politics is to completely miss the larger point about the dangers of clandestinely commensurating computing. Here’s a nice intro to that topic:
http://www.theglobeandmail.com/news/technology/the-algorithm-method-programming-our-lives-away/article1815869/page4/
Anonymous Coward - February 20, 2011 at 6:29 pm
@Frank: I want to reiterate what I wrote above in that I don’t think everyone advocating this is paid off by lobbyists or that the discussion is completely without merit. You in particular were not who I had in mind.
But for example, it appears from its content that the website searchneutrality.org linked in the comment above has some connection to the Google competitor Foundem. This is hardly irrefutable proof that its content suffers from a bias toward harming their largest competitor, or more cynically that the website was set up in order to attempt to win public support for the antitrust suit, but it does seem to me rather suspicious.
I suppose what is bothering me from your post is the co-opting of “search neutrality” which doesn’t seem to very well match the sort of inscrutable black box algorithm concerns you’re expressing. If I understand your concerns correctly, the problem is more a lack of transparency than a need for government regulation of the content of search results. And so using the term “search neutrality” which I normally expect to be associated with “‘Googlephobia’ or some sort of insider DC politics” is at least to me somewhat confusing, and disheartening because it implies that lobbyists and corporate public relations departments have been effective in infecting the public discourse with their newspeak. Perhaps the term means something different to you.
Seth Finkelstein - February 20, 2011 at 7:41 pm
@Anonymous I’ve become extremely cynical about net-politics. But while I agree with a follow-the-money angle in general, I think your analysis is off. Note, in every controversy, there’s a mix – sincere people who have been in area since before it got hot, camp-followers who go where the action is, hired guns who are pushing the agenda of their paymasters, and so on. Net Neutrality was/is a business fight between Google (and others) and telecomms. It was NOT “public interest”. It was Google, etc, not wanting to be worked-over by telecomms regarding bandwidth (and telecomms wanting to do so). Interestingly, the telecomms in fact did not retaliate against Google over search – this is clear because if you look at the lobbying fronts on that side, they were all talking about the horrors of “regulating the Internet”. There wasn’t any sort of big counter-campaign for Google-is-EVIL.
The money behind making “search neutrality” a serious issue (as opposed to obscure law/policy papers
) is I believe from Microsoft, doing it as a way to give Google some grief in the search market, where Microsoft is trying to compete.
But if you want to talk about “lobbyists and corporate public relations departments have been effective in infecting the public discourse with their newspeak.”, your first example might be their making “Net Neutrality” apparently mean anything bad done by a telecomm in any context.
I’m not sure what academics can do – the whole topic of turning the public interest into an argument over which big corporation wins, is beyond my ability to solve.
Anonymous Coward - February 20, 2011 at 9:16 pm
It is. At the same time, the class of companies represented by Google includes a variety of small tech startups who could be put out of business or otherwise significantly impacted by increases in the price or transaction costs of reaching their customers. Whereas the class of companies on the other side of the spat includes almost exclusively large, well-funded and highly profitable telecommunications and cable companies. It is in the public interest to avoid a wealth transfer from small and fragile startups to large utility companies, at least in my opinion.
That may very well be, Microsoft certainly has that incentive. My impression (and I could be wrong) was that, perhaps in addition to Microsoft, it was the ISPs hedging their bets. It is most powerfully in the ISPs’ interest to win the debate on “regulating the internet” so that they don’t have to abide by network neutrality. But by pushing “search neutrality” they make it more difficult for Google to advocate network neutrality, because if the ISPs can convince people that “search neutrality” is analogous to network neutrality and therefore that if one is necessary so is the other, then it may convince Google to stop advocating network neutrality so strongly to avoid the risk that “search neutrality” will be imposed along with it.
That has certainly been a problem. It actually makes the debate almost meaningless. For example, only the most radical advocates of “don’t regulate the internet” would think it should be acceptable for e.g. AT&T to block all packets detected as voice over IP unless the VOIP provider pays a fee so large that it would make them uncompetitive with AT&T’s voice service. It wouldn’t surprise me at all to discover that this is already illegal (perhaps antitrust?). But if that sort of regulation is a given then the question is not whether there should be network neutrality but rather what network neutrality means, and yet everybody is arguing about its propriety without first agreeing on what it is.
Leave a Reply