In my forthcoming article, Copyright and Copy-Reliant Technology, I investigate the significance of transaction costs in the context of technologies that copy expressive works for nonexpressive ends. These “copy-reliant technologies”, such as Internet search engines and plagiarism detection software do not read, understand, or enjoy copyrighted works, nor do they deliver these works directly to the public. They do, however, necessarily copy them in order to process them as grist for the mill, raw materials that feed various algorithms and indices.
Copy-reliant technologies usually, but not invariably, incorporate some kind of technologically enabled opt-out mechanism to maintain their preferred default rule of open access. For example, every major Internet search engine relies on the Robots Exclusion Protocol to prevent their automated agents from indexing certain content and to remove previously indexed material from their databases as required. A robots.txt file at the root level of a website in the form of: User–Agent:* Disallow: / will banish all compliant search engine robots from a website.
The Robots Exclusion Protocol is pretty easy to implement and it is highly customizable. The interesting question for copyright law is “does the provision of an opt-out make any difference?”
In the Article, I argue that it opt-outs are significant in the context of a fair use analysis. The doctrinal analysis is in the paper, but the basic point is that when transaction costs are otherwise high, opt-out mechanisms can play a critical role in preserving a default rule of open access while still allowing individuals to have their preferences respected.
The notion that the rights of the property owner can be protected under permissive default rules coupled with an opt-out is hardly new. Robert Ellickson famously describes the “fencing out” rule whereby cattle were allowed to roam freely on the property of others unless that property was fenced. Landowners still maintained their property rights, subject to the burden of fencing out neighbors’ cattle. Presumably, if cows could read, a sign not unlike the Robots Exclusion Protocol would have been sufficient.