Archive for the ‘Cyberlaw’ Category

Ubiquitous Infringement

posted by Derek Bambauer

Lifehacker‘s Adam Dachis has a great article on how users can deal with a world in which they infringe copyright constantly, both deliberately and inadvertently. (Disclaimer alert: I talked with Adam about the piece.) It’s a practical guide to a strict liability regime – no intent / knowledge requirement for direct infringement – that operates not as a coherent body of law, but as a series of reified bargains among stakeholders. And props to Adam for the Downfall reference! I couldn’t get by without the mockery of the iPhone or SOPA that it makes possible…

Cross-posted to Info/Law.

  February 27, 2012 at 2:14 pm   Posted in: Anonymity, Architecture, Culture, Current Events, Cyberlaw, DRM, Education, Google and Search Engines, Innovation, Intellectual Property, Interviews, Media Law, Movies & Television, Politics, Social Network Websites, Technology, Web 2.0   Print This Post   2 Comments

Stealing the Throne

posted by Derek Bambauer

Ever-brilliant Web comic The Oatmeal has a great piece about piracy and its alternatives. (The language at the end is a bit much, but it is the character’s evil Jiminy Cricket talking.) It mirrors my opinion about Major League Baseball’s unwillingness to offer any Internet access to the postseason, which is hard on those of us who don’t own TVs (or subscribe to cable). Even if you don’t agree with my moral claims, it’s obvious that as the price of lawful access diverges from the price of unlawful access (which is either zero, or the expected present value of a copyright suit, which is darn near zero), infringement goes up.

So, if you want to see Game of Thrones (and I do), your options are: subscribe to cable plus HBO, or pirate. I think the series rocks, but I’m not paying $100 a month for it. If HBO expects me to do so, it weakens their moral claim against piracy.

Unconvinced? Imagine instead that HBO offers to let you watch Game of Thrones for free – but the only place on Earth you can view the series is in the Kodak Theater in Hollywood. You’re located in rural Iowa? Well, you’ve no cause for complaint! Fly to LA! I suspect that translating costs into physical costs makes the argument clearer: HBO charges not only for the content, but bundles it with one particular delivery medium. If that medium is unavailable to you, or unaffordable, you’re out of luck.

Unless, of course, you have broadband, and can BitTorrent.

As a minimum, I plan not to support any SOPA-like legislation until the content industries begin to offer viable Internet-based delivery mechanisms that at least begin to compete with piracy…

Cross-posted at Info/Law.

  February 22, 2012 at 12:21 pm   Posted in: Architecture, Culture, Current Events, Cyber Civil Rights, Cyberlaw, DRM, Innovation, Intellectual Property, Legal Ethics, Media Law, Movies & Television, Politics, Technology, Web 2.0   Print This Post   55 Comments

Cyberbullying and the Cheese-Eating Surrender Monkeys

posted by Derek Bambauer

(This post is based on a talk I gave at the Seton Hall Legislative Journal’s symposium on Bullying and the Social Media Generation. Many thanks to Frank Pasquale, Marisa Hourdajian, and Michelle Newton for the invitation, and to Jane Yakowitz and Will Creeley for a great discussion!)

Introduction

New Jersey enacted the Anti-Bullying Bill of Rights (ABBR) in 2011, in part as a response to the tragic suicide of Tyler Clementi at Rutgers University. It is routinely lauded as the country’s broadest, most inclusive, and strongest anti-bullying law. That is not entirely a compliment. In this post, I make two core claims. First, the Anti-Bullying Bill of Rights has several aspects that are problematic from a First Amendment perspective – in particular, the overbreadth of its definition of prohibited conduct, the enforcement discretion afforded school personnel, and the risk of impingement upon religious and political freedoms. I argue that the legislation departs from established precedent on disruptions of the educational environment by regulating horizontal relations between students rather than vertical relations between students and the school as an institution / environment. Second, I believe we should be cautious about statutory regimes that enable government actors to sanction speech based on content. I suggest that it is difficult to distinguish, on a principled basis, between bullying (which is bad) and social sanctions that enforce norms (which are good). Moreover, anti-bullying laws risk displacing effective informal measures that emerge from peer production. Read the rest of this post »

  February 21, 2012 at 10:20 pm   Posted in: Anonymity, Blogging, Bright Ideas, Civil Rights, Conferences, Constitutional Law, Culture, Current Events, Cyber Civil Rights, Cyberlaw, Education, First Amendment, Media Law, Politics, Privacy (Gossip & Shaming), Psychology and Behavior, Race, Religion, Social Network Websites, Technology, Web 2.0   Print This Post   3 Comments

LTAAA Symposium Wrap-up

posted by Samir Chopra

I want to wrap up discussion in this wonderful online symposium on A Legal Theory for Autonomous Artificial Agents that Frank Pasquale and the folks at Concurring Opinions put together. I appreciate you letting me hijack your space for a week! Obviously, this symposium would not have been possible without its participants–Ken Anderson, Ryan Calo, James Grimmelmann, Sonia Katyal, Ian Kerr, Andrea Matwyshyn, Deborah DeMott, Paul Ohm,  Ugo Pagallo, Lawrence Solum, Ramesh Subramanian and Harry Surden–and I thank them all for their responses. You’ve all made me think very hard about the book’s arguments (I hope to continue these conversations over at my blog at samirchopra.com and on my Twitter feed at @EyeOnThePitch). As I indicated to Frank by email, I’d need to write a second book in order to do justice to them. I don’t want to waffle on too long so let me just quote from the book to make clear what our position is with regards to artificial agents and their future legal status:

Read the rest of this post »

  February 20, 2012 at 5:19 pm  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Cyberlaw, Symposium (Autonomous Artificial Agents)   Print This Post   No Comments

LTAAA Symposium: Complexity, Intentionality, and Artificial Agents

posted by Samir Chopra

I would like to respond to a series of related posts made by Ken Anderson, Giovanni Sartor, Lawrence Solum, and James Grimmelmann during the LTAAA symposium. In doing so, I will touch on topics that occurred many times in the debate here: the intentional stance, complexity, legal fictions (even zombies!) and the law. My remarks here will also respond to the very substantive, engaged comments made by Patrick O’Donnell and AJ Sutter to my responses over the weekend. (I have made some responses to Patrick   and AJ in the comments spaces where their remarks were originally made).

Read the rest of this post »

  February 20, 2012 at 4:32 pm  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Articles and Books, Cyberlaw, Legal Theory, Symposium (Autonomous Artificial Agents), Technology   Print This Post   4 Comments

Ben Stein and the ABA’s Facepalm

posted by Derek Bambauer

The American Bar Association is kicking off its 2012 tech show with an address by… Ben Stein. Yes, who better to celebrate the march of technological progress and innovation than a leading defender of intelligent design? Who better to celebrate rigorous intellectual discourse than a man who misquotes Darwin and fakes speeches to college audiences?

This is a pretty embarrassing misstep. The ABA is irrelevant in the IP / tech world, and this facepalm is a nice microcosm of why. (Wait, what is the ABA relevant to? Now that’s a hard question.) We geeks don’t like it when you dis science. Thanks anyway, ABA – maybe you should stick to having your judicial recommendations ignored.

Hat tip: health law expert Margo Kaplan.

Update: I found the perfect keynote speaker for ABA’s 2013 TechShow: Marshall Hall!

Cross-posted at Info/Law.

  February 20, 2012 at 11:27 am   Posted in: Blogging, Bright Ideas, Conferences, Culture, Current Events, Cyberlaw, Education, Humor, Innovation, Intellectual Property, Media Law, Movies & Television, Politics, Science Fiction, Technology   Print This Post   8 Comments

LTAAA Symposium: Response to Pagallo on Legal Personhood

posted by Samir Chopra

Ugo Pagallo, with whom I had a very useful email exchange a few months ago, has written a very useful response to A Legal Theory for Autonomous Artificial Agents.  I find it useful because I think in each of his four allegedly critical points, we are in greater agreement than Ugo imagines.
Read the rest of this post »

  February 19, 2012 at 6:40 pm  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Articles and Books, Cyberlaw, Legal Theory, Symposium (Autonomous Artificial Agents), Technology   Print This Post   No Comments

LTAAA Symposium: Response to Surden on Artificial Agents’ Cognitive Capacities

posted by Samir Chopra

I want to thank Harry Surden for his rich, technically-informed response  to A Legal Theory for Autonomous Artificial Agents, and importantly, for seizing on an important distinction we make early in the book when we say:

There are two views of the goals of artificial intelligence. From an engineering perspective, as Marvin Minsky noted, it is the “science of making machines do things that would require intelligence if done by men” (Minsky 1969, v). From a cognitive science perspective, it is to design and build systems that work the way the human mind does (Shanahan 1997, xix). In the former perspective, artificial intelligence is deemed successful along a performative dimension; in the latter, along a theoretical one. The latter embodies Giambattista Vico’s perspective of verum et factum convertuntur, “the true and the made are…convertible” (Vico 2000); in such a view, artificial intelligence would be reckoned the laboratory that validates our best science of the human mind. This perspective sometimes shades into the claim artificial intelligence’s success lies in the replication of human capacities such as emotions, the sensations of taste and self-consciousness. Here, artificial intelligence is conceived of as building artificial persons, not just designing systems that are “intelligent.”

The latter conception of AI as being committed to building ‘artificial persons’ is what, it is pretty clear, causes much of the angst that LTAAA’s claims seem to occasion. And even though I have sought to separate the notion of ‘person’ from ‘legal persons’ it seems that some conflation has continued to occur in our discussions thus far.

I’ve personally never understood why artificial intelligence was taken to be, or ever took itself to be, dedicated to the task of replicating human capacities, faithfully attempting to build “artificial persons” or “artificial humans”. This always seemed such like a boring, pointlessly limited task. Sure, the pursuit of cognitive science is entirely justified; the greater the understanding we have of our own minds, the better we will be able to understand our place in nature. But as for replicating and mimicking them faithfully: Why bother with the ersatz when we have the real? We already have a perfectly good way to make humans or persons and it is way more fun than doing mechanical engineering or writing code. The real action, it seems to me, lay in the business of seeing how we could replicate our so-called intellectual capacities without particular regard for the method of implementation; if the best method of implementation happened to be one that mapped on well to what seemed like the human mind’s way of doing it, then that would be an added bonus. The multiple-realizability of our supposedly unique cognitive abilities would do wonders to displace our sense of uniqueness, acknowledge the possibility of other modes of existence, and re-invoke the sense of wonder about the elaborate tales we tell ourselves about our intentionality, consciousness, autonomy or freedom of will.

Having said this, I can now turn to responding to Harry’s excellent post.
Read the rest of this post »

  February 19, 2012 at 3:26 pm  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Articles and Books, Cyberlaw, Legal Theory, Psychology and Behavior, Symposium (Autonomous Artificial Agents), Technology   Print This Post   6 Comments

LTAA Symposium: Response to Matwyshyn on Artificial Agents and Contracting

posted by Samir Chopra

Andrea Matwyshyn’s reading of the agency analysis of contracting  (offered in A Legal Theory for Autonomous Artificial Agents and also available at SSRN) is very rigorous and raises some very interesting questions. I thank her for her careful and attentive reading of the analysis and will try and do my best to respond to her concerns here. The doctrinal challenges that Andrea raises are serious and substantive for the extension and viability of our doctrine. As I note below, accommodating some of her concerns is the perfect next step.

At the outset, I should state what some of our motivations were for adopting agency doctrine for artificial agents in contracting scenarios (these helped inform the economic incentivizing argument for maintaining some separation between artificial agents and their creators or their deployers.

First,

[A]pplying agency doctrine to artificial agents would permit the legal system to distinguish clearly between the operator of the agent i.e., the person making the technical arrangements for the agent’s operations, and the user of the agent, i.e., the principal on whose behalf the agent is operating in relation to a particular transaction.

Second,

Embracing agency doctrine would also allow a clear distinction to be drawn between the authority of the agent to bind the principal and the instructions given to the agent by its operator.

Third, an implicit, unstated economic incentive.

Read the rest of this post »

  February 19, 2012 at 2:10 pm  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Contract Law & Beyond, Cyberlaw, Economic Analysis of Law, Legal Theory, Symposium (Autonomous Artificial Agents), Technology, Tort Law   Print This Post   No Comments

LTAA Symposium: Response to Sutter on Artificial Agents

posted by Samir Chopra

I’d like to thank Andrew Sutter for his largely critical, but very thought-provoking, response to A Legal Theory for Autonomous Artificial Agents. In responding to Andrew I will often touch on themes that I might have already tackled. I hope this repetition comes across as emphasis, rather than as redundancy. I’m also concentrating on responding to broader themes in Andrew’s post as opposed to the specific doctrinal concerns (like service-of-process or registration; my attitude in these matters is that the law will find a way if it can discern the broad outlines of a desirable solution just ahead; service-of-process seemed intractable for anonymous bloggers but it was solved somehow).
Read the rest of this post »

  February 18, 2012 at 3:05 pm  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Articles and Books, Cyberlaw, Symposium (Autonomous Artificial Agents), Technology   Print This Post   7 Comments

LTAAA Symposium: Artificial Agents and the Law of Agency

posted by Samir Chopra

I am gratified that Deborah DeMott, whose work on agency doctrines was so influential in our writing has written such an engaged (and if I may so, positive)  response to our attempt, in A Legal Theory for Autonomous Artificial Agents, to co-opt the common law agency doctrine for use with artificial agents. We did so, knowing the fit would be neither exact, nor precise, and certainly would not mesh with all established intuitions.
Read the rest of this post »

  February 18, 2012 at 12:47 pm  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Articles and Books, Contract Law & Beyond, Cyberlaw, Legal Theory, Symposium (Autonomous Artificial Agents)   Print This Post   No Comments

LTAAA Symposium: Legal Personhood for Artificial Agents

posted by Samir Chopra

In this post, I’d like to make some brief remarks on the question of legal personhood for artificial agents, and in so doing, offer a response to Sonia Katyal’s and Ramesh Subramanian’s thoughtful posts on A Legal Theory for Autonomous Artificial Agents. I’d like to thank Sonia for making me think more about the history of personhood jurisprudence, and Ramesh for prompting to me to think more about the aftermath of granting legal personhood, especially the issues of “Reproduction, Representation, and Termination” (and for alerting me to  Gillick v West Norfolk and Wisbech Area Health Authority)

I have to admit that I don’t have as yet, any clearly formed thoughts on the issues Ramesh raises. This is not because they won’t be real issues down the line; indeed, I think automated judging is more than just a gleam in the eye of those folks that attend ICAIL conferences. Rather, I think it is that those issues will perhaps snap into sharper focus once artificial agents acquire more functionality, become more ubiquitous, and more interestingly, come to occupy roles formerly occupied by humans. I think, then, we will have a clearer idea of how to frame those questions more precisely with respect to a particular artificial agent and a particular factual scenario.
Read the rest of this post »

  February 18, 2012 at 10:54 am  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Articles and Books, Contract Law & Beyond, Cyberlaw, Symposium (Autonomous Artificial Agents)   Print This Post   3 Comments

The Memory Hole

posted by Derek Bambauer

On RocketLawyer’s Legally Easy podcast, I talk with Charley Moore and Eva Arevuo about the EU’s proposed “right to be forgotten” and privacy as censorship. I was inspired by Jeff Rosen and Jane Yakowitz‘s critiques of the approach, which actually appears to be a “right to lie effectively.” If you can disappear unflattering – and truthful – information, it lets you deceive others – in other words, you benefit and they are harmed. The EU’s approach is a blunderbuss where a scalpel is needed.

Cross-posted at Info/Law.

  February 17, 2012 at 12:01 pm   Posted in: Anonymity, Architecture, Civil Rights, Consumer Protection Law, Culture, Current Events, Cyber Civil Rights, Cyberlaw, First Amendment, Google and Search Engines, Innovation, Media Law, Political Economy, Politics, Privacy, Technology, Web 2.0   Print This Post   No Comments

Autonomous Agents and Extension of Law: Policymakers Should be Aware of Technical Nuances

posted by Harry Surden

This post expands upon a theme from Samir Chopra and Lawrence White’s excellent and thought-provoking book – A Legal Theory for Autonomous Artificial Agents.  One question pervading the text: to what extent should lawmakers import or extend existing legal frameworks to cover the activities of autonomous (or partially autonomous) computer systems and machines?   These are legal frameworks that were originally created to regulate human actors.  For example, the authors query whether the doctrines and principles of agency law can be mapped onto actions carried out by automated systems on behalf of their users?  As the book notes, autonomous systems are already an integral part of existing commercial areas (e.g. finance) and may be poised to emerge in others over the next few decades (e.g. autonomous, self-driving automobiles). However, it is helpful to further expand upon one dimension raised by the text: the relationship between the technology underlying autonomous agents, and the activity or results produced by the technology.

Two Views of Artificial Intelligence

The emergence of partially autonomous systems – computer programs (or machines) carrying out activities at least partially in a self-directed way, on behalf of their users, is closely aligned with the field of Artificial Intelligence (AI) and developments therein. (AI is a sub-discipline of computer science.) What is the goal of AI research? There is probably no universally agreed upon answer to this question – as there have been a range of approaches and criteria for systems considered to be successful advances in the field. However, some AI researchers have helpfully clarified two dimensions along which we can think about AI developments. Consider a spectrum of possible criteria under which one might label a system to be a “successful” AI product:

View 1) We might consider a system to be artificially intelligent only if it produces “intelligent” results based upon processes that model, approach or replicate the high-level cognitive abilities or abstract reasoning skills of humans ;or

View 2) We might most evaluate a system primarily based upon the quality of the output it produces – if it produces results that humans would consider accurate and helpful – even if the results or output came about through processes that do not necessarily model , approach, or resemble actual human cognition, understanding, or reasoning.

We can understand the first view as being concerned with creating systems that replicate to some degree something approaching human thinking and understanding, whereas the second is more concerned with producing results or output from computer agents that would be considered “intelligent” and useful, even if produced from systems which likely do not approach human cognitive processes. (Russell and Norvig, Artificial Intelligence: A Modern Approach, 3 Ed, 2009, 1-5). These views represent poles on a spectrum, and many actual positions fall in between. However, this distinction is more than philosophical.  It has implications on the sensibility of extending existing legal doctrines to cover the activities of artificial agents. Let us consider each view briefly in turn, and some possible implications upon law.

View 1 – Artificial Intelligence as Replicating Some or All Human Cognition

The first characterization – that computer systems will be successful within AI when they produce activities resulting from processes approaching the high-level cognitive abilities of humans, is considered an expansive and perhaps more ambitious characterization of the goals of AI. It also seems to be the one most closely associated with the view of AI research in the public imagination. In popular culture, artificially intelligent systems replicate and instantiate – to varying degrees – the thinking facilities of humans (e.g. the ability to engage in abstract thought, carry on an intelligent conversation, or to understand or philosophize concerning concepts at a depth associated with intelligence). I raise this variant primarily to note that despite   (what I believe is a) common lay view of the state of the research- this “strong” vision of AI is not something that has been realized (or is necessarily near realization) within the existing state-of-the art systems that are considered successful products of AI research. As I will suggest shortly, this nuance may not be something within the awareness of lawmakers and judges who will be the arbiters of such decisions concerning systems that are labeled artificially intelligent.  Although AI research has not yet produced artificial human-level cognition, that does not mean that AI research has been unsuccessful.  Quite to the contrary – over the last 20 years AI research has produced a series of more limited, but spectacularly successful systems as judged by the second view.

View 2 – “Intelligent” Results (Even if Produced by Non-Cognitive Processes)

The second characterization of AI is perhaps more modest, and can be considered more “results oriented.”  This view considers a computer system (or machine) to be a success within artificial intelligence based upon whether it produces output or activities that people would agree (colloquially speaking) are “good” and “accurate” and “look intelligent.”  In other words, a useful AI system in this view is characterized by results or output are likely to approach or exceed  that which would have been produced by a human performing the same task.  Under this view, if the system or machine produces useful, human-like results, this is a successful AI machine – irrespective as to whether these results were produced from a computer-based process instantiating or resembling human cognition, intelligence or abstract reasoning.

In this second view, AI “success” is measured based upon whether the autonomous system produces “intelligent” (or useful) output or results.  We can use what would be considered “intelligent” conduct of a similarly situated human as a comparator. If a modern auto-pilot system is capable of landing airplanes in difficult conditions (such as thick fog) at a success rate that meets or exceeds human pilots under similar conditions, we might label it a successful AI system under this second approach. This would be the case even if we all agreed that the autonomous autopilot system did not have a meaningful understanding of the concepts of “airplanes”, “runways”, or “airports.” Similarly, we might label IBM’s Jeopardy playing “Watson” computer system to be a successful AI system since it was capable of producing highly accurate answers, to a surprisingly wide and difficult range of questions – the same answers that a strong, human Jeopardy champions would have produced. However, there is no suggestion that Watson’s results were the result of the same high-level cognitive understanding and processes that likely animated the result of the human champions like Ken Jennings. Rather, Watson’s accurate output came from techniques such as highly sophisticated statistical machine-learning algorithms that were able to quickly rank possible candidate answers through immense parallel processing of large amounts of existing written documents that happened to contain a great deal knowledge about the world.

Machine-Translation: Automated Translation as an Example

To understand this distinction between AI views rooted in computer-based cognition and those in “intelligent” or accurate results, it is helpful to examine the history of computer-based language translation (e.g. English to French). Translation (at least superficially) appears to be a task deeply connected to the human understanding of the meaning of language, and the conscious replication of that meaning in the target language. Early approaches to machine translation followed this cue, and sought to convey aspects to computer system – like the rules of grammar in both languages, and the pairing of words with the same meanings in both language – that might mimic the internal structures undergirding human cognition and translation. However, this meaning and rules-based approach to translation proved limited and surprised researchers by producing somewhat poor results based upon the rules of matching and syntactical construction. Such system had difficulty in determining whether the word “plant” in English should be translated to the equivalent of “houseplant” or “manufacturing plant” in French. Further efforts attempted to “teach” the computer rules about how to understand and make more accurate distinctions for ambiguously situated words but still did not produce marked improvements in translation quality.

Machine Learning Algorithms: Using Statistics to Produce Surprisingly Good Translations

However, over the last 10-15 years, a markedly different approach to computer translation occurred – made famous by Google and others. This approach was not primarily based upon top-down communication of the basics of constructing and conveying knowledge to a computer system (e.g. language pairing and rules of meaning). Rather, many of the successful translation techniques developed were largely statistical in nature, relying on machine-learning algorithms to scour large amounts of data and create a complex representation of correlations between languages. Google translate – and other similar statistical approaches – work in part by leveraging vast amounts of data that has previously been translated by humans. For example, the United Nations and the European Union frequently translate official documents into multiple languages using professional translators. This “corpus” of millions of paired and translated documents became publicly available electronically over the last 20 years to researchers. Systems such as Google Translate are able to process vast numbers of documents and leverage these paired, translated translation to create statistical models which are able to produce surprisingly accurate translation results using probabilities – for arbitrary new texts.

Machine-Learning Models: Producing “intelligent”, highly useful results 

The important point is that these statistical and probability-based machine-learning models (often combined with logical-knowledge based rules about the world) often produce high-quality and effective results (not quite up to the par of nuanced human translators at this point), without any assertion that the computers are engaging in profound understanding with the underlying “meaning” of the translated sentences or employing processes whose analytical abilities approach human-level cognition (e.g. view 1). (It is important to note that the machine-learning translation approach does not achieve translation on its own but “leverages: previous human cognition through the efforts of the original UN translators that made the paired translations.)  Thus, for certain, limited tasks,  these systems have shown that it is possible for contemporary autonomous agents to produce “intelligent” results without relying upon what we would consider processes approaching human-level cognition.

Distinguishing “intelligent results” and actions produced via cognitive intelligence

The reason to flag this distinction, is that such successful AI systems (as judged by their results), will pose a challenge to the task of importing and extending of existing legal doctrinal frameworks – (which were mostly designed to regulate people) into the domain of autonomous computer agents.  Existing “type 2″ systems that produce surprisingly sophisticated, useful, and accurate results without approaching human cognition are the basis of many products now emerging from earlier AI research and are becoming integrated (or are poised to become ) integrated into life.    These include IBM’s Watson, Apple’s SIRI, Google Search – and in perhaps the next decade or two – Stanford’s/Google’s Autonomous self-driving cars, and autonomous music composing software.  These systems often use statistics to leverage existing, implicit human knowledge.  Since these systems produce output or activities that in some cases appear to approach or exceed humans in particular tasks, and the results that are autonomously produced are often surprisingly sophisticated, and seemingly intelligent – such “results-oriented”, task specific (e.g. driving, answering questions, landing planes) systems seem to be the near path of much AI research.

However, the fact that these intelligent-seeming results do not result from systems approaching human-cognition is a nuance that should not be lost on policymakers (and judges) seeking to develop doctrine in the area of autonomous agents. Much – perhaps most of law – is designed and intended to regulate the behavior of humans (or organizations run by humans).  Thus embedded in many existing legal doctrines are underlying assumptions about cognition and intentionality that are implicit and are so basic that they are often not articulated.   The implicitness of such assumptions may make these assumptions easy to overlook.

Given current trends, many contemporary (and likely future) AI systems that will be integrated into society (and therefore more likely the subject of legal regulation) will use algorithmic techniques focused upon producing “useful results” (view 2), rather than focusing on systems aimed at replicating human-level cognition, self-reflection, and abstraction (view 1).  If lawmakers merely follow the verbiage (e.g. a system that has been labeled “artificially intelligent” did X or resulted in Y) and employ only a superficial understanding of AI research, without more closely understanding these technical nuances, there is the possibility of conflation in extending existing legal doctrines to circumstances based upon “intelligent seeming” autonomous results.   For example, the book authors explore the concept of requiring fiduciary duties on the part of autonomous systems in some circumstances. But it will take a careful judge or lawmaker to distinguish existing fiduciary/agency doctrines with embedded (and often unarticulated) assumptions of human-level intentionality among agents (e.g. self-dealing) from those that may be more functional in nature (e.g. duties to invest trust funds). In other words, an in-depth understanding of the technology underlying particular autonomous agents should not be viewed as a technical issue.   Rather it is a serious consideration which should be understood in some detail by lawmakers in any decisions to extend or create new legal doctrine from our existing framework to cover situations involving autonomous agents.

  February 16, 2012 at 10:41 pm  Tags: A Legal Theory for Autonomous Artificial Agents, Automated law, computer agents  Posted in: Cyberlaw, Google and Search Engines, Symposium (Autonomous Artificial Agents)   Print This Post   2 Comments

Our Bots, Ourselves

posted by Andrea Matwyshyn

In an extremely forward-looking and thought-provoking book, Samir Chopra and Lawrence F. White rekindle important legal questions with respect to autonomous artificial agents or bots.  It was a pleasure to engage with the questions that the authors raise in A Legal Theory for Autonomous Artificial Agents, and the book is a valuable scholarly contribution.  In particular, because of my own research interests, Chapter 2 Artificial Agents and Contracts was of special interest to me.

In Chapter 2, the authors apply the agency theory that they advocate in Chapter 1 to the context of contracts.  They challenge the view that bots are “mere tools” used for extension of the self by contracting parties.[1]    In doing so, they assert differences between “closed” and “open”[2] systems and various theoretical types of bots, arguing that parties who use bots as part of contracting should be protected from contract liability in some cases of bot error or malfunction.   From my reading, they argue in favor of using principles of agency law to replace some traditional contract law constructs when bots are involved in contracts.

Their argument is nuanced and thoughtful from an economic and agency law perspective.  In the comments that follow, I raise five sets of questions for thought, admittedly from the perspective of my own research on contract law, consumer privacy in technology contexts, and information security law.

1. Private ordering and accepting responsibility for imprudent technology risks.   The authors are concerned with providing better liability protection to contracting parties who use bots.  They assert that “[a] strict liability principle [which views bots as mere tools or means of communication] may not be fair to those operators and users who do not anticipate the contractual behavior of the agent in all possible circumstances and cannot reasonably be said to be consenting to every contract entered into by the agent.”[3]   As I was reading this chapter, I pondered whether bots do indeed warrant special contract law rules.  How is a failure to anticipate the erratic behavior of a potentially poorly-coded bot not simply one of numerous categories of business risk that parties may fail to foresee?   Applying a contract law perspective, one might argue that the authors’ approach usurps for law what should be left to private ordering and risk management.  No one forces a party to use a bot in contracting; perhaps choosing to do so is simply an information risk that should be planned around with insurance?[4]

2. Traditional contract law breach and damages analysis and the expectations of the harmed party.  The authors opt away from a discussion of traditional breach analysis and damages remedies when addressing bot failures.  Instead, they apply a tort-like calculation of a lowest cost avoider principle, which they argue “correctly allocate[s] the risk of specification or induction errors to the principle/operator and of malfunction to the principle or user, depending on which was the least–cost avoider “[5]  However, should we perhaps temper this analysis by recognizing that contract law as embodied by the UCC and caselaw is not concerned solely or even primarily with efficiency in contractual relationships?  How does the authors’ efficiency analysis square with traditional consideration sufficiency (versus adequacy) analysis, where courts enforce contracts with bad deal terms regularly, choosing not to question the choices of the parties?   A harmed consumer who was not using a bot in a contract pitted against a sophisticated company using a poorly-coded bot (because it chose to hire a bargain programmer) may indeed have inefficient needs, but is not the consumer the party more in need of the court’s protection as a matter of equity?[6]

The authors note that, for example, prices quoted by a bot are akin to pricing details provide by a third party – a scenario that they assert may make it unfair to bind the bot-using party to terms of a contract executed by his bot when he does not pre-approve each individual deal.  “In many realistic settings involving medium–complexity agents, such as modern shopping websites, the principal cannot be said to have a pre-existing “intention” in respect of the particular contract that is “communicated “to the user.”[7]  Again, to what extent are such bot dynamics truly unforeseeable?  Can it be argued that coding up your bot to offer very specific deal terms when a consumer clicks on something constitutes an indication of actual knowledge and intention similar to a price list?  Is a coding error with a wrong price not simply akin to a mismarked pricetag in real space? But even assuming that we agree with the argument that coding up a bot is a relinquishment of control to a third party of sorts, how would the bot dynamics at issue differ from those in real space contracts where prices are specified using a variable third party index or where performance details are left variable – dynamics that have been found unproblematic in real space contract cases? [8]

3.  The bot problems that currently exist in contract law.  The authors take us through two cases with respect to bots – eBay v. Bidder’s Edge and Register v. Verio.com, arguing them primarily through the lens of tort, particularly trespass to chattels.    I found myself wondering about the authors’ agency analysis in the contract-driven bot cases where the trespass to chattels line of argument was deemed unpersuasive.  For example, how would the authors’ agency analysis apply in the context of the two Ticketmaster v. Tickets.com bot cases, particularly the second where the trespass to chattels claim was dismissed and the contract count was the only count to survive summary judgment?   Also, I would be curious to hear more about the extrapolation of their agency approach to the current wave of bot cases that blend contract formation questions with allegations of computer intrusion, such as Facebook v. Power Ventures and United States v. Lowson.

4.  Duties of information security.  Turning to information security, the authors point out that a party may try to hack a bot used by the other party in order to gain contracting advantage.[9]  While this is a valid computer intrusion concern, another pressing contract concern is that a malicious third-party (who is not one of the parties seeking to be in privity) will chooses to hack the bot to steal money on an ongoing basis.  If the bot is vulnerable to easy attack because of information security deficits of its coding, should the party using it get a free pass for its failure to exercise due care in information security?  Is it fair to impose information security losses on the other contracting party who was prudent enough not to use a vulnerable bot in contracting?   Would a straightforward ‘your vulnerability, your responsibility’ approach create better incentives for close monitoring and better information security practices, a goal already recognized by Congress as a social good?

5. The broader implications for “b0rked” code.  The separateness of bots from their creators came across to me as an underlying premise for the entirety of the authors’ conversation.   For example, the authors reference situations where the bot autonomously “misrepresents” information that its wielding party would not approve.[10]   Is it not perhaps more accurate to say that the bot contains programming bugs its wielding party failed to catch and rectify?   Is not a bot simply lines of code written by a human (who may or may not be skilled in a particular code language) that will always be full of errors (because a human authored it)?   Is perhaps the appropriate goal not to protect bots but to incentivize bot creators to make fewer errors and rectify errors once they are found after “shipping” the code?

The authors argue that holding a contracting party accountable for bot malfunction is “unjust”[11] in some circumstances.    Is this consonant with the contract law approach that drafting errors and ambiguities are construed against the drafter?[12]  Is the author/operator of the error-ridden code considered the drafter here?  How is choosing a bad programmer to build your flawed bot different from choosing a bad lawyer to draft the flawed language of your contract?

The analogy of a bot to a rogue beer-fetching dog I found to be a particularly apt one.[13]  Many scholars would argue that, much like having a pet, using a code-based creation such as a bot in contracting is a choice and an assumption of responsibility.  Both dogs and bots are things that are optional and limited in their capacities: we choose to unleash them on the rest of the world.   If a dog or a bot causes harm, even when the owner has not expressly directed it to do so, isn’t it always the owner’s failure to supervise that is to blame?  I fear that comparing a bot to a human of any sort – slave, child, employee – at the current juncture for purposes of crafting law may be premature.  No machine is capable of replicating human behavior smoothly at present.  Will one arrive in the future?  Yes, it is likely.  However, I fear that aggressive untethering of the legal responsibility of the coder from her coded creation may send us down an undesirable path of uncompensated consumer harms in our march toward our brave new cyborg world.[14]

The book’s purposes are ambitious, and I truly enjoyed pondering the questions it raises.  I thank the organizers for allowing me to participate in this symposium.

 

  February 15, 2012 at 5:52 pm  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Contract Law & Beyond, Cyberlaw, Symposium (Autonomous Artificial Agents), Uncategorized   Print This Post   2 Comments

Cary Sherman and the Lost Generation

posted by Derek Bambauer

The RIAA’s Cary Sherman had a screed about the Stop Online Piracy and PROTECT IP Acts in the New York Times recently. Techdirt’s Mike Masnick brilliantly gutted it, and I’m not going to pile on – a tour de force requires no augmentation. What I want to suggest is that the recording industry – or, at least, its trade group – is dangerously out of touch.

Contrast this with at least part of the movie industry, as represented by Paramount Pictures. I received a letter from Al Perry, Paramount’s Vice President Worldwide Content Protection & Outreach. He proposed coming here to Brooklyn Law School to

exchange ideas about content theft, its challenges and possible ways to address it. We think about these issues on a daily basis. But, as these last few weeks [the SOPA and PROTECT IP debates] made painfully clear, we still have much to learn. We would love to come to campus and do exactly that.

Jason Mazzone, Jonathan Askin, and I are eagerly working to have Perry come to campus, both to present Paramount’s perspective and to discuss it with him. We’ll have input from students, faculty, and staff, and I expect there to be some pointed debate. We’re not naive – the goal here is to try to win support for Paramount’s position on dealing with IP infringement – but I’m impressed that Perry is willing to listen, and to enter the lion’s den (of a sort).

And that’s the key difference: Perry, and Paramount, recognize that Hollywood has lost a generation. For the last decade or so, students have grown up in a world where content is readily available via the Internet, through both licit and illicit means; where the content industries are the people who sue your friends and force you to watch anti-piracy warnings at the start of the movies you paid for; and where one aspires to be Larry Lessig, not Harvey Weinstein. Those of us who teach IP or Internet law have seen it up close. In another ten years, these young lawyers are going to be key Congressional staffers, think tank analysts, entrepreneurs, and law firm partners. And they think Hollywood is the enemy. I don’t share that view – I think the content industries are amoral profit maximizers, just like any other corporation – but I understand it.

And that’s where Sherman is wrong and Perry is right. The old moves no longer work. Buying Congresspeople to pass legislation drafted behind closed doors doesn’t really work (although maybe we’ll find out when we debate the Copyright Term Extension Act of 2018). Calling it “theft” when someone downloads a song they’d never otherwise pay for doesn’t work (even Perry is still on about this one).

One more thing about Sherman: his op-ed reminded me of Detective John Munch in Homicide, who breaks down and shouts at a suspect, “Don’t you ever lie to me like I’m Montel Williams. I am not Montel Williams.” Sherman lies to our faces and expects us not to notice. He writes, “the Protect Intellectual Property Act (or PIPA) was carefully devised, with nearly unanimous bipartisan support in the Senate, and its House counterpart, the Stop Online Piracy Act (or SOPA), was based on existing statutes and Supreme Court precedents.” Yes, it was carefully devised – by content industries. SOPA was introduced at the end of October, and the single hearing that was held on it was stacked with proponents of the bill. “Carefully devised?” Key proponents didn’t even know how its DNS filtering provisions worked. He argues, “Since when is it censorship to shut down an operation that an American court, upon a thorough review of evidence, has determined to be illegal?” Because censorship is when the government blocks you from accessing speech before a trial. “A thorough review of evidence” is a flat lie: SOPA enabled an injunction filtering a site based on an ex parte application by the government, in contravention of a hundred years of First Amendment precedent. And finally, he notes the massive opposition to SOPA and PROTECT IP, but then asks, “many of those e-mails were from the same people who attacked the Web sites of the Department of Justice, the Motion Picture Association of America, my organization and others as retribution for the seizure of Megaupload, an international digital piracy operation?” This is a McCarthyite tactic: associating the remarkable democratic opposition to the bills – in stark contrast to the smoke-filled rooms in which Sherman worked to push this legislation – with Anonymous and other miscreants.

But the risk for Sherman – and Paramount, and Sony, and other content industries – is not that we’ll be angry, or they’ll be opposed. It’s that they’ll be irrelevant. And if Hollywood takes the Sherman approach, rather than the Perry one, deservedly so.

Cross-posted at Info/Law.

  February 14, 2012 at 7:40 pm   Posted in: Architecture, Culture, Cyber Civil Rights, Cyberlaw, DRM, First Amendment, Google & Search Engines, Innovation, Intellectual Property, Media Law, Political Economy, Politics, Technology, Web 2.0   Print This Post   5 Comments

Three Roads to Legal Agency and the Personhood of Autonomous Artificial Agents (AAAs)

posted by Ugo Pagallo

I enjoyed A Legal Theory for Autonomous Artificial Agents (LTAAA) by Samir Chopra and Laurence White, and, still, I have some doubts. In order to clarify this reaction, let me distinguish three different kinds of legal agents.

A1 – Agents can be a “source” of responsibility for other agents in the legal system

A2 – Agents can be considered as “strict agents” in civil (as opposed to criminal) law

A3 – Agents can be “proper persons” with rights (and duties) of their own

I found convincing how LTAAA deals with A1 in Chapter 4 (tort law) and A2 in Chapters 2 and 3 (contracts). My doubts revolve around A3 in Chapter 5 (which includes matters of legal agency that concern the criminal law field as well).

Although I reckon that the hottest legal issues of AAAs today concern A1 and A2, let me thus dwell on A3.

In a nutshell, the thesis of LTAAA is that “none of the philosophical objections to personhood for artificial agents – most but not all of them based on ‘a missing something argument’ – can be sustained, in the sense that artificial agents can be plausibly imagined that display that allegedly missing behaviour or attribute. If this is the case, then in principle artificial agents should be able to qualify for independent legal personality, since it is the closest legal analogue to the philosophical conception of a person” (op. cit., 182).

To be sure, I concede the point as a matter of principle. In the wording of Lawrence Solum’s Legal Personhood for Artificial Intelligence: “one cannot, on conceptual grounds, rule out in advance the possibility that AIs should be given the rights of constitutional personhood” (1992: 1260).

Besides, I agree with LTAAA that (some types of) AAAs can, or should, properly be conceived as strict agents in civil law (A2). For example, I have proposed the parallel between the Roman law mechanism for A2 in the case of slaves, that is, the peculium, and today’s A2 for AAAs. However, what LTAAA claims is different. Forms of artificial accountability such as the “digital peculium” would not be unsatisfactory because, say, the parallels between AAAs and slaves are deemed unethical or anthropologically biased. Rather, the autonomy granted by such forms of accountability is reckoned insufficient because once we accept that some artificial agents may be properly conceived of as strict agents in the field of contracts, their legal personhood would then follow as a result. Moreover, “at the risk of offending humanist sensibilities,” LTAAA argues that we should yield before the fact that, sooner or later, AAAs will be a sort of “being sui juris,” capable of “sensitivity to legal obligations” and even of “susceptibility to punishment,” that finally allows us “to forgive a computer” (op. cit., 180).

My doubts on how LTAAA addresses A3 for AAAs can be summed up in 4 points.

First, the example of the legal status of slaves under ancient Roman law shows that strict legal agency in contract law (A2) and the legal personhood of AAAs (A3) are not correlated. Aside from ethical aberrations of humans being treated as mere things, there are no particular reasons for claiming that the legal personhood of AAAs (A3) is necessarily entwined with their status of strict agents in the civil law field (A2). Even the European Union, after all, existed for decades without enjoying its own legal personhood!

Second, according to the current state of the art technology, AAAs are far away from achieving a human-like endowment of free will, autonomy, and moral sense, despite the controversial meaning of such expressions. I would admit that some AAAs are endowed with self-knowledge and autonomy “in the engineering meaning of these words” (EURON 2007). However, it is the engineering meaning of these words that reminds us of the very difference between civil and criminal law. The level of autonomy of some AAAs, which is sufficient to have relevant effects in the field of contracts, arguably is insufficient to bring AAAs before judges and have them found guilty by criminal courts.

Third, LTAAA should explain the pragmatic (rather than conceptual) reasons of their stance. As far as I understand, “not only is according artificial agents with legal personality a possible solution to the contracting problem, it is conceptually preferable to the other agency law approach to legal agency without legal personality, because it provides a more complete analogue with the human case” (op. cit., 162, italics added). However, had not these same authors insisted on the thesis that the dependent legal responsibility of AAAs is “based on a combination of human chauvinism and a misunderstanding of the notion of legal person”? (op. cit., 27) Why should we endorse “analogy with the human case” in the case of AAAs?

Finally, I may admit that, once a novel generation of AAAs endowed with human-like free will, autonomy, or moral sense materializes, lawyers should be ready to tackle both A3 and the constitutional rights of AAAs seriously.  But, if we accept the thesis of LTAAA, it is more than likely that the meaning of traditional notions such as contracts, torts, or constitutional rights, will change. As a matter of fact, what the meaning of such legal concepts would be is still assigned to the imagination of science fiction writers, rather than the science faction-analysis of legal experts. Would an AAA lawyer be an advocate of the tradition of natural law, so that rules should be viewed as an objective imperative whose infringement implies a violation of the nature of the artificial agent? Would the lawyer vice versa be a sort of legal realist, so that norms depend on how AAAs affect human understanding of the world, their own knowledge and environment? And how about the institutional stances of AAA lawyers who, contrary to their fellow colleagues keen to follow the Kelsenian lesson of the pure doctrine of the law, focus on the substantive mechanisms of a new artificial order?

 

  February 14, 2012 at 9:14 am  Tags: A Legal Theory of Autonomous Artificial Agents, AI & the Law, artificial agents  Posted in: Cyberlaw, Symposium (Autonomous Artificial Agents), Uncategorized   Print This Post   2 Comments

LTAAA Symposium: Campaign 2020′s Bots United

posted by Frank Pasquale

A Legal Theory of Autonomous Artificial Agents offers a serious look at several legal controversies set off by the rise of bots. “Autonomy” is one of the key concepts in the work. We would not think of a simple drone programmed to fly in a straight line as an autonomous entity. On the other hand, films like Blade Runner envision humanoid robots that so closely mimic real homo sapiens that it seems churlish or cruel to dismiss their claims for respect and dignity (and perhaps even love). In between these extremes we find already well-implemented, cute automatons. As Sherry Turkle has noted, when confronted by Paro (above right), children “move from inquiries such as “Does it swim?” and “Does it eat?” to “Is it alive?” and “Can it love?””

For today’s post, I want to move to another, perhaps childish, question: can the bot speak? The question will be particularly urgent by 2020, but is relevant even now because corporate and governmental entities want to promote armies of propagandizing bots to disseminate their views and drown out opposing voices. Consider the experiment run by Tim Hwang, of the law firm Robot, Robot, & Hwang, on Twitter, as explained in conversation with Bob Garfield:

GARFIELD: Earlier this year, 500 or so Twitterers received tweets from someone with the handle @JamesMTitus who posed one of several generic questions: How long do you want to live to, for example, or do you have any pets? @JamesMTitus was cheerful and enthusiastic, kind of like those people who comment on the weather and then laugh heartily. Perhaps because of that good nature or perhaps because of his inquiring spirit and interest in others, @JamesMTitus was able to strike up a fair number of continuing conversations. Only thing is, there is no @JamesMTitus. He, or it, is a bot, a software program designed to engage actual humans in social networks.

Read the rest of this post »

  February 14, 2012 at 9:08 am  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Cyberlaw, Symposium (Autonomous Artificial Agents)   Print This Post   2 Comments

Tempest in Tempe: First Amendment in the Desert

posted by Derek Bambauer

In the spirit of the excellent colloquy here about Marvin’s thinking on First Amendment architectures, I bring up this news item: Arizona State University blocked both Web access to, and e-mail from, the change.org Web site. ASU students had begun a petition demanding that the university reduce tuition. The university essentially made three claims as to why it did so (below, in order of increasing stupidity):

1. It was a technical mistake;
2. Change.org was spamming ASU; and
3. ASU needs to “protect the use of our limited and valuable network resources for legitimate academic, research and administrative uses.”

#1 and #2 run together. If spam is the problem, you don’t need to block access to the Web site. However, if you are concerned that students are going to read the petition, and sign it, you do need to block access to the Web site.

For #2, sorry, ASU, this isn’t spam. Spam is unsolicited bulk commercial e-mail. Change.org is, allegedly, sending unsolicited political e-mail. And that’s protected by the First Amendment – see, for example, the Virginia Supreme Court’s analysis of that state’s anti-spam law that covered political messages. Potential political spammers have a sharp disincentive to fill recipient’s inboxes – it’s a sure-fire way to annoy them into opposing your position.

For #3, ASU doesn’t get to determine what academic and research uses are “legitimate.” If they throttle P2P apps, that’s fine. If they limit file sizes for attachments, no problem. But deciding that the message from Change.org is not “legitimate” is classic, and unconstitutional, viewpoint discrimination.

This looks like censorship. I think it’s more likely to be stupidity: someone in ASU’s IT department decided to block these messages as spam, and to filter outbound Web requests to the site contained within those messages. But: with great power over the network comes great responsibility. Well-intentioned constitutional violations are still unlawful. It would also help if ASU’s spokesperson simply admitted the mistake rather than engaging in idiotic justification.

As I mention in Orwell’s Armchair, public actors are increasingly important sources of Internet access. But when ASU and other public universities take on the role of ISP, they need to remember that they are not AOL: their technical decisions are constrained not merely by tech resources, but by our commitment to free speech. Let’s hope the Sun Devils cool off on the filtering…

Cross-posted at Info/Law.

  February 10, 2012 at 5:10 pm   Posted in: Architecture, Civil Rights, Constitutional Law, Current Events, Cyber Civil Rights, Cyberlaw, First Amendment, Politics, Social Network Websites, Technology, Web 2.0   Print This Post   No Comments

Symposium Next Week on “A Legal Theory for Autonomous Artificial Agents”

posted by Frank Pasquale

On February 14-16, we will host an online symposium on A Legal Theory for Autonomous Artificial Agents, by Samir Chopra and Laurence White. Given the great discussions at our previous symposiums for Tim Wu’s Master Switch  and Jonathan Zittrain’s Future of the Internet, I’m sure this one will be a treat.  Participants will include Ken Anderson, Ryan Calo, James Grimmelmann, Sonia Katyal, Ian Kerr, Andrea Matwyshyn, Deborah DeMott, Paul Ohm,  Ugo Pagallo, Lawrence Solum, Ramesh Subramanian and Harry Surden.  Chopra will be reading their posts and responding here, too.  I discussed the book with Chopra and Grimmelmann in Brooklyn a few months ago, and I believe the audience found fascinating the many present and future scenarios raised in it.  (If you’re interested in Google’s autonomous cars, drones, robots, or even the annoying little Microsoft paperclip guy, you’ll find something intriguing in the book.)

There is an introduction to the book below the fold.  (Chapter 2 of the book was published in the Illinois Journal of Law, Technology and Policy, and can be found online at SSRN).  We look forward to hosting the discussion!

Read the rest of this post »

  February 8, 2012 at 10:43 am  Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents  Posted in: Contract Law & Beyond, Criminal Law, Current Events, Cyberlaw, Social Network Websites, Symposium (Autonomous Artificial Agents), Technology, Tort Law   Print This Post   11 Comments

• « Older Entries
• Newer Entries »