Speaking of Automated Systems
posted by Danielle Citron
Thanks so much to everyone participating in the LTAAA symposium: what a terrific discussion. Given my work on Technological Due Process, I could not help but think about troubled public benefits system in Colorado known as CBMS. Ever since 2004, the system has been riddled with delays, faulty law embedded in code, and system crashes. As the Denver Post reports, the state has a $44 million contract with Deloitte consultants to overhaul the system–its initial installation cost $223 million with other private contractors. CBMS is a mess, with thousands of overpayments, underpayments, delayed benefits, faulty notices, and erroneous eligibility determinations. And worse. In the summer of 2009, 9-year-old Zumante Lucero died after a pharmacy — depending upon the CBMS system — wouldn’t fill his asthma prescription despite proof the family qualified for Medicaid help. In February 2011, CBMS failed eight different tests in a federal review, with auditors pointing to new “serious” problems while saying past failures are “nearly the same” despite five years of fixes. The federal Centers for Medicare and Medicaid Services (CMS), which provides billions of dollars each year for state medical aid, said Colorado risks losing federal money for programs if it doesn’t make changes from the audit. All of this brings to mind whether a legal theory of automated personhood moves this ball forward. Does it help us sort through the mess of opacity, insufficient notice, and troubling and likely unintended delegation of lawmaking to computer programmers? Something for me to chew on as the discussion proceeds.
Image: Wikimedia Commons
February 15, 2012 at 5:55 pm
Tags: A Legal Theory for Autonomous Artificial Agents, artificial agents
Posted in: Administrative Law, Architecture, Symposium (Autonomous Artificial Agents)
Print This Post
Responses (3)
Samir Chopra - February 15, 2012 at 11:42 pm
Danielle, believe it or not, one chapter that got left on the cutting floor addressed precisely these issues – and it was a direct spin-off from your paper. I’ll try and write a separate blog post on that later. Thanks for reminding me of this.
Danielle Citron - February 16, 2012 at 8:35 am
I so look forward hearing all about it! Much thanks for the terrific book and for letting us discuss it. Danielle
Steven M. Bellovin - February 16, 2012 at 10:21 am
Danielle, the history of development of large software projects is replete with expensive failures. One, the FBI’s Trilogy project, was so troubled that they asked the National Research Council to study the situation. The report is at http://www.nap.edu/catalog.php?record_id=10991; the committee’s conclusion was blunt: the effort was “not currently on a path to success” [emphasis in the original]. Regardless of the issues of legal personhood for automated agents, the problem of managing such complex efforts is immense. One can argue whether the blame lies with the procuring organization or the contractor; often, they share the responsibility.
To use a very bad analogy, the systems resulting from such flawed efforts are, if legal persons, “handicapped” because of incompetence or negligent choices by its “parents”. Where then, should liability, civil or criminal, lie? Bad outcomes are not inevitable but they’re remarkably common. (Those interested in the subject should follow Peter Neumann’s RISKS Digest; see http://www.csl.sri.com/users/neumann/#3 for a precis, and perhaps skim ftp://ftp.csl.sri.com/pub/users/neumann/illustrative.pdf or read Peter’s 1995 book Computer-Related Risks.)
Leave a Reply