As my work on Technological Due Process explored, government increasingly uses automated systems to help human administrators make decisions about people’s important rights. Sometimes, the computers make the decisions with varying degrees of oversight. Government decision-making systems include data-matching programs, which compare two or more databases with an algorithmic set of rules that determine the likelihood that two sets of personal identifying information represent the same individual.
Data-matching programs frequently misidentify individuals because they use crude algorithms that cannot distinguish between similar names. Sometimes, this accords with policy. Better to have more false positives when it comes to finding terrorists, than more false negatives. Other times, it’s a problem that humans resolve before anyone gets hurt. Yet, time and again, human operators fall down on the job.
Here’s a recent example. An anti-terrorism facial recognition system scans databases of state driver’s license images to prevent terrorism, reduce fraud, and improve the accuracy of identification documents issued by states. Massachusetts started using the software after receiving a $1.5 million grant from the U.S. Department of Homeland Security. On March 22, Massachusetts resident John Gass received a letter from the state motor vehicles registry informing him that he had to cease driving because his license had been revoked. From various news reports, it seems that the letter did not tell Mr. Glass why he lost his license. It was only after various calls and a hearing with motor vehicle officials that he learned that the system identified his license as evidence of potential fraud. The system flagged Glass because he looked like another driver, not because his image was used to create a fake identity. The motor vehicles registry reinstated his license after ten days of wrangling “to prove he is who he says he is.” Not surprisingly, Gass is not alone. The system picked out more than 1,000 cases last year that resulted in investigations, and some were guilty of nothing more than looking like someone else. Read More