When Police Let Algorithms Investigate

Key quote:

What looks like a second, independent identification is in fact the same algorithmic error surfacing twice, creating an illusion of confirmation that drives officers to treat the result as reliable.

Why it matters: Robert Dillon is a commercial crabber from Fort Myers who’d never been to Jacksonville Beach and was arrested due to a combination of faulty technologies and questionable decisions by the “human in the loop”. An AI-powered facial recognition system called FACES flagged him as a “93% match” to a grainy surveillance photo of a man who tried to lure a child from a McDonald’s. Officer Scott O’Connell built his warrant application around that number and left out everything that contradicted it, like the negative license plate reader results. He also omitted the manager’s statement that the suspect was a regular customer 300 miles away, and the scar Dillon described on their phone call. Under Franks v. Delaware, if those omitted facts were to go back in, the probable cause disappears because the magistrate never got to weigh them.

But the Franks theory is the floor, not the ceiling, of this complaint because two other arguments matter more for anyone running shared technology systems. The first is what the complaint calls an “illusion of confirmation” where a photo lineup built around an FRT-identified candidate doesn’t corroborate the algorithm. The fillers are chosen to resemble the candidate, not the actual perpetrator, so the candidate will almost always be the closest visual match in the array. The lineup doesn’t produce a second, independent identification because it reproduces the same algorithmic error dressed up as human judgment. More than half of the 15 known FRT-driven wrongful arrests in the U.S. involved exactly this pattern as a design flaw in the investigative methodology.

The second is the Monell claim against Pinellas County Sheriff Bob Gualtieri, whose office operates FACES and distributes it to 196+ agencies. The complaint argues that running a 38.5 million image database without minimum image quality thresholds is itself the municipal “policy” that caused the constitutional violation. If a court buys this, the exposure travels upstream to every agency operating a shared FRT database rather than just the ones mishandling results. That’s a different calculus than suing one officer, especially since Sheriff T.K. Waters publicly said he’d throw an officer out of his office for treating an FRT hit as probable cause. However, his own sergeant, James Walters, transmitted the result as “a 93% match on facial features” with no qualifying language, creating a gap that Monell claims address.

Twenty jurisdictions have already banned police use of facial recognition, while Detroit and Indiana prohibit arrests based on FRT results followed by photo lineups. The question in Dillon v. City of Jacksonville Beach isn’t whether the technology is imperfect – since everyone agrees it is. The real issue is whether the institutional refusal to build safeguards around a known-imperfect system counts as a constitutional violation in itself. The answer to that will shape who bears the cost the next time an algorithm points at the wrong person.