From BuzzFeed News
We Tested The FBI’s Most Wanted List On Amazon’s Celebrity Recognition Tool. Then Things Got Weird.
By Davey Alba and the BuzzFeed Tech Team
Earlier this week, as a follow-up to the ACLU's excellent report on Amazon’s facial recognition tool, Rekognition, BuzzFeed News decided to run a few experiments of our own. Among the tests, we ran photos of the FBI's historical Most Wanted list (507 photos) and the National Institute of Standards and Technology's mugshot identification database (3,248 photos) through Amazon’s celebrity recognition tool — a feature Amazon introduced in June 2017, six months after the broader Rekognition platform launched. (Amazon, as you’ll see below, says this was a very silly thing to do.)
Things got weird. On the FBI test, Amazon’s tool returned 17 matches that identified criminals as celebrities with 95% to 100% confidence — among them the late Grateful Dead guitarist Jerry Garcia, Sons of Anarchy actor Kim Coates, the late television painting instructor Bob Ross, and Darryl McDaniels, a member of Run DMC. In our second test, using NIST’s mugshot database, the celebrity recognition tool matched, with 96% confidence, the arrest photo of a black man to former secretary of state Condoleezza Rice...
“If they’re different services, it wouldn’t be fair to compare them,” Joshua Kroll, a computer scientist at the School of Information at UC Berkeley, told BuzzFeed News. “But if they present it all under one brand, then as a consumer you just think you know how it works — and that’s confusing at best.” ...
Kroll said confusion around Amazon’s facial recognition capabilities could be mitigated if the company would disclose more details around how it builds and runs its AI recognition tools. The inscrutability in the system is “the company’s own choice,” he said. It’s proprietary technology, and Amazon considers it a trade secret. In essence, Amazon’s facial recognition platform is a black box.
“Amazon is not particularly open about how its system works or what the confidence score even means with respect to the model,” [Kroll] said. “The company just says in its documentation that higher scores are better, with scores going from 0 to 100.” (After the ACLU’s report last week, Amazon recommended setting a confidence threshold of 95% and higher when using Rekognition. A day later, the company nudged that up to 99% for sensitive applications of its face recognition tools, such as law enforcement activities.)
And yet these confidence scores, Kroll told BuzzFeed News, are inherently meaningless. “Confidence scores are very specific to the AI model you’ve built, and the way you’ve trained it,” he said.
Joshua A. Kroll is a computer scientist studying the relationship between governance, public policy, and computer systems. He is a Postdoctoral Research Scholar at the School of Information.
