If you have been on social media at all in the past week, chances are you might have seen people sharing photos of themselves tagged #ImageNetRoulette, accompanied by amusing, sometimes less than flattering, annotations. Indeed, you may have been perplexed or even angered by these viral images, as the captions tipped over from amusing to offensive.

As with other recent viral art initiatives like FaceApp and Google Arts & Culture’s art Doppelgänger-finder, people were uploading images of themselves to a website where an AI, trained on the most widely used image recognition database, analyzed what it saw. At one point last week it was spitting out as many as 100,000 labels an hour, according to the New York Times.

When I run my own image through the website, I get labeled “mediatrix: a woman who is a mediator,” which is humorous enough. But scroll through the hashtag on Twitter, and you can see where the amusing gaffes of the algorithm dip into the deeply problematic. I see, for instance, people of color sharing their own labels: a dark-skinned man is labeled as “wrongdoer, offender,” an Asian woman as a “Jihadist.”

As it turns out, this is all part of an art project initiated by the artist Trevor Paglen and an AI researcher, Kate Crawford, aimed at exposing how systemic biases have been passed onto machines through the humans who trained their algorithms. 

ImageNetRoulette’s interpretation of artnet News’s Naomi Rea.

“I’ve been really surprised by the attention it’s gotten online, and heartened by how many people ‘get it’ in terms of seeing the bigger point I’m trying to make with the piece about how dangerous it is for machine learning systems to be in the business of ‘classifying’ humans and how easily those efforts can—and do—go horribly wrong,” Paglen tells artnet News.

He adds that the images uploaded to the site are deleted instantly, and that data of the people using it is not collected.

Training Humans

Paglen and Crawford’s project is on view in an exhibition called “Training Humans,” which opened at the Fondazione Prada’s Osservatorio space in Milan last week, and is on view through February 24. As artificial intelligence and facial recognition technologies have been creeping more and more into our daily lives, the pair wanted to conduct a sort of archaeology of the images used to “recognize” humans in computer vision and AI systems.

Exhibition view of “Kate Crawford, Trevor Paglen: Training Humans” Osservatorio Fondazione Prada, through Februrary 24, 2020. Photo by Marco Cappelletti, courtesy Fondazione Prada.

“Understanding the politics within AI systems matters more than ever, as they are quickly moving into the architecture of social institutions: deciding whom to interview for a job, which students are paying attention in class, which suspects to arrest, and much else,” Paglen and Crawford write in an essay accompanying the exhibition.

Surveillance cameras equipped with facial recognition technology are increasingly being used around the world. Indeed, some have even recently been approved for use in London’s cultural institutions, including the Barbican Center, where Paglen’s work on the subject is about to be shown in an upcoming exhibition.

Paglen and Crawford’s project hones in on an expansive database of photographs called ImageNet, used by researchers to train artificial intelligence systems in how to understand the world. The database was first compiled by researchers at Stanford University in 2009 to develop the algorithms applied in deep learning, the process by which machines are trained to recognize images. It is one of the most widely used training sets for machine learning.

To simplify how it works, an algorithm is trained to recognize images of say, dogs or flowers based on being fed a great number of images labeled “dog” or “flower.” But among the more than 14 million photographs ImageNet was trained on were thousands of photographs of people sorted into descriptive categories from “cheerleaders” to more loaded terms like “slattern, slut, slovenly woman, trollop.”

These labels were assigned manually by humans in labs, or strangers paid to label the images through crowdsourced tools like Amazon’s Mechanical Turk. Humans categorized what they saw in terms of race, gender, age, emotion, and sometimes personal character. In so doing, they injected their own conscious and unconscious opinions and biases into the tissue of the algorithm.

Warning Signs

The outcomes of ImageNet Roulette expose the biases and politics behind these problematic datasets, which are the bedrock of the AI systems used to classify humans today. While AI seems to offer an objective look at what something is, the process of classification at its root is a continuation of embedded systemic biases such as racism and misogyny.

“The project is meant to call attention to the real harms that machine learning systems can perpetuate,” Paglen says. Confronting people with photographs next to the labels given by the AI to identify them shows just how the boundaries between science, history, politics, prejudice, and ideology can be blurred in artificial intelligence, and how the outcomes tilt in favor of those who have the power to build these systems. 

Exhibition view of “Kate Crawford, Trevor Paglen: Training Humans” Osservatorio Fondazione Prada, through Februrary 24, 2020. Photo by Marco Cappelletti, courtesy Fondazione Prada.

The artwork has already had an impact. Last week, in the wake of the attention received by Crawford and Paglen’s project, the researchers behind ImageNet announced that they would scrub more than half of the 1.2 million pictures in the dataset’s “people” category.

“Science progresses through trial and error, through understanding the limitations and flaws of past results,” ImageNet said in a statement. “We believe that ImageNet, as an influential research dataset, deserves to be critically examined, in order for the research community to design better collection methods and build better datasets.”

Still, Paglen and Crawford stress that their project calls attention to the problem of categorizing people in this way at all, given the rapid uptake of these algorithmic systems within institutions from education to healthcare to law enforcement.

“There is no easy technical ‘fix’ by shifting demographics, deleting offensive terms, or seeking equal representation by skin tone,” they write. “The whole endeavor of collecting images, categorizing them, and labeling them is itself a form of politics, filled with questions about who gets to decide what images mean and what kinds of social and political work those representations perform.”