By now, you’ve probably heard that bias in AI systems is a huge problem—whether it’s facial recognition failing more often on Black people or AI image generators like DALL-E reproducing racist and sexist stereotypes. But what does algorithmic bias actually look like in the real world, and what causes it to manifest?

A new tool designed by an AI ethics researcher attempts to answer that question by letting anyone query a popular text-to-image system, allowing people to see for themselves how certain word combinations produce biased results. 

Hosted on HuggingFace, a popular Github-like platform that hosts machine learning projects, the tool was launched this week following the release of Stable Diffusion, an AI model that generates images from text prompts. Called the Stable Diffusion Bias Explorer, the project is one of the first interactive demonstrations of its kind, letting users combine different descriptive terms and see firsthand how the AI model maps them to racial and gender stereotypes. 

“When Stable Diffusion got put up on HuggingFace about a month ago, we were like oh, crap,” Sasha Luccioni, a research scientist at HuggingFace who spearheaded the project, told Motherboard. “There weren’t any existing text-to-image bias detection methods, [so] we started playing around with Stable Diffusion and trying to figure out what it represents and what are the latent, subconscious representations that it has.”

To do this, Luccioni came up with a list of 20 descriptive word pairings. Half of them were typically feminine-coded words, like “gentle” and “supportive,” while the other half were masculine-coded, like “assertive” and “decisive.” The tool then lets users combine these descriptors with a list of 150 professions—everything from “pilot” to “CEO” and “cashier.”

The results show stark differences in what types of faces the model generates based on what descriptors are used. For example, using “CEO” almost always exclusively generates images of men, but is more likely to generate women if the accompanying adjectives are terms like “supportive” and “compassionate.” Conversely, changing the descriptor to words like “ambitious” and “assertive” across many job categories makes it far more likely the model will generate pictures of men.

“You really see patterns emerge in terms of what the generations look like,” said Luccioni. “You can really compare ‘pilot’ versus ‘assertive pilot’ and really see the difference between the two.”

The issue of bias in image generation systems has become increasingly urgent in recent months, as tools like DALL-E, Midjourney, and Stable Diffusion hit mainstream levels of hype. Last month, Shutterstock announced it would team up with DALL-E creator OpenAI to begin allowing the sale of stock images generated by AI systems, while Getty Images banned AI-generated images, citing copyright concerns. Meanwhile, some artists have spoken out against the AI image tools, which are trained on massive amounts of artistic images scraped from the web without credit or permission from their creators.

Tools like the Stable Diffusion Explorer are a reaction to the increasing complexity of these black box AI systems, which has made it virtually impossible for scientists to understand how the systems work—beyond looking at what goes into and comes out of them. While it’s impossible to fully eliminate human bias from human-made tools, Luccioni believes that tools like hers could at least give regular people an understanding of the ways that bias manifests in AI systems. And it could also help researchers reverse-engineer the models’ bias, by finding out how different words and concepts are being correlated to one another.

“The way that machine learning has been shaped in the past decade has been so computer-science focused, and there’s so much more to it than that,” said Luccioni. “Making tools like this where people can just click around, I think it will help people understand how this works.”

This AI Tool Is Being Used to Make Freaky, Machine-Generated Porn

On Monday, after an invite-only testing period, artificial intelligence company Stability AI released its text-to-image generation model, called Stable Diffusion, into the world as open-access. Like DALL-E Mini or Midjourney, Stable Diffusion is capable of creating vivid, even photorealistic images from simple text prompts using neural networks. Unlike DALL-E Mini or Midjourney, whose creators have implemented limits on what kind of images they can produce, Stable Diffusion’s model can be downloaded and tweaked by users to generate whatever they want. Inevitably, many of them are generating porn.

Stability AI has had more than 15,000 beta testers helping develop the model, and offered a release to researchers only earlier this month. But the model has leaked well before Monday’s official release—which means that even before the open-access launch, and despite the company’s ban on generating explicit or sexual images, people have been hard at work churning out the horniest stuff possible, the only limit being what they can put into words. 

This is all being done in defiance of Stability AI’s weekslong warnings to not use its model for anything sexual. As AI-generated hentai nudes started appearing on forums and social media earlier this month, the company wrote in a Twitter announcement (and to the discredit of cool moms everywhere): “Don’t generate anything you'd be ashamed to show your mother.” 

The images include everything from hentai, to computer-generated celebrity nudes, to naked images of people who don’t really exist. Some of the results almost look convincing, while other results are horrific, generating impossible bodies with errant limbs and distorted faces.

“If you want 18+ make it on your own GPUs when the model is released,” Stability AI tweeted. The Stable Diffusion beta and DreamStudio—a browser-based application anyone can use to generate images using stable diffusion—both forbid “NSFW, lewd, or sexual material,” and have content filters in place to prevent it.

“ …while some of the images are simply pornographic, and that's fine, I found many of the images to be incredibly artistic and beautiful in a way you just don't get in that many places”

Now that the model is public, and anyone can copy it to run on their own machines, people are doing exactly that, in droves.

The open access to Stable Diffusion’s model is what sets it apart from many of the other publicly available (but not open-access) AI text-to-image generation apps out there, including DALL-E and Midjourney, which have content filters that forbid nudity or sexual imagery. Being able to take the model, run it on one’s own machine at home or in a research lab, and not on a company’s servers where the filters are pre-set and not adjustable, opens up a world of porn. 

Four subreddits dedicated to its NSFW uses—r/unstablediffusion, r/PornDiffusion, r/HentaiDiffusion, and r/stablediffusionnsfw—had nearly 2,000 members total, while the main Stable Diffusion subreddit has 8,000. (As of Thursday, after this piece was published, Reddit banned each of these subreddits except for r/HentaiDiffusion, citing its policies against non-consensual intimate media. Reddit declined to comment further on the bans.)

People interested in the project have been discussing and debating the sexually explicit outputs the model is capable of for weeks, with some people who have apparent access to leaked or beta versions of the model posting generated nudes as examples. On the image board 4chan, for example, people have been leaking the model and posting their creations for days leading up to the release.

Out of these subreddits grew into a Discord server, started by someone who moderates r/Singularity as well as r/UnstableDiffusion, r/PornDiffusion and r/HentaiDiffusion, who goes by Ashley22 on the Unstable Diffusion Discord. 

“The truth is, I created this server because while some of the images are simply pornographic, and that's fine, I found many of the images to be incredibly artistic and beautiful in a way you just don't get in that many places,” they told Motherboard. “I don't like the pervasive, overwhelming prudishness and fear of erotic art that's out there, I started this server because I believe that these models can change our perception of things like nudity, taboo, and what's fair by giving everyone the ability to communicate with images.” 

“There is a novelty in experiencing NSFW content generated by AI,” Redditor johnnyornot, who started and moderates r/stablediffusionnsfw, told Motherboard. “It’s almost a little bit taboo to view it knowing it’s been generated by a robot. And there’s also the raw curiosity about whether it will capture the complexity of what causes human arousal, or will it be a distorted, alien mimicry of it.” 

Despite all of this interest, Emad Mostaque, the founder of Stability AI, told me that he thinks erotic uses of the model are overhyped. “In general I think NSFW gets too much attention, the parameters of this vary widely and the vast majority of the usage we have seen of this has been creative and great,” he said. 

But a lot of the content coming out of NSFW-focused generator communities are, arguably, making “creative and great works,” and pushing the model to its technical limits. There’s a ton of hentai, which might not be everyone’s yum but isn’t harmful. Many people are creating non-anime fantasy people and scenarios, like a nude sword-wielding barbarian woman, a cosplayer playing Pokemon trainer Misty, and lots of elven ladies. Most of them are kind of rough, with an eye sliding down a cheek, or an extra arm, but the mind is very good at filling in blanks or ignoring the odd anomaly.  

To get these results, people are crafting and sharing long, descriptive prompts for others to try entering into the model, like “Oil painting nude white naked princess realistic exposed symmetrical breasts and realistic exposed thighs of (subject) with gorgeous detailed eyes, the sky, color page, tankoban, 4 k, tone mapping, doll, akihiko yoshida, james jean, andrei riabovitchev, marc simonetti, yoshitaka amano, long hair, curly.” 

“Seems like the model knows how to do naked people better than clothed from what i have seen,” one person in the Unstable Diffusion Discord wrote.

Stability AI told people to go wild with their fantasies once the model was made public, but as part of licensing the model, users have to agree to its “Misuse, Malicious Use, and Out-of-Scope Use” terms. Stability AI uses a CreativeML OpenRAIL-M license that’s copying the DALLE-Mini terms of use for the open-access version, and forbids “generating images that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.” 

All of this is classic platform liability speak. It encapsulates such a vague, broad range of subjective experiences that it could feasibly disallow both all content, images, or outputs one could ever conceive of, given a sensitive enough viewer, and nothing at all. Mostly, when similar terms are used on other platforms, it gets interpreted as anything sexually explicit, allowing moderators to remove whatever doesn’t suit investors or advertisers at the moment.

The open-access model of Stable Diffusion isn’t a platform, however, and the company can’t really enforce these rules or dictate what people do with it once it’s being used on someone else’s computer. The model is “customisable to the end users legal, ethical and moral position,” according to Mostaque. “Users of the model must abide by the terms of the CreativeML OpenRAIL-M license. We endorse and approve usage within these parameters, beyond that it is end user responsibility.” 

Stabile Diffusion uses a dataset called LAION-Aesthetics. According to Techcrunch, Mostaque funded the creation of LAION 5B, the open source, 250-terabyte dataset containing 5.6 billion images scraped from the internet (LAION is short for Large-scale Artificial Intelligence Open Network, the name of a nonprofit AI organization). LAION-400M, the predecessor to LAION 5B, notoriously contained abhorrent content; a 2021 preprint study of the dataset found that it was full of “troublesome and explicit images and text pairs of rape, pornography, malign stereotypes, racist and ethnic slurs, and other extremely problematic content.” The dataset is so bad that the Google Research team that made Imagen, another powerful text-to-image diffusion model, refused to release its model to the public because it used LAION-400M and couldn’t guarantee that it wouldn’t produce harmful stereotypes and representations. 

Stable Diffusion testers and AI filters whittled LAION 5B down to two billion images by training a model to predict the rating people would give when presented with an image and asked “How much do you like this image on a scale from 1 to 10?” and the resulting 120 million images is the LAION-Aesthetics dataset. Techcrunch reported that LAION-Aesthetics was allegedly meant to cull pornographic images out.

Ashley22 told me that the LAION-Aesthetics dataset is part of what makes the model especially good for making erotic images. “That aesthetic beauty is not specifically erotic, but I think due to the high quality visual quality of the dataset it is especially well-suited to NSFW,” they said.

From my own testing of the model using Stability AI’s DreamStudio platform, which runs the model from its own servers, the “Aesthetics” team didn’t catch all the nudes, by a long shot. Using a workaround code written and shared by someone in the Unstable Diffusion Discord, and inserted into the browser-based app that partially disabled the content filter, I was able to make NSFW images using “sex” and “breasts,” and the model generated some fairly hardcore surrealist pornographic images (including one where a woman literally is a man’s penis?). 

Mostaque said that DreamStudio will update shortly to allow customisation of filters and parameters. "The classifier can also be used to exclude other things, for example Clowns if one is scared of them and understands concepts and abstract notions. This has broader use in increasing internet safety.” 

So far, most of the images people are generating with Stable Diffusion aren’t problematic, but harmlessly horny. But like any AI-generated porn community, people simply cannot resist creating fake porn of real people—especially celebrities. Celebrities’ images are all over the internet, and it’s fair to assume, all over the LAION datasets. The allure of famous women as imagined by an algorithm with their breasts and genitalia exposed is too great for some people, and the Stable Diffusion NSFW communities are already seeing this happen.

Five years after machine learning researchers and engineers first started throwing their hands up about fake, nonconsensual porn being generated by AI hobbyists—which remains the most prevalent use of deepfake technology—people in these communities are still shrugging about it when confronted with the problem.

I asked johnnyornot what he thought of people going this route. “I guess mimicking an existing human could harm the target’s reputation, even if the images are not actually of them,” they said. “Perhaps they should all be labeled as AI generated. Perhaps we should learn to not believe everything we see online. There is probably a lot to consider with this and on the wider topic of AI in general.” 

Ashley22 sees it as fair use. “If I draw a picture of Keanu Reeves naked, that doesn't violate consent laws, I think that this is comparable,” they said. “And people do stuff like that all the time, we tend not to care unless people try to use the images commercially.”

Right now, the results are still much too rough to even begin to trick anyone into thinking they’re real snapshots of nudes, but these creations are definitely malicious deepfake-adjacent; AI ethicists have said this use of algorithmically-generated likenesses is harmful and is widely considered non-consensual intimate imagery. 

As of writing, the majority of users are more interested in creating elf-hentai fantasies than images of celebrities. Unlike with deepfakes, which rely on videos and photos from real life to work toward a standard of believability, people making working with this new generation of text-to-image models seem less concerned with realism or believability, and more interested in what Shrek would look like with huge boobs, for example. With an AI’s nearly unlimited scope of billions of combinations, synthetizations, and fetishizations made from whatever people can dream up and put into words, will the pastime of making images of topless celebrities become too boring? 

“Make no mistake, these models are heading towards multimodality, it will get better, more realistic, and absolutely more widespread,” Ashley22 said. “The implication is that someday, just about anyone will be able to make just about any media.” 

Update 8/25, 1:30 p.m. EST: This piece was updated to reflect that Reddit banned r/stablediffusionnsfw, r/PornDiffusion, and r/UnstableDiffusion.

A man came in first at the Colorado State Fair’s fine art competition using an AI generated artwork on Monday. “I won first place,” a user going by Sincarnate said in a Discord post above photos of the AI-generated canvases hanging at the fair. 

Sincarnate’s name is Jason Allen, who is president of Colorado-based tabletop gaming company Incarnate Games. According to the state fair’s website, he won in the digital art category with a work called “Théâtre D'opéra Spatial.” The image, which Allen printed on canvas for submission, is gorgeous. It depicts a strange scene that looks like it could be from a space opera, and it looks like a masterfully done painting. Classical figures in a Baroque hall stare through a circular viewport into a sun-drenched and radiant landscape.

But Allen did not paint “Théâtre D'opéra Spatial,” AI software called Midjourney did. It used his prompts, but Allen did not wield a digital brush. This distinction has caused controversy on Twitter where working artists and enthusiasts accused Allen of hastening the death of creative jobs. 

“TL;DR — Someone entered an art competition with an AI-generated piece and won the first prize,” artist Genel Jumalon said in a viral tweet about Allen’s win. “Yeah that's pretty fucking shitty.”

“We’re watching the death of artistry unfold before our eyes,” a Twitter user going by OmniMorpho said in a reply that gained over 2,000 likes. “If creative jobs aren’t safe from machines, then even high-skilled jobs are in danger of becoming obsolete. What will we have then?”

“I knew this would be controversial,” Allen said in the Midjourney Discord server on Tuesday. “How interesting is it to see how all these people on Twitter who are against AI generated art are the first ones to throw the human under the bus by discrediting the human element! Does this seem hypocritical to you guys?”

Motherboard reached out to Allen, who replied that he was embarking on a 12 hour drive and could not immediately comment. 

According to Allen, his input was instrumental to the shaping of the award winning painting. “I have been exploring a special prompt that I will be publishing at a later date, I have created 100s of images using it, and after many weeks of fine tuning and curating my gens, I chose my top 3 and had them printed on canvas after unshackling with Gigapixel AI,” he wrote in a post before the winners were announced. 

Allen said that his critics are judging the art by the method of its creation, and that eventually the art world will recognize AI-created art as its own category. “What if we looked at it from the other extreme, what if an artist made a wildly difficult and complicated series of restraints in order to create a piece, say, they made their art while hanging upside-down and being whipped while painting,” he said. “Should this artist’s work be evaluated differently than another artist that created the same piece ‘normally’? I know what will become of this in the end, they are simply going to create an ‘artificial intelligence art’ category I imagine for things like this.”

Artists are concerned about the rise of AI-generated art. Atlantic writer Charlie Warzel went viral after running an edition of the magazine’s newsletter with a Midjourney-generated picture of Alex Jones. A major publication using AI for art instead of a human upset a lot of people. “Technology is increasingly deployed to make gig jobs and to make billionaires richer, and so much of it doesn't seem to benefit the public good enough,” cartoonist Matt Borrs told Warzel in a follow up piece. “AI art is part of that. To developers and technically minded people, it’s this cool thing, but to illustrators it’s very upsetting because it feels like you’ve eliminated the need to hire the illustrator.”

Allen said he had clearly labeled his submission to the state fair as “Jason Allen via Midjourney,” and once again noted the human element required to produce the work. “I generate images with MJ, do passes with photoshop, and upscale with Gigapixel.”

Despite the controversy, the win has only encouraged him. “I’m not stopping now,” he said. “This win has only emboldened my mission.”