It’s right behind you, breathing down your neck, seemingly aware of your every move. You turn left, then right, weaving between pillars to collect precious orbs while dodging its advance. For a moment, it seems you’ve lost it. But then your foe appears around a corner, and bam! It has you.

The familiar Game Over screen appears—but then, just below, something different. “Please report difficulty level.” You lasted just a few seconds in this hair-raising chase, so, adrenaline still pumping, you tap “Hard.” Next time, the thing stalking your every step will be a tad less aggressive.

Romain Trachel and Alexandre Peyrot, machine-learning specialists at Eidos-Sherbrooke, demonstrated the game I just described at Unreal Fest 2022. It combines machine learning with an Unreal Engine feature called the Environment Query System (EQS), which lets developers use spatial data to inform AI decisions. 

Normally, this is handled through behavior trees that layer variables and branching possibilities. But in this demo, the AI behavior is driven by a machine-learning model. Unreal EQS acts as the AI’s eyes and ears, providing information about its environment, while the machine-learning model becomes its brain and decides how it should respond.

The game is not as frightful as I made it sound, mostly because of its top-down presentation and placeholder visuals, but its gameplay is a classic cat-and-mouse chase that tasks players with collecting orbs strewn across a map. It’s Pac-Man, basically—but the ghost’s behaviors are no longer scripted.

“So, for instance, if a developer decided to activate a stronger chase mode, the only thing to do is to increase a reference value in the EQS tests,” Trachel and Peyrot say in an email. “It really has the potential to simplify the development workflow, because in actual game productions, it would be up to a game designer to decide which game variables must be tuned in order to change the difficulty.”

The key phrase in this explanation is “up to a game designer.” A traditional behavior tree can become unwieldy, requiring back-and-forth between designers, programmers, and other developers to fine-tune behavior. Tweaking a machine-learning model could be an easier option, giving designers a way to model difficulty without diving into branches of a behavior tree. Able to put that aside, designers may be better able to focus on what’s important: whether the AI makes the game feel more challenging and more fun.

Machine learning can be used to create a brutal foe. IBM’s Deep Blue and Google’s DeepMind AlphaStar have proven that. However, that isn’t always the desirable—not only because it raises the difficulty, but also because the AI’s specific tactics may run counter to enjoyable gameplay. 

Trachel and Peyrot tried using AI for several game modes, including a “multi-output model” that learned to predict the player’s score (earned by collecting orbs) and cut them off. “But in this game mode, the enemy tended to camp on the orbs’ positions. It wasn’t fun and engaging to play against, so we didn’t show these results.”

Camping at the location of orbs is a solid strategy: The player must pick up orbs to win (imagine if Pac-Man’s ghosts merely lingered near the entrances to each corner of the map). It also makes the game less fun. Players no longer experience an exciting chase. Instead, the AI might spring an unpredictable ambush. Trachel and Peyrot say their goal is “not to create superhuman bots—that would not be fun and engaging for a novice player—but instead to find ways to incorporate machine learning into game AI tools already used in production.”

That might sound dull to players craving better AI. Yet the machine-learning techniques shown by Trachel and Peyrot remain helpful for tuning difficulty even when the foes that players face in the finished game don’t use it. Julian Togelius, cofounder and research director at Modl.ai, has spent nearly five years using AI to test games. Modl.ai uses bots to hunt graphical glitches, find flaws in world geometry, and sniff out situations that make it impossible to win. 

“You can tell us what kind of failure state you are interested in. And then basically it runs. You send off a job, and it runs depending on how much you want to explore,” says Togelius. “And of course, we can cluster these for you and provide a report, saying here’s where you seem to have issues, and so on.” 

Modl.ai’s testing bots use machine learning to adapt to each game tested, though its current implementation limits those adaptations to each specific title. Togelius says the company is prototyping the addition of deep learning that will train bot behavior across multiple games. Once in use, Modl.ai’s bots will learn to emulate the behavior of real players, which should more efficiently uncover issues that players would find.

When it comes to difficulty, then, machine learning can be both a problem and a solution. But crafting a fair, fun challenge isn’t the only hurdle facing developers who want to use machine learning in games. The problems run deeper—so deep, in fact, they may force a rethink of how games are built.

Performance is one barrier. Machine learning requires lots of training data for worthwhile results, and that data can only be acquired by playing a game thousands or tens of thousands of times (though bots can lighten the load, a tactic Trachel and Peyrot used in building their demo). And once the training data is collected, the resulting model can become burdensome to execute in real time. 

“Yes, performance is clearly an issue, notably with large ML models that process frames for each tick of the game clock,” Trachel and Peyrot said in an email. “In our case, to avoid performance issues, we used a small neural network that was only inferring at precise moments of the game.” Scaling up to the huge open-world environments that modern players expect is another matter entirely.

Togelius says the way modern game engines work exacerbates the problem. Machine learning, he says, “will by necessity be slow because game engines are not built for this. One of the many reasons we don’t see more interesting modern AI in games is because Unreal and Unity and all their ilk are basically terrible—anti-AI in so many ways.” 

Animation is another issue. Most modern game engines expect animations to be strictly defined frame by frame. This works well when animators know with certainty how game characters will behave, but an AI controlled by machine learning might behave in ways the animators didn’t expect. Designers can work around this with a physics-based approach to animation, but this places even more performance strain on a game console or computer’s hardware and comes with its own development challenges.

In short, developers face a monster of their own making. Game engines are built to use behavior trees and prescripted actions to craft worlds of AI-controlled NPCs that work well even on meager hardware. But as machine learning gains steam, these classic solutions will need to be reconsidered.

“If you go talk to a machine-learning researcher who doesn’t know game design, they’ll be like, ‘Why don’t you use new things and get NPCs that are more lifelike and adapt to how you play,’ and so on,” says Togelius. “But you can’t just plug this into an existing game. You have to rethink what the game even is.”