Generative AI in Gaming: Muse and the Future of Game Design[1]Since middle school, many Americans have spent countless hours immersed in video games. From the iconic[2]Resident Evil[3]and Final Fantasy to modern hits like World of Warcraft and Fortnite , gaming has become a staple of American culture. But the gaming industry isn’t just about entertainment; it’s a multi-billion-dollar powerhouse that has surpassed Hollywood in revenue. Now, with the introduction of generative AI models like Muse, the future of game design is poised for a revolutionary shift.

This week, Microsoft Research released Muse, an AI designed to generate diverse new scenarios within a game. Like ChatGPT and Gemini, Muse is a generative model trained on roughly 500,000 human gameplay sessions from Microsoft-owned Ninja Theory’s multiplayer shooter Bleeding Edge. The AI can create gameplay scenarios where characters obey the game’s internal physical rules and associated controller actions.

The team behind Muse emphasizes that the AI isn’t intended to replace human game designers. Instead, it serves as a creative tool to inspire designers. As study author Katja Hofmann wrote in a blog post, “In our research, we focus on exploring the capabilities that models like Muse need to effectively support human creatives.”

Muse is currently trained on Bleeding Edge, but its potential extends far beyond this single game. The AI learned from human gameplay data without any preconception of the game’s physics, making it adaptable to other games with sufficient training data. “We believe generative AI can boost this creativity and open up new possibilities,” wrote Fatima Kardar, corporate vice president of gaming AI at Microsoft, in a separate blog post.

Whole New Worlds

Generative AI has already made significant strides in the digital universe, and game developers are now exploring its potential to build entirely new worlds. In 2024, Google introduced GameNGen, the first game engine powered by neural networks. GameNGen recreated the classic video game Doomwithout accessing the game’s original code, instead learning from millions of small decisions made during gameplay. The result is an AI-based copy that can be played for up to 20 seconds with all its original functionality intact.

Modern video games, however, present a more complex challenge. Most games are now in 3D, with unique worlds and physical rules. A game’s maps, non-player characters, and other designs can change with version updates, but the core mechanics—how a player interacts with the virtual world—remain consistent. Glitches can be fun, but if the game’s physics constantly break, players lose their sense of immersion.

To better understand how AI could help, the team interviewed 27 video game designers from indie studios and industry giants across multiple continents. Several themes emerged, including the need for new and different scenarios that still maintain the game’s framework. “Generative AI still has kind of a limited amount of context,” one designer noted. “This means it’s difficult for an AI to consider the entire experience…and following specific rules and mechanics [inside the game].”

Other designers emphasized the need for iteration and divergent paths. An assistant AI should be flexible enough to adopt designer-proposed changes, and different player actions should have meaningful consequences. “Divergent paths were also a top priority, in that if a player chooses a different action, those actions will each have different and meaningful consequences.”

WHAM: The World and Human Action Model

Based on this feedback, the team created WHAM, or the World and Human Action Model, nicknamed Muse. Each part of the AI was carefully crafted to accommodate the game designers’ needs. Its backbone algorithm is similar to the one powering ChatGPT and has previously been used to model gaming worlds.

The team fed Muse on human gameplay data gathered from Bleeding Edge , a four versus four collaborative shooter game in 3D. With videos from the battles and controller input, the AI learned how to navigate the game from the equivalent of seven years of continuous play. When given a prompt, Muse could generate new scenarios in the game and their associated controller inputs. The characters and objects obeyed the game’s physical laws and branched out in new explorations that matched the game’s atmosphere. Newly added objects or players stayed consistent through multiple scenes.

“What’s groundbreaking about Muse is its detailed understanding of the 3D game world, including game physics and how the game reacts to players’ controller actions,” wrote Kardar.

Not everyone is convinced the AI could help with gaming design. Muse requires tons of training data, which most smaller studios don’t have. “Microsoft spent seven years collecting data and training these models to demonstrate that you can actually do it,” Georgios Yannakakis at the University of Malta told New Scientist. “But would an actual game studio afford [to do] this?”

Skepticism aside, the team is exploring ways to further develop the technology. One idea is to “clone” classic games that can no longer be played on current hardware. According to Kardar, the team wants to one day revive nostalgic games. “Today, countless classic games tied to aging hardware are no longer playable by most people. Thanks to this breakthrough, we are exploring the potential for Muse to take older back catalog games from our studios and optimize them for any device,” she wrote.

Meanwhile, the technology could also be adapted for use in the physical world. For example, because Muse “sees” environments, it could potentially help designers reconfigure a kitchen or play with building layouts by exploring different scenarios.

“From the perspective of computer science research, it’s pretty amazing, and the future applications of this are likely to be transformative for creators,” wrote Peter Lee, president of Microsoft Research.

As the gaming industry continues to evolve, generative AI models like Muse offer a glimpse into the future of game design. With the ability to create new worlds, scenarios, and experiences, AI could revolutionize how games are developed and played. Whether it’s reviving classic games or helping designers create more immersive experiences, the potential applications are vast and exciting. The future of gaming is here, and it’s powered by AI.