Talk at CEDEC: Joshua Romoff talks about how we can build Efficient player bots using Deep Reinforcement Learning

This year Joshua Romoff had the opportunity to give at talk at CEDEC, a world renown video game conference that takes place in every year in Japan. It is Japan's largest conference for computer entertainment, where developers share their knowledge and experience on video game technology.

Joshua Romoff, who is a Research and Development Scientist at Ubisoft La Forge, gave a talk titled "Deep Reinforcement Learning for Efficient Player Bots," which shed light on how this cutting-edge technology is reshaping the way we approach AI in gaming. This falls in line with Joshua's work within Ubisoft which focuses working on developing state of the art video game prototypes using Deep Reinforcement Learning.

Deep Reinforcement Learning (DRL) is a subset of artificial intelligence that has been making waves across various industries, and the gaming world is no exception. Ubisoft La Forge is pushing the boundaries by employing DRL to create more realistic and believable player bots.

Navigating Game Worlds:

One of the fascinating applications of DRL discussed in the talk was navigation. Traditional AI methods, such as a NavMesh,  often struggle to handle complex and dynamic game environments. Using DRL, offers a way to train bots to navigate with player controls, allowing for more believable navigation patterns and a better experience for the player.

For more information on this check out this blog post: https://www.ubisoft.com/en-us/studio/laforge/news/6bRtGllmfhuDqTHRS6KVLj/deep-reinforcement-learning-for-navigation-in-aaa-video-games.

Specifying Specific Behaviors

The talk also touched on the idea of using DRL to specify specific behaviors. This means that game developers can train bots to perform highly tailored behaviors, creating an easier to use pipeline and opening the so-called reinforcement learning black box. They highlighted an example inspired by self-driving cars, showing how DRL can be used to teach bots intricate maneuvers like respecting speed limits and staying on a particular path.

If you want to learn more about the techniques used to control our bots, check out the following blog posts: https://www.ubisoft.com/en-us/studio/laforge/news/3l6xqVYzNUsd8UgfKFQk4G/constrained-reinforcement-learning-and-self-driving-cars-an-interns-journey  and https://www.ubisoft.com/en-us/studio/laforge/news/6phoEAQOzIdkiMv8T2kLNS/direct-behavior-specification-via-constrained-reinforcement-learning.

Teaching bots how to fight in  "For Honor"

One of the most exciting aspects of the presentation was the application of DRL in Ubisoft's "For Honor."

This competitive fighting game posed an interesting challenge to training DRL bots since, unlike in navigation where the task is very self-contained, winning in a fighting game is completely dependent on your opponent.  Building a bot that can't be easily exploited by players is not a simple task. To accomplish this, traditional methods typically cheat - by knowing exactly what the player is about to do and countering it. Instead, we chose to train our DRL agent using competitive self-play, where the agent continuously learns against previous versions of itself. This led to some very interesting results and a bot that is pretty hard to beat!

Here's a video of our new AI, called the DeepBot, training. Each DeepBot trains in a 1on1 duel, with 10 matches per instance of the game running. We then run 5 instances of the game, each running at double speed, making for a DeepBot that trains 50 matches at once at twice the speed.

Take away:

Although there is still a lot to discover and many hurdles to overcome, by harnessing the power of DRL, Ubisoft is paving the way for more engaging player bots, ultimately enhancing the gaming experience for players.

Stay tuned for more exciting updates from Ubisoft La Forge!