Recently, Google DeepMind introduced SIMA (Scalable Instructable Multiworld Agent)—as its name suggests, a scalable, instructable, multi-world AI agent.
According to the company, this is the first general-purpose AI agent capable of following natural language instructions across a wide range of 3D virtual environments and video games. It is designed to act as a player companion, helping users perform in-game tasks—although it is currently still in the research stage.
For example, SIMA can:
Drive a vehicle in Goat Simulator 3
Mine resources in Satisfactory
Find water sources in Valheim
Pilot a spacecraft, shoot asteroids, and collect resources in No Man’s Sky
Frederic Besse, a research engineer at Google DeepMind involved in the project, said:
“SIMA is able to leverage shared concepts across games to learn better skills and execute instructions more effectively.”
Before SIMA, DeepMind had already made significant progress in AI + gaming, including AlphaStar, a system capable of playing StarCraft II at a professional level against human players. SIMA, however, is described by DeepMind as a “new milestone,” shifting from single-game agents to a general system that works across multiple games and follows language instructions.
A New Way of Gaming
To expose SIMA to diverse environments, the DeepMind team collaborated with multiple game studios to collect keyboard and mouse control data from humans playing 3D environments across 10 different games.
This human gameplay data was then fed into a language model similar to those powering modern chatbots, which learns language understanding from massive text datasets. Based on user commands, SIMA can then perform corresponding in-game actions.
Finally, human evaluators assessed SIMA’s performance across different games, generating data for further fine-tuning.
After this training process, SIMA is capable of responding to hundreds of instructions from players, such as:
“Turn left”
“Go to the spaceship”
“Walk through the gate”
“Chop down a tree”
The system can perform more than 600 types of actions, ranging from exploration and combat to tool usage.
Importantly, researchers deliberately avoided training SIMA on games involving violent content, in line with Google’s AI ethics guidelines.
Tim Harley, another member of the DeepMind team, said:
“This is still very much a research project. However, one can imagine a future where an AI agent like SIMA plays games alongside you and your friends.”
Strong Generalization Ability
Interestingly, even in games it was not explicitly trained on, SIMA performs comparably to specialized agents trained directly on those specific datasets. This suggests that SIMA has a strong ability to generalize across unfamiliar environments.
While this is a promising early result, researchers emphasize that SIMA still requires significant further development before reaching human-level performance across both known and unknown games.
Ultimately, the goal is for SIMA to learn how to play virtually any video game, including open-world titles without linear endings.
However, it is not intended to replace existing game AI systems. Instead, it is envisioned as a collaborative teammate that works alongside players.
As Tim Harley explained:
“SIMA is not trained to win games. It is trained to operate within them and follow instructions.”
