MIRA: Multiplayer Interactive World Models with Representation Autoencoders#

We train a model to simulate Rocket League, Epic Games' car-football game. It allows four players to play a 2v2 match together, responding in real time to their actions and generating a consistent view of the game at 20 fps, with a resolution of 576p split across the four players. It's a 5B-parameter diffusion transformer paired with a 600M-param video representation codec. We're open-sourcing the dataset as well as training and inference code.

The project is a stepping stone to physical AI, where data is messier and scarcer. Before we can make things work in the controlled video game setting, why would we hope to succeed on real-world data? So this is a demo to show what's possible; the world model tech is not being used to develop Rocket League.

It works#

The model manages to simulate many aspects of Rocket League. The cars move, they can kick the ball, you can score goals. The four cars move in sync on the four screens. The model tracks how much boost you have and doesn't let you boost when you're out. Demolitions work. It shows event messages like "Shot on goal" or "Save".

That sounds trivial when said out loud, but it's a small wonder it works: MIRA has no physics engine, no rendering engine, and no explicit 3D representation at all. It's just videos and actions crammed into a transformer that learns everything purely from data.

Autopilot#

We train the model using action dropout, meaning that during training we hide some of the keys that the players were pressing and just tell the model to predict what would happen without this information. That allows us to have an option to use autopilot mode in the demo. Since the data is gameplay footage of a strong bot (Nexto), the model predicts what the bot would have done. This makes four-player autopilot showdowns quite a sight to behold:

Stability#

One of the most remarkable things about the model is how stable it is: as far as we can tell, it can run infinitely without diverging. A longstanding challenge in causal video models like this is staying stable over longer rollouts; most models lean on dedicated tricks and multi-stage training to avoid diverging, whereas we get stability with a fairly simple recipe (explained in the Architecture section below). This is likely helped by our relatively narrow visual domain so the approach might not generalize, but it's a strong outcome nonetheless.

Why simulate video games?#

A natural question to ask is: what is the point of simulating video games in the first place? Why play MIRA when you can just play Rocket League?

To which we say: You should play Rocket League!

Our work is not meant to replace video games with AI models. The reason researchers are excited about world models is rather physical AI: think robotics and self-driving. In those domains, it's expensive to collect data and testing the models is dangerous, since you don't want to crash a car or break a robot. If we train a world model that predicts with high accuracy how a car would behave in the real world driven by a model we're testing, we can evaluate it by running it in the world model, which is safer and quicker. If the world model is accurate enough, you could even directly train in the world model. ^·

Simulating video games is therefore a stepping stone to physical AI: it's a setting where data is abundant and it's cleaner than real-world videos since it's generated by a game engine. But learning the mechanics of an AAA video game purely from observing video is still no easy feat, so the hope is that what we learn by studying video game world models will be useful in real-world settings later.

A more direct reason for why video games can be helpful for physical AI is what's known in the business as sim-to-real transfer. The argument goes that video games are already somewhat similar to the real world: Rocket League is basically football with cars,^· so maybe a model that understands it would also have an easier time understanding cars and/or football. More specifically, you could pre-train a model on a large volume of video game data and then fine-tune with a much smaller corpus of real-world data, and the model would work significantly better than if trained purely on the real-world data.^· We don't yet know how far this sim-to-real transfer can take you when scaled up.

Data#

We train on roughly 10,000 match-hours of recorded 2v2 Rocket League matches, generated entirely by self-play between game-playing bots; no human gameplay at all. The model was not trained on any Rocket League player gameplay or data. Each match yields four synchronized first-person recordings, one per player, and every recording pairs the gameplay video with the player's action stream, all aligned to a shared timeline. We also log the underlying physics state, but use it only for evaluation, never for training: the model only ever sees pixels and actions.

We restrict the game to three arenas chosen for visual and layout diversity. Every car in every match is driven by Nexto, the highest-skilled publicly available Rocket League bot.^· Fifteen times per second, it emits three-valued axis commands for steer, throttle, yaw, pitch, and roll, together with binary jump, boost, and handbrake buttons. Using a single bot to drive all four cars removes any dependence on human players, at the cost of behavioural diversity.

Alongside video and actions, we log the game's privileged physics state via BakkesMod, the community modding framework that exposes Rocket League's internal state, such as the exact position and orientation of the ball and the cars. This signal is not consumed by the world model, which trains on pixels and actions alone; instead it serves as ground truth for evaluation.

For training the model, we re-encode the per-player video to a standardized 288p / 20 fps and tile it into fixed-length 4-second chunks, which are what the model consumes.

We publicly release Rocket Science, a 4,000-hour slice of this data (meaning 1,000 hours of matches, times four players) at 720p resolution paired with the action streams and physics states: everything you need to train your own model. Get started on our GitHub repo.

Architecture#

MIRA is trained with diffusion forcing in the latent space of a video representation codec.^· The codec compresses each frame into a compact latent, and the world model learns to predict future latents from past latents and player actions.

At training time, we denoise multiple frames of the video at once, allowing each frame to look at the ones that came before it. We do this in the latent space of the codec - see below.

At inference time, we unroll autoregressively, feeding in the generated latents back into the transformer to generate more frames:

Importantly, during training the model sees the noised versions of the past frames, so it learns to deal with uncertainty. This allows us to stabilize the rollouts by intentionally adding noise to the past frames: the model knows not to trust the past too much.

We're essentially doing causal video generation, conditioned on actions rather than text. One of the most challenging things about video generation is preventing drift: models are trained on next-frame prediction but at inference time, we feed them back frames that they themselves generated.^·

There is an entire cottage industry of papers tackling this drift problem^· for causal text-to-video, some of which are also used in recent world modeling papers^·.

We manage to train a video representation codec that is so stable it allows us to get away without using any explicit anti-drift techniques. We simply train using diffusion forcing and the model works "out of the box". Part of why this works is certainly that the Rocket League data is easier to model than messy real-world videos, but even here, stability is not a given.

Video representation codec#

Diffusion models originally generated images directly in pixel space, which spends most of the model's capacity on high-frequency texture rather than content. We tried JiT, a modern pixel-space approach, but it didn't really work well on our task:

Latent diffusion changed that: train an autoencoder (here we call those codecs in analogy to neural audio codecs, see 1 2 3) to compress the data first, run the diffusion process in the resulting latent space, and decode at the end. This is now the standard way to do things,^· and indeed on our task it does work a bit better:

However, just because a codec manages to compress and reconstruct the image doesn't mean the latent space is favorable for generation. One optimization is semantic distillation: aligning the codec's latent space with a self-supervised image representation model such as DINO - this hurts reconstruction slightly, but significantly improves generation.

And even more recently, researchers have taken this idea to its logical conclusion: instead of just aligning, completely replace the encoder by an image representation model like DINO. You need to train a decoder to reconstruct images from DINO latents, which was previously thought difficult, since a priori there is no reason these latents should be invertible back to pixel space. It turns out that with some tricks (see technical report), it's doable and works well for generation:

We started with a codec with semantic distillation as our baseline. Making video representation codecs work ended up being what gave us the highest leap in modeling performance. It's also when the model became infinitely stable. The semantic distillation codecs would diverge faster and never recover.

Evaluation metrics#

Evaluating a world model is tricky: since the rollouts don't match any ground-truth data, we cannot compare frame-by-frame like we can with the codec. Instead, we want to compare distributions: does the distribution of the videos generated by the model look like the training distribution? For this, we use Fréchet Inception Distance, the standard metric in image generation literature. To apply it to video, we simply consider all frames separately. ^·

We found FID to be a robust metric. It clearly distinguishes the three codecs discussed above, both as a function of the training step and the inference rollout time:

FID only tells us whether rollouts look like real play, not faithfulness to the game or action-following. We measure those using two probes.

Game-state probing reads the true physical state (positions and velocities of the cars and ball) out of the model's activations, checking that the dynamics are right and not just realistic-looking.

Action following checks controllability: we train an inverse-dynamics model to detect which controls are pressed from the video, and compare how recoverable each command is from the generation versus a plain codec reconstruction. Controllability keeps improving well after image quality plateaus, and rarer actions like air-rolls are harder to control than common ones like forward.

Limitations#

The model also breaks in entertaining ways.

Replays#

After a goal, Rocket League cuts to a replay of the shot. But the model only has a roughly four-second context window, so by the time the replay plays it has no memory of what actually happened: it simply makes something up. The replay it generates looks plausible but doesn't match the goal that was really scored.

The multiplayer model sometimes just diverges outright when it's supposed to show a replay. Interestingly, we don't observe that with our single-player models: they generally do manage to show a goal.

Hidden information#

The multiplayer model works well in part because it sees almost everything: with all four views and all four action streams tiled together, very little is hidden. A single-player model has a harder job: it only sees one view, so the positions of off-screen players are hidden state it has to infer. It tends to forget where those players are or model them inconsistently. Cars that get occluded by the ball often disappear.

Out-of-distribution and resets#

The model's stability depends heavily on how in-distribution it is. If you play reasonably, as in, you try to get the ball into the opposing net, the model works forever.

Maybe you don't want to kick the ball around and score goals: maybe you want to hang out with the four cars at the opposite side of the field from the ball. This will make the model unhappy.

After we switched to using the video representation codec, we noticed a curious property: in the cases where the model does diverge, it will eventually snap back almost instantly. Our earlier distillation-based codecs, in contrast, would diverge and consistently remain diverged.

Open source#

We're releasing everything you need to build on MIRA:

• Rocket Science Dataset: a 1,000-hour slice of our training data at 720p, with the paired action streams and physics states.
• Code: our training and inference code, so you can reproduce the model and the video representation codec.
• Technical report: the full details behind the model, codec, and evaluation.

Authors#

General Intuition: Anthony Hu*, Chris Mulder*, Aditya Makkar†, Adam Jelley, Eloi Alonso, Florian Laurent, Fredrik Norén, James Swingos, Jan Hünermann, Kent Rollins, Lucas Hosseini, Matthieu Le Cauchois, Maxim Peter, Pim de Witte, Tim Brown, Vincent Micheli

Kyutai: Václav Volhejn*, Adrien Ramanana Rahary*^·, Amélie Royer†, Manu Orsini†, Alyx Liao†, Moritz Böhle, Gabriel de Marmiesse, Patrick Pérez

Epic Games: Viktoriia Sharmanska, Lucia Specia, Michael Black

*equal contributions      †core contributors