The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

We present The Matrix, a foundational realistic world simulator capable of generating infinitely long 720p high-fidelity real-scene video streams with real-time, responsive control in both first-and third-person perspectives. Trained on limited data from video games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains-deserts, grasslands, water bodies, and urban landscapes-in continuous, uncut hour-long sequences. With speeds of up to 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting-an environment present in neither gaming data nor real-world sources. This approach showcases the potential of game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data. See https://github.com/MatrixTeam-AI/matrix, https://matrixteam-ai.github.io/pages/TheMatrix/ for code data and project page.

Figure 1: The Matrix is a foundational realistic world simulator capable of generating infinitely long 720p high-fidelity real-scene video streams with real-time, precise moving control. Due to size limitation, we recommend you to go to the website provided in the abstract for videos.