AI Learns To Re-Create Super Mario Bros. By Watching Someone Else Play It

Usually, when we get AI systems to watch video games, we expect them to play the games afterward. That’s how computers have beaten everything from the board game Go to various Atari titles.

But a group of researchers from the Georgia Institute of Technology is trying something different: they’re getting AI to learn how video games work instead.

In a paper called “Game Engine Learning from Video,” the team from Georgia Institute of Technology explains how the (remarkably accurate, considering) result is based on learning not just how to mimic the game, but also how to mimic the engine powering it – the one designed by Nintendo.

When you realize that, you can ignore the glitching and lack of details, because although it’s not perfect, by golly it’s close.

For a start, the AI system isn’t learning everything about the game from scratch. It’s supplied with two important sets of information: first, a visual dictionary featuring all the sprites in the game; and secondly, a set of basic concepts, like the position of objects and their velocity, which it uses to analyze what it sees. With these tools in hand, the AI breaks down the gameplay frame-by-frame, labels what it sees, and looks for rules that explain the action.

What makes this so amazing is the fact that – to reemphasize – the neural network has no access to the code – it has ‘programmed’ a platform game from sight and basic logic. Well, okay, most of a game.

The system is limited to working on 2D platformers. That’s because it relies on humans to define what can happen in any particular game. It can’t-do is work in 3D. To do that would require a lot more information to be provided to the system that was offered at this stage.

Gaming is becoming an increasingly important part of teaching neural networks on how to behave.

In the future, though, the team from Georgia Tech thinks technology like this could be used to work out not only how video games work — but real life, too. This would take a number of breakthroughs in the capacity of AI to comprehend the world as humans understand it.