When you walk around in a version of the video game Minecraft from the AI companies Decart and Etched, it feels a little off. Sure, you can move forward, cut down a tree, and lay down a dirt block, just like in the real thing. If you turn around, though, the dirt block you just placed may have morphed into a totally new environment. That doesn’t happen in Minecraft. But this new version is entirely AI-generated, so it’s prone to hallucinations. Not a single line of code was written.
For Decart and Etched, this demo is a proof of concept. They imagine that the technology could be used for real-time generation of videos or video games more generally. “Your screen can turn into a portal—into some imaginary world that doesn’t need to be coded, that can be changed on the fly. And that’s really what we’re trying to target here,” says Dean Leitersdorf, cofounder and CEO of Decart, which came out of stealth this week.
Their version of Minecraft is generated in real time, in a technique known as next-frame prediction. They did this by training their model, Oasis, on millions of hours of Minecraft gameplay and recordings of the corresponding actions a user would take in the game. The AI is able to sort out the physics, environments, and controls of Minecraft from this data alone.
The companies acknowledge that their version of Minecraft is a little wonky. The resolution is quite low, you can only play for minutes at a time, and it’s prone to hallucinations like the one described above. But they believe that with innovations in chip design and further improvements, there’s no reason they can’t develop a high-fidelity version of Minecraft, or really any game.
“What if you could say ‘Hey, add a flying unicorn here’? Literally, talk to the model. Or ‘Turn everything here into medieval ages,’ and then, boom, it’s all medieval ages. Or ‘Turn this into Star Wars,’ and it’s all Star Wars,” says Leitersdorf.
A major limitation right now is hardware. They relied on Nvidia cards for their current demo, but in the future, they plan to use Sohu, a new card that Etched has in development, which the firm claims will improve performance by a factor of 10. This gain would significantly cut down on the cost and energy needed to produce real-time interactive video. It would allow Decart and Etched to make a better version of their current demo, allowing the game to run longer, with fewer hallucinations, and at higher resolution. They say the new chip would also make it possible for more players to use the model at once.
“Custom chips for AI hold the potential to unlock significant performance gains and energy efficiency gains,” says Siddharth Garg, a professor of electrical and computer engineering at NYU Tandon, who is not associated with Etched or Decart.
Etched says that its gains come from designing their cards specifically for AI development. For example, the chip uses a single core, which it says makes it possible to handle complicated mathematical operations with more efficiency. The chip also focuses on inference (where an AI makes predictions) over training (where an AI learns from data).
“We are building something much more specialized than all of the chips out on the market today,” says Robert Wachen, cofounder and COO of Etched. They plan to run projects on the new card next year. Until the chip is deployed or its capabilities are verified, Etched’s claims are yet to be substantiated. And given the extent of AI specialization already in the top GPUs on the market, Garg is “very skeptical about a 10x improvement just from smarter or more specialized design.”
But the two companies have big ambitions. If the efficiency gains are close to what Etched claims, they believe, they will be able to generate real-time virtual doctors or tutors. “All of that is coming down the pipe, and it comes from having a better architecture and better hardware to power it. So that’s what we’re really trying to get people to realize with the proof of concept here,” says Wachen.
For the time being, you can try out the demo of their version of Minecraft here.