Nvidia researchers have made a generative model that can create virtual environments using real-world videos from sources like YouTube.<\/b>
\nNvidia researchers have made a generative model that can create virtual environments using real-world videos from sources like YouTube \u2014 a way of generating graphics that could have implications for the future of gaming, AI, and graphics.
\u201cIt\u2019s a new kind of rendering technology, where the input is basically just a sketch, a high-level representation of objects and how they are interacting in a virtual environment. Then the model actually takes care of the details, elaborating the textures, and the lighting, and, and so forth, in order to make a fully rendered image,\u201d Nvidia VP of applied deep learning Bryan Catanzaro told VentureBeat in a phone interview.
The system was trained using Apolloscape video from Baidu\u2019s autonomous driving project. Sketches of where to find things \u2014 like trees, buildings, cars, and pedestrians \u2014 are input to the model.
The method is detailed in a paper titled \u201cVideo-to-Video Synthesis\u201d whose authors include Catanzaro, other Nvidia researchers, and a MIT CSAIL student.
The model could make it possible to quickly generate synthetic environments for training reinforcement learning agents or help people who are playing games feel like they\u2019re in the same place.
\u201cImagine if you could play as yourself in a video game. And, you know, just from a cell phone video that you took, that was enough information to add your character as yourself, you know, as the knight in shining armor that\u2019s going off to, you know, do some great deed,\u201d he said. \u201cNvidia\u2019s been working on graphics for a long time, so we\u2019re really excited to bring graphics into the AI era and use AI to generate graphics, where we learn how things work from videos of the real world, and then can synthesize content based on that knowledge.\u201d
The system is able to take into account things like the location of objects, relationships between objects, and maps to define the edges of objects.
Nvidia will demo videos and share additional details of the technique at the Neural Information Processing Systems conference (NeurIPS, formerly NIPS) being held this week in Montreal.
The team at Nvidia was initially inspired to take this approach by the work of Alexei Efros and other researchers at University of California, Berkeley and their creation of the Pix2Pix system. In response, working in tandem with AI practitioners from UC Berkeley, Nvidia created Pix2PixHD.
Earlier this year, UC Berkeley researchers also produced models that were able to dance and do flips and 20 other acrobatic moves .
\u201cI think this is the first interactive AI rendering, and we\u2019re really proud of the progress we\u2019ve made. But it is early stage, and I think there\u2019s going to be a lot of progress to make the output higher quality and more general so that we can handle more kinds of scenes. And so I\u2019m really excited about where this is going to go in the future,\u201d Catanzaro said.<\/p>\n