Omniverse Machinima gets new Audio2Gesture system
Nvidia has released Omniverse Machinima 2022.2, the latest beta release of its AI-driven Omniverse app for creating character animation for game cinematics.
The update adds Audio2Gesture, a new system for generating body animation automatically from dialogue.
A work-in-progress Omniverse app for creating game cinematics
First released last year, Omniverse is designed to enable users to collaborate in real time to animate characters, using a set of AI-driven tools.
Despite its name, Nvidia is pitching it more for the creation of game cinematics than actual machinima, describing it as for “technical artists, content creators, and industry professionals”.
Currently – it’s still in beta – it’s more a kit of parts than a unified application, with the online documentation advising users that they need to “connect the dots” between the extensions themselves.
Generate character animation using a set of AI-trained tools
Workflow begins with readymade 3D characters and props: Nvidia provides a library of assets from commercial games including Mount & Blade II: Bannerlord and Squad, or users can import their own.
Although Machinima has conventional character animation tools – the set of Omniverse extensions provided include a timeline and curve editor – they’re intended mainly for editing animation.
To generate animation, users get a number of AI-trained tools, including Pose Tracker, which generates full-body character motion from source video of a human actor.
Users can then set up lights and cameras, render clips using Omniverse’s RTX Renderer, and edit those clips into a cinematic using the Sequencer extension.
Omniverse Machinima 2022.2: Audio2Gesture converts speech into body animations
To that, Omniverse Machinima 2022.2 adds Audio2Gesture, a new AI-trained extension for generating animation for “human-like bodies” automatically from audio sources.
In this case “audio sources” really means “dialogue” – it isn’t intended to get a character dancing to music.
There are separate solvers for processing streaming audio and for running Audio2Gesture in an offline pipeline with recorded speech, and users can choose to generate either full-body or upper-body motion.
(Audio2Gesture doesn’t generate facial animation from audio, although it is possible to do that outside of Omniverse Machinima, through a separate application, Omniverse Audio2Face.)
The result can be customised by selecting a range of readymade movement styles, including Calm speech, Big gestures and Public speech, the latter including a variant for characters speaking from behind a table.
You can judge the results from the video above: with some styles, you could argue about whether the character is enacting the dialogue or just moving its hands about in time to it, but it’s a pretty good start.
Pricing and availability
Omniverse Machinima 2022.2 is available in beta for Windows 10 only, and requires a compatible Nvidia GPU: at least a GeForce RTX 3070 or RTX A4000. It installs via the Omniverse launcher.
Ominverse is free for individual artists. You can find details of Enterprise pricing for teams in this story.
Read a full list of new features in Omniverse Machinima 2022.2 in the online release notes