GauGAN360 turns doodles into 360-degree environments
Nvidia has launched GauGAN360, an experimental new online art tool that turns rough sketches into 360-degree environments that can be used in 3D scenes.
The tool, based on the same tech as Nvidia’s original GauGAN AI painting app, lets users paint in the overall form of a landscape, and have GauGAN 360 generate a matching cube map or equirectangular image.
Turn quick doodles into synthetic 360-degree environments for use in 3D scenes
Nvidia describes GauGAN360 as the “next evolution” of GauGAN, its popular AI painting app.
Launched in 2019, GauGAN uses an AI model based on Generative Adversarial Networks to generate photorealistic images matching users’ rough sketches.
GauGAN does pretty much the same thing, but instead of a conventional 2D image, the result is a cubemap or equirectangular image that can be used as an environment map inside 3D software.
Nvidia’s promo video shows its own Omniverse Create software, but the maps could be used in other 3D apps like Blender, or in game engines like Unity and Unreal Engine.
Different brush colours let you paint plants, ground, landscape features and skies
Workflow in GauGAN360 is similar to GauGAN: you paint the rough form of the image you want to generate onto the online template, click the button, and wait for the AI to do the rest.
Different brush colours represent different landscape elements: there are 25 available, divided up into ground elements like mud and sand; land and sky features like rivers, mountains and clouds; and plants.
The controls are pretty basic – a choice of three brush shapes, and a brush size slider – but that’s kind of the point: it’s designed to generate images from very quick doodles.
You can also upload a real image to use as a source, with GauGAN350 generating a synthetic image with the same overall composition as the original.
Limited functionality in the current online demo
In our quick tests, we found that the best results came from blocking out broad areas of a landscape: painting finer strokes to suggest smaller features resulted in some oddly distorted-looking trees, for example.
The current beta also only generates natural landscapes: while there were brush colours for roads and pavements, there were no options to generate buildings or other man-made features.
The output was an 8-bit PNG file, so while it could be used as a background environment, it wouldn’t be suitable for environment lighting and, at 1,024 x 768px, it was fairly low-resolution.
More advanced features to come?
However, Nvidia’s teaser video (embedded at the top of the story) suggests that there may be more features to come: the footage shows options to generate HDRIs as well as low dynamic range panoramas.
The interface shown in the video also includes a Building category not available in the current online demo, with brush colours corresponding to man-made features like houses, bridges and walls.
In addition, the description of the video on YouTube says that GauGAN360 generates 8K panoramas: a much higher resolution than the images we were able to download.
We’ve contacted Nvidia to ask when these options will be available, and will update if we hear back.
However, even in its current form: GauGAN360 is a lot of fun to play around with: as with GauGAN itself, there’s something very satisfying about seeing your rough scribbles turned into fantasy environments.
The technology behind GauGAN also underpins Canvas, Nvidia’s free downloadable AI painting app, so if that gets updated in the same way, the same features could also become available in offline tools.
Updated 11 August 2022: Nvidia told us that the GauGAN360 AI model itself supports 8K and HDRI output, but that the online beta uses a smaller version of the full research.
It declined to comment on if or when there would the updates to the online version.
Updated 17 March 2023: 360-degree output is now available in Canvas 1.4.
System requirements and licensing
GauGAN360 runs in a web browser. To generate an image, you have to accept Nvidia’s terms of use, which gives Nvidia the right to use or distribute any content you upload or create.