Recent advancements in graphics processing technology have permitted software and hardware vendors to collaborate on real-time ray tracing, a long-standing “holy grail” of computer graphics. Ray-tracing has been used for a couple of decades now, but has always been used in pre-rendered graphics – often in movies or other video playback that doesn’t require on-the-fly processing. The difference with going real-time is that we’re dealing with sparse data, and making fewer rays look good (better than standard rasterization, especially) is difficult.
NVidia has been beating this drum for a few years now. We covered nVidia’s ray-tracing keynote at ECGC a few years ago, when the company’s Tony Tamasi projected 2015 as the year for real-time ray-tracing. That obviously didn’t fully realize, but the company wasn’t too far off. Volta ended up providing some additional leverage to make 60FPS, real-time ray-tracing a reality. Even still, we’re not quite there with consumer hardware. Epic Games and nVidia have been demonstrating real-time ray-tracing rendering with four Titan V100 GPUs lately, functionally $12,000 worth of Titan Vs, and that’s to achieve a playable real-time framerate with the ubiquitous “Star Wars” demo.
As an industry, we’re getting there. Another generation and real-time ray-tracing should become a more feasible capability on single-GPU solutions.
Interestingly, learning about ray-tracing has somewhat shifted our expectations for next-generation consumer graphics. We already know that the next-generation, learned from our Titan V testing, will focus on asynchronous compute pipelines and improving performance drastically on low-abstraction APIs (Vulkan, Dx12). Until recently, we didn’t know that the Tensor cores could also possess usefulness outside of deep learning. Tensor cores are used in accelerating real-time ray-tracing in the recent demonstrations, which caught us off-guard: by definition, a Tensor core is deployable for dense data, as found in machine learning routines. Ray-tracing in real-time is inherently dealing with sparse data, particularly in these early days of limiting the amount of traced rays to a handful, leveraging denoising for cleaning-up the sparsely traced rays. We’re not sure if future gaming GPUs will host Tensor cores, particularly because the current consumer use case is a ray-tracing demo that hasn’t propagated to “real” games yet, but put us down for “maybe.” Maybe not next generation, but in some nebulously timed future.
Regardless, we joined nVidia’s Tom Petersen to talk about real-time ray-tracing difficulties recently, and wanted to use the opportunity to break-down how ray tracing works.
Ray tracing is more of a physical version of drawing than traditional game graphics approaches, which are more comparable to art through rasterization. Ray tracing is a calculation of the physics of light, originating from the camera’s viewpoint and tracing into the camera frustum. As far as we’re concerned, as players, that’d mean that the rays being traced originate in our ‘eyes,’ then project outward and onto the scene. These rays trace bounces off of objects, working in reverse from the camera, to the object, to the source of light. Once we’ve found the source of light, the GPU can compute the physically accurate rendition of how that ray bounces off of surfaces and reaches our camera.
Modern rendering relies upon heuristics, textures, albedo, materials, and (sometimes) physically-based rendering, which is describable as rendering as based upon physically accurate properties of energy conservation by materials impacted by light. This version of rendering uses a lot of cheats to best approximate realistic graphics, resulting in occasional rendering artifacts that might break immersion to the viewer. Some of these are fairly obvious: An old side-effect of screenspace reflections, for instance, is the inability of mirrored surfaces to reflect objects unseen to the camera. If our view frustum contains a straight line of objects – camera, NPC (facing us), mirror – and we are rendering screenspace reflections, the mirror will not reflect the back of the NPC’s head. Instead, as a dynamic object, the NPC would either be omitted or incorrectly rendered. Developers often opt for the former, as the incorrect rendering is jarring – you end up with the front of the NPC rendered on the mirror, despite the fact that the NPC’s back, not front, is facing the mirror. This can be solved for by using expensive (resource-intensive) planar reflections, but ray-tracing also begins the stages of addressing realistic and believable reflections and refractions.
Another example of traditional rendering artifacts could include shadows, which can be well-approximated (particularly with voxel approximation), but are never quite perfect. Ray-tracing helps with shadow drawing and softness, and can be used to more correctly soften or harden shadows pursuant to the light source’s diffuse index.
All of this is hard to do. As stated earlier, ray tracing isn’t new, but going real-time is (at a playable framerate, anyway).
In nVidia’s present demonstrations of RTX, Futuremark’s ray tracing demos for 3DMark, and Microsoft’s DirectX Ray Tracing extensions, limited samples and traced rays are being taken for the renders. In the real world, we have functionally infinite rays being bounced all over. Light touches nearly everything. It’s impossible to digitally simulate this level of light in any reasonable amount of time, although we can try to come as close as is reasonable to suspend disbelief. Our video interview above contains many in-house-made (and externally made) examples of ray tracing, including examples of an image with hundreds of traces versus single-digit rays traced. GN’s own intro animation – that two-second pre-roll – took about two weeks to render, and uses ray tracing to correctly simulate light across the metallic surface of the spinning GN logo.
For real-time simulations, only a couple of samples are being traced, and denoisers are sometimes applied to assist in smoothing-over sparse data input (a limited count of samples) into something viewable – otherwise, you end up with dot art.
This process of real-time ray tracing generates a sparse image, then uses deep-learned denoise filters to smooth-over pops of simulated pixels adjacent to pixels left unsimulated.
We still have a long way to go before real-time ray tracing is realistically achievable on consumer hardware, but it’d appear that nVidia’s next-generation GPUs will continually improve on this very specific front. Expect ray tracing in the near future of game graphics, hopefully alongside greater popularity of properly implemented low-abstraction APIs.
View the video above for some examples of ray tracing by our team and for Tom Petersen’s explanations.
Editorial: Steve Burke
Video: Andrew Coleman