Metro: Exodus RTX Benchmarks, In-Game vs. Built-In Test, Dx11 vs Dx12

By Published February 14, 2019 at 9:08 pm
  •  

Metro: Exodus is the next title to include NVIDIA RTX technology, leveraging Microsoft’s DXR. We already looked at the RTX implementation from a qualitative standpoint (in video), talking about the pros and cons of global illumination via RTX, and now we’re back to benchmark the performance from a quantitative standpoint.

The Metro series has long been used as a benchmarking standard. As always, with a built-in benchmark, one of the most important things to look at is the accuracy of that benchmark as it pertains to the “real” game. Being inconsistent with in-game performance doesn’t necessarily invalidate a benchmark’s usefulness, though, it’s just that the light in which that benchmark is viewed must be kept in mind. Without accuracy to in-game performance, the benchmark tools mostly become synthetic benchmarks: They’re good for relative performance measurements between cards, but not necessarily absolute performance. That’s completely fine, too, as that’s mostly what we look for in reviews. The only (really) important thing is that performance scaling is consistent between cards in both pre-built benchmarks and in-game benchmarks.

Bench Components

 

Component

Courtesy of

CPU

Intel i7-8086K 5.0GHz

GamersNexus

GPU

This is what we’re testing!

Often the company that makes the card, but sometimes us (see article)

Motherboard

ASUS ROG Maximus X Hero

ASUS

RAM

Corsair Vengeance LPX 32GB 3200MHz

Corsair

PSU

Corsair AX1600i

Corsair

Cooler

NZXT Kraken X62

NZXT

SSD

Plextor 256-M7VC
Crucial MX300 1TB

GamersNexus

Test settings for the game are as follows:

Tessellation on/full
Gameworks (HairWorks) disabled
Motion blur - Low
Preset as designated on chart
RTX as designated on chart
DLSS off
Shading rate 1.0/100%
Vsync off
Physx off
AF 16x
Resolution as defined on chart

Built-in Benchmark vs. In-Game Benchmark

One of the most important things to look at with a major game launch is whether its built-in benchmark is actually reliable. With Final Fantasy, we discovered that the game was improperly culling some objects to the extent that they affected performance from miles away. With other games, like Civilization VI, we have found that the turn time benchmarks are accurate to actual gameplay.

1 metro exodus bench vs game scaling all

For Metro: Exodus, we’ll start by checking actual in-game performance versus the benchmark scene. We’re starting with only the 2080 Ti at 4K resolution, scaling across low to extreme settings and with varying RTX configurations. We found that the built-in benchmark operated at only 84FPS AVG with the 2080 Ti at 4K and low settings, with RTX off, whereas our in-game testing posted 152FPS AVG performance results. As always, this is tested multiple times and averaged for parity. At medium settings and with RTX off, we found performance decay to 72FPS AVG for the built-in test and 118FPS AVG for actual gameplay. This further drops to 60FPS AVG for High settings with the built-in benchmark and 94FPS AVG for the in-game test. For the next data point, we have to highlight something important: With Ultra settings and RTX off, we’re still running at 87FPS AVG, which is technically better than the built-in benchmark’s 84FPS AVG starting point at Low settings. If ever there were any question as to whether the built-in benchmark is an accurate depiction of gameplay, this answers it by three entire drops in graphics quality.

For Extreme settings, we hit 70FPS AVG without any RTX enabled and 40FPS with all graphics settings maxed. As we descend the chart, we eventually see that the 2080 Ti hits 53FPS AVG with RTX on High and graphics set to Ultra with in-game performance, versus 36FPS for the built-in test. We then hit 48FPS AVG with RTX Ultra, a drop of about 9% from High.

That sets the scale for us. Now, as for what this means, it’s pretty straight-forward: As long as the card-to-card scaling remains equally spaced in the built-in test as it is in the game, e.g. an RTX 2080 Ti retains X% lead over Radeon VII in both tests, where X doesn’t change, then it is acceptable as a synthetic gauge of performance. It is not useful as an absolute gauge of performance, and anyone wanting to play Metro: Exodus should look at in-game benchmarks instead. For comparing two cards relative to each other, the built-in test may still be useful, but it just comes down to having equidistant relative scaling between devices. We’re going to stick with in-game testing for our work, though, as it is more universally accurate.

Let’s look at more detailed performance numbers.

DirectX 12 vs. DirectX 11 Performance Scaling

Except for this one, all tests in our benchmark today are with DirectX 12 enabled, as it’s required for DXR. We still wanted to test DirectX 11, though, as sometimes game developers will lazily wrap the more abstracted API or fail to really leverage Dx12.

2 metro exodus bench fps dx11 vs dx12

For this one, we took the RTX 2080 XC Ultra and tested it at 1440p and Ultra settings, with RTX again disabled. We still aren’t heavily loading the CPU here, so this is very much a GPU-bound benchmark. With DirectX 11, we measured performance at 114FPS AVG, 91FPS 1% low, and 89FPS 0.1% lows. The DirectX 12 in-game test put us at 112FPS AVG, 84FPS 1% lows, and 79FPS 0.1% lows. That’s more or less tied, but we need to look at frametimes to better determine if we’re operating outside of variance.

3 metro exodus bench dx11 vs dx12 all

As always, frametime plots show us frame-to-frame present intervals to get the best picture for the per-second experience of gameplay. Lower is better in frametime, but consistency is more important than just being lower. For this one, we see that DirectX 12 testing plots us as starting at around 8ms, then climbing to around 10ms and eventually hitting 12ms frametimes. The more important aspect is that the spikiness of the frametimes measured swings more than DirectX 11 does. These are both completely playable and the frametime variance never exceeds 8ms, which means that the average user really won’t know the difference, but Dx11 does have a tighter consistency and is objectively better. It’s just that, when it comes down to it, the objective superiority doesn’t really matter. It’s measurable, not perceivable. 4A seems to have roughly equal performance in Dx12 and Dx11, maybe slightly favoring Dx11.

DirectX 12 Performance Scaling

Now that we’ve whittled testing down to just focusing on DirectX 12, we can look at graphics setting scaling more granularly.

4 metro exodus bench rtx on vs off 2080ti

For in-game testing, performance scaling from the 2080 Ti at 4K shows reasonably strong results at 4K Ultra with RTX off, giving us an 87FPS AVG. This grants High settings with RTX off a gain of about 7-8% against Ultra settings, or Ultra a gain of about 25% against Extreme settings. We think that Ultra is the best middle-ground on average, considering that Extreme shows severe performance fall-off. Medium is another major step, but texture and shadow quality start to become noticeable quickly.

For Ultra settings with RTX off, our 87FPS AVG is 65% higher than Ultra settings with RTX on its lowest setting, called “high,” presumably because NVIDIA does not want to dirty its brand with an option named “low.” If you prefer to look at it the other way, it’s about a 40% performance drop from RTX off. Going to Ultra drags us down an additional 8% from RTX High, rendering even in-game framerates undesirable.

The impact of RTX appears to minimally be about a 40% FPS loss to a 45% loss at Ultra. As for what that gets you, again, we’ll show some of our footage from our previous technical analysis of the graphics quality. The end result is that some scenes, particularly those outside where light bounces off snow for a less-faked global illumination, we get additional contrast and more gradual shadow gradients. In other scenes, like the actual namesake metro of the game, there is functionally zero impact from RTX in visuals, but still a 40% to 50% hit to framerate. Learn more about the visual appearance in our previous video.

Results Skewed on First Run

A quick note before progressing further: We noticed that the first run of a benchmark would produce numbers sometimes 8-10FPS lower than subsequent benchmark runs. These mostly average-out when considering our multi-pass approach to testing, but just be aware that your first exposure to a new area may be at a lower framerate than repeat or prolonged exposures to those areas. The numbers that have been on the screen sort of demonstrate that.

The below are some preliminary test results (not necessarily final runs) that we used to determine whether the benchmark passes were consistent:

  Low, RTX Off  
146.3 106.1 99.1
152.3 106 99.3
155.3 106.2 101.5
154.5 105.8 100
  Med, RTX Off  
114.4 82.3 76.1
117.9 84.6 80.5
118.6 83.5 79.4
118.2 83.9 77.6

Metro: Exodus - RTX On: 1080p/Ultra

6 metro exodus bench rtx 1080p ultra

RTX-only testing is next. For this, because we’re specifically looking at RTX scaling card-to-card, the charts will be limited to only compatible 20-series devices. At 1080p/Ultra with RTX set to high, the 2080 Ti tops the chart at 138FPS AVG, with lows at 99FPS and 94FPS 1% and 0.1%. These are good numbers overall and reasonably consistent, so frametime pacing isn’t suffering here; that said, compared to RTX off numbers, we are still obviously taking a large performance hit. The 2080 non-Ti ends up at 114FPS AVG, allowing the 2080 Ti a lead of 21% and leading the RTX 2070’s 90FPS AVG by 26.6%. The 2070 then leads the 2060’s 80FPS AVG by 13%. This gap is much less impressive than the previous two and, considering our review of the RTX 2060, we know that an overclocked 2060 could somewhat easily catch the stock 2070 in performance; granted, you could also overclock the 2070, but it’s not hard to get baseline stock performance from an overclocked 2060.

Metro: Exodus - RTX On: 1440p/Ultra

7 metro exodus bench rtx 1440p ultra

1440p is next. For these settings, the RTX 2080 Ti FE ends up at 101 FPS AVG, posting a loss of about 13% from the previous chart. Lows are still tightly timed and within reason. The RTX 2080 runs at 77FPS AVG, granting the 2080 Ti a lead of 32%. This impressive gap should widen as resolution continues to increase. The 2070 ends up at 60FPS AVG and the 2060 isn’t far behind, demonstrating significant value when considered against the 2070. Again, a quick overclock would balance things. Keep in mind further that dropping settings to “High” would give us some performance back, although RTX is responsible for eradicating the majority of framerate.

Metro: Exodus - RTX On: 4K/Ultra

8 metro exodus bench rtx 4k ultra

The last chart is almost embarrassingly barren, but to be fair, that’s not really our fault. At some level – notably, below 30FPS – it sort of stops becoming pertinent to test lower-end video cards. At 4K/Ultra and with RTX set to High, the RTX 2080 Ti leads the RTX 2080 by about 36%. This again expands its lead as the RTX 2080 begins to fall behind from sheer pixel throughput, but both devices are struggling in the face of RTX enablement; still, it’s far more playable than the pre-built benchmark scenes. Dropping settings would make this a bit easier, but it’d make the most sense to just disable RTX altogether if in desperate need of higher FPS at 4K.

Conclusion

We will likely continue to revisit Metro: Exodus as it gets updates or as video cards launch. It looks to be a worthwhile test case, though we will stick to in-game testing for more representative benchmarks. The impact to performance from enabling RTX is substantial, as it was with Battlefield, but NVIDIA has previously worked with developers to improve performance over time. We don’t yet have a precedent for this happening with global illumination, but it is (of course) possible that we will see further improvement in an otherwise new (ish) technology.

As for whether it’s actually worth enabling, we’ll leave that to you to decide. Check our qualitative analysis for some graphics / visual comparisons.

Editorial, Testing: Steve Burke
Video: Andrew Coleman

Last modified on February 14, 2019 at 9:08 pm
Steve Burke

Steve started GamersNexus back when it was just a cool name, and now it's grown into an expansive website with an overwhelming amount of features. He recalls his first difficult decision with GN's direction: "I didn't know whether or not I wanted 'Gamers' to have a possessive apostrophe -- I mean, grammatically it should, but I didn't like it in the name. It was ugly. I also had people who were typing apostrophes into the address bar - sigh. It made sense to just leave it as 'Gamers.'"

First world problems, Steve. First world problems.

We moderate comments on a ~24~48 hour cycle. There will be some delay after submitting a comment.

Advertisement:

  VigLink badge