Our test platform includes 2x Titan V cards, the EVGA X299 Dark motherboard, an Intel i9-7980XE, and 32GB 3866MHz GSkill Trident Z Black. We were using the Titan Vs under air for these tests. When overclocked, they were set to a stable OC that was achievable on both cards -- +150 core and HBM2.
Ashes is being run at 4K with completely maxed settings. We set them to “Crazy,” then manually increment all options to the highest point, including 8xMSAA. This was required to ensure adequate GPU work, thereby reducing the potential for a CPU bottleneck.
PCIe 3.0 was introduced in 2010, brought with it an encoding scheme upgrade that reduced bandwidth overhead from 20% to 2%, and significantly improved transfer rates. We’ve had trouble saturating theoretical maximum PCIe bandwidth recently, with even single 1080 Ti cards not showing much more than a 1% difference – largely within error margins. This test should find the limits.
Ashes of the Singularity: Single vs. “SLI” Titan Vs
“SLI” in scare-quotes, here.
Starting first with our dual versus single Titan V benchmark, we overclocked the cards equivalently in each setup, then tested with 16 PCIe lanes available. The cards tested at 63.3FPS AVG for the dual configuration, with lows at 25 and 22FPS for 1% and 0.1% low frametimes. The single Titan V configuration operated at 37.6FPS AVG, immediately establishing that we are seeing large, noteworthy games. Unfortunately, Ashes is where they stop, as we were unable to get other games to detect both Titan Vs.
The stock cards compare at 33FPS AVG versus 57FPS AVG, which is a noteworthy gain of 72%.
Either way, we’re already seeing 68.4% gains from adding a second card, which is significant; no, the Titan V is not a “gaming card,” or even a gaming architecture, but it is scaling in games.
PCIe 3.0 x8/x8 vs. x16/x16 on Titan Vs
Moving on to the PCIe lane bandwidth testing, our charts now focus on x16 and x8 configurations, both overclocked to push the limits of the PCIe clock.
We’re measuring 63FPS AVG for two cards with an x16 interface, and measuring 56FPS AVG for two cards with an x8 interface. That’s about 12-13% of performance improvement with PCIe Gen3 x16, and is significant in illustrating that we’re nearing the end of Gen 3’s bandwidth, which is 15.75GB/s for 16 lanes. PCIe Gen 4 will push about 2GB/s per lane, and will resolve any potential issues, but nVidia is also using its $600 NVLink bridge to solve for this.
As for single cards, we did not notice any appreciable difference between single Titan V configurations in x8 versus single configurations in x16. It appears that we start encountering the most issues when transacting across two cards.
The question becomes whether or not you’d notice this in other applications. Not every application cares about PCIe bandwidth; mining is a great, modern example, where GPU miners use x1 slots for all their work. It’s possible that applications more tailored for the Titan V would not run into this issue, as the two cards may not need to transact between each other as much. If you use these types of devices for their intended, scientific purposes, please let us know and we can do more bandwidth testing – let us know what applications you use, too, as we aren’t experts in that area.
Conclusion: Finally Finding the Limits, but --
This isn’t representative of the whole. We’ve tested one game, here, and that’s about the limit of what is even compatible with 2x Titan Vs. Production software, like Blender or Premiere, won’t stress the PCIe interface in the same way – the cards don’t need to talk to each other, in these scenarios, and can operate independently on a tile-by-tile or frame-by-frame basis. Gaming puts more load on the PCIe bus as the cards transact more data to create each frame. These devices, as stated before, aren’t meant for gaming, so that’s largely a non-issue. They also aren’t really compatible in 2-way configurations, so that further eliminates the realism of this test.
What we are left with, however, is a somewhat strong case for waning PCIe bandwidth sufficiency as we move toward the next generation – likely named something other than Volta, but built atop it. SLI or HB SLI bridges may still be required on future nVidia designs, as it’s possible that a 1080 Ti successor could encounter this same issue, and would need an additional bridge to transact without limitations.
Editorial, Testing: Steve Burke
Video: Andrew Coleman