Look – these technologies are new, and that's precisely why they should be tested. There are two primary reasons we've used new titles to benchmark Dx12 and Vulkan performance: (1) An early indication of performance shows whether the tech works and plots the first data point in a long-haul look at performance progression; (2) validating claims made by multi-billion and multi-million dollar companies is an assurance of consumer protection against run-away hype, and the checks and balances afforded by independent media help keep companies honest.
The first point is the most critical in the case of APIs, as it's only a matter of time before Dx12 and Vulkan begin an en-masse switch-over from Dx11 and OpenGL for PC gaming.
Our approach to tested content is deeply embedded in the scientific method. It is our belief that, with these new APIs and GPUs and what-have-you, testing early and often builds a data-set which can later be analyzed for plotting trendlines and creating a pool of results which helps project into the future. That's valuable knowledge. Maybe we revisit these things in a few months and find major, 30% hikes in performance that bring parity between Dx11 and Vulkan; that'd be critical, and it'd show a very serious investment in the future by accommodating developers. Some time later – maybe it's a year from now, who knows – we retest again and find that Vulkan or Dx12 have overtaken Dx11 in most tested games. That's more big news, and helps us establish a clear, precise “switchover” point in the time-line of technology.
If we decided not to test the new APIs until everything's perfectly built, ground-up as it should be, we'd be waiting years – and there's certainly no harm in knowing more, even if it's early information. Not to mention that, from a competitive standpoint (because this is ultimately a business), no one really cares at that point. That's a large part of why people read these sites – to get information ahead of purchases or launches – and so it makes sense that we'd follow that trend with API testing.
Were we to sensationalize headlines with statements like “Vulkan 30% Slower than Dx11,” that'd be one thing – but we didn't, our headline for Vulkan (specifically) was “Initial Vulkan Benchmark vs. DirectX 11 - AMD & NVidia in Talos Principle.” Did you catch it? “Initial.” That's not there by accident, and neither is the multi-paragraph section entitled “Read This First!,” which goes on to quote the developers' own FAQ about early support being unoptimized. It's also, honestly, not that compelling of a headline to click on given just how sterile and neutral the phrasing is.
The same is true for our Dx12 benchmark, which used a fantastically sterile headline of “AMD+NVIDIA Multi-GPU DirectX12 Benchmark.”
We're approaching these topics neutrally and with great emphasis on explaining the state of affairs within the API world, with routine conclusions that follow-ups should plot an exciting, upward trend for the tech. And, if for some reason that doesn't happen, we'll be here to catch it and start asking questions of the relevant IHVs and ISVs.
That's why we test this stuff early.
Just thought we'd point that out. The majority of our readers and viewers have supported these efforts, but a select few comments on YouTube and reddit have inspired this post as a means to hopefully provide education on the process. It's very frustrating as a smaller media outlet to see outlier comments condemning hours- and days-long efforts to benchmark new technology, often for reasons shrouded in either hate or shortsightedness. That's part of the deal with posting online, of course, and so we're used to it – but this particular topic required discussion.
If you like our content, feel free to scope us out on Patreon.
Editorial: Steve “Lelldorianx” Burke
Video Editing: Andrew “ColossalCake” Coleman