Our Response to Those Who Say Not to Test Vulkan & Dx12 “Early”

By Published March 02, 2016 at 11:35 am

Here's the thing: That comment in the headline has always been shortsighted, and will always be shortsighted. We've seen it a few times lately – the majority of comments on our DirectX 12 explicit multi-GPU and Vulkan benchmarks have been positive – but a stand-out few have explicitly scolded our efforts for testing new APIs on games which are known to be incomplete. Our articles and videos contain massive sections that fully detail just how far along any new tech is, disclaiming the possibility that – like with Vulkan – it may be under-performing due to an early build state.

But that's not a reason to leave something untested, and to think as such is a mix of denial and naivety.

Here's an example comment: “None of these tests matter right now as Vulcan is not fully optimized.” [sic]

These comments are rooted in denial that's resultant of marketing build-up for the new APIs. Anything short of game-changing is seen as an indication that it is “too early” to test, and disregarded for being unimportant. But it's not too early to test; these early adopter games are living pieces of software, and they constantly change – that makes them perfect to build test data for a new API.

Look – these technologies are new, and that's precisely why they should be tested. There are two primary reasons we've used new titles to benchmark Dx12 and Vulkan performance: (1) An early indication of performance shows whether the tech works and plots the first data point in a long-haul look at performance progression; (2) validating claims made by multi-billion and multi-million dollar companies is an assurance of consumer protection against run-away hype, and the checks and balances afforded by independent media help keep companies honest.

The first point is the most critical in the case of APIs, as it's only a matter of time before Dx12 and Vulkan begin an en-masse switch-over from Dx11 and OpenGL for PC gaming.

Our approach to tested content is deeply embedded in the scientific method. It is our belief that, with these new APIs and GPUs and what-have-you, testing early and often builds a data-set which can later be analyzed for plotting trendlines and creating a pool of results which helps project into the future. That's valuable knowledge. Maybe we revisit these things in a few months and find major, 30% hikes in performance that bring parity between Dx11 and Vulkan; that'd be critical, and it'd show a very serious investment in the future by accommodating developers. Some time later – maybe it's a year from now, who knows – we retest again and find that Vulkan or Dx12 have overtaken Dx11 in most tested games. That's more big news, and helps us establish a clear, precise “switchover” point in the time-line of technology.

If we decided not to test the new APIs until everything's perfectly built, ground-up as it should be, we'd be waiting years – and there's certainly no harm in knowing more, even if it's early information. Not to mention that, from a competitive standpoint (because this is ultimately a business), no one really cares at that point. That's a large part of why people read these sites – to get information ahead of purchases or launches – and so it makes sense that we'd follow that trend with API testing.

Were we to sensationalize headlines with statements like “Vulkan 30% Slower than Dx11,” that'd be one thing – but we didn't, our headline for Vulkan (specifically) was “Initial Vulkan Benchmark vs. DirectX 11 - AMD & NVidia in Talos Principle.” Did you catch it? “Initial.” That's not there by accident, and neither is the multi-paragraph section entitled “Read This First!,” which goes on to quote the developers' own FAQ about early support being unoptimized. It's also, honestly, not that compelling of a headline to click on given just how sterile and neutral the phrasing is.

The same is true for our Dx12 benchmark, which used a fantastically sterile headline of “AMD+NVIDIA Multi-GPU DirectX12 Benchmark.”

We're approaching these topics neutrally and with great emphasis on explaining the state of affairs within the API world, with routine conclusions that follow-ups should plot an exciting, upward trend for the tech. And, if for some reason that doesn't happen, we'll be here to catch it and start asking questions of the relevant IHVs and ISVs.

That's why we test this stuff early.

Just thought we'd point that out. The majority of our readers and viewers have supported these efforts, but a select few comments on YouTube and reddit have inspired this post as a means to hopefully provide education on the process. It's very frustrating as a smaller media outlet to see outlier comments condemning hours- and days-long efforts to benchmark new technology, often for reasons shrouded in either hate or shortsightedness. That's part of the deal with posting online, of course, and so we're used to it – but this particular topic required discussion.

If you like our content, feel free to scope us out on Patreon.

Editorial: Steve “Lelldorianx” Burke
Video Editing: Andrew “ColossalCake” Coleman

Last modified on March 02, 2016 at 11:35 am
Steve Burke

Steve started GamersNexus back when it was just a cool name, and now it's grown into an expansive website with an overwhelming amount of features. He recalls his first difficult decision with GN's direction: "I didn't know whether or not I wanted 'Gamers' to have a possessive apostrophe -- I mean, grammatically it should, but I didn't like it in the name. It was ugly. I also had people who were typing apostrophes into the address bar - sigh. It made sense to just leave it as 'Gamers.'"

First world problems, Steve. First world problems.

We moderate comments on a ~24~48 hour cycle. There will be some delay after submitting a comment.


  VigLink badge