Explanation of AMD OCAT Overlay for PresentMon Benchmarking

By Published December 09, 2016 at 8:30 am
  •  

Our full OCAT content piece is still pending publication, as we ran into some blocking issues when working with AMD’s OCAT benchmarking utility. In speaking with the AMD team, those are being worked-out behind the scenes for this pre-release software, and are still being actively documented. For now, we decided to push a quick overview of OCAT, what it does, and how the tool will theoretically make it easier for all users to perform Dx12 & Vulkan benchmarks going forward. We’ll revisit with a performance and overhead analysis once the tool works out some of its bugs.

The basics, then: AMD has only built the interface and overlay here, and uses the existing, open source Intel+Microsoft amalgam of PresentMon to perform the hooking and performance interception. We’ve already been detailing PresentMon in our benchmarking methods for a few months now, using PresentMon monitoring low-level API performance and using Python and Perl scripts built by GN for data analysis. That’s the thing, though – PresentMon isn’t necessarily easy to understand, and our model of usage revolves entirely around command line. We’re using the preset commands established by the tool’s developers, then crunching data with spreadsheets and scripts. That’s not user-friendly for a casual audience.

Just to deploy the tool, Visual Studio package requirements and a rudimentary understanding of CMD – while not hard to figure out – mean that it’s not exactly fit to offer easy benchmarking for users. And even for technical media, an out-of-box PresentMon isn’t exactly the fastest tool to work with.

What Does OCAT Do?

OCAT is AMD’s addition to PresentMon. AMD has plans to open the source for OCAT shortly, but from preliminary testing, it looks like OCAT should impose minimal overhead and retain the exact dataset that PresentMon outputs. When OCAT works, it adds an overlay that functions brilliantly to aid in test execution by informing testers that the benchmark has started, stopped, and actually captured data. Hotkey, target exe, and time expiry settings can all be configured through the UI or through a bundled-in configuration text file. We find this useful for automation purposes, but it’s generally just nice to have text file access to configurations.

OCAT also provides on-the-fly frametime and framerate numbers in its overlay, further aiding the validation process. We find this live, during-bench feedback to be important for sort of heuristically ensuring accurate data collection. You wouldn’t want to run endurance FPS benchmarks and find that the tool wasn’t capturing correctly, for instance, and that’s theoretically resolved with OCAT’s overlay.

amd-ocat-screenshot

This is entirely a play to expand usability of PresentMon, which is an already trusted, understood, and well-documented tool that has hands-on development from industry veterans. Again, Intel and Microsoft worked together to hook PresentMon into ETW, the tool monitors both onPresent and onDisplay variables, and outputs a ton of frametime data for careful analysis. It’s also fully functional with Dx11, Dx12, UWP, OpenGL, and Vulkan, and we’ve yet to encounter a game where PresentMon fails to perform. Given the nature of shims between applications and APIs in this new world of low-level programming interfaces, PresentMon was a major saving grace that helped get us off the reliance upon aged tools. FRAPS, for instance, simply doesn’t officially work with the newer APIs (though can sometimes be forced to work), and some of the other existing tools just ignore a lot of important frametime data. Alternatively, they may just measure data differently, or calculate percentiles with formulas that seem confusing at times.

OCAT retains both the onPresent and onDisplay measurements for frames, allowing us to better leverage software to analyze performance for different use cases. onDisplay, for instance, has some serious value adds when conducting multi-GPU benchmarks. onPresent is what we prefer for almost everything, but we still collect both types of data to better understand the game’s (or hardware’s) behavior.

Just to re-iterate something, here’s a sample file from PresentMon:

amd-ocat-presentmon-sample

And here’s a sample file from OCAT (built atop PresentMon):

amd-ocat-bf1-sample

The columns are the same. Nothing moves. Nothing is added. The only reason you are seeing differences is because they’re different test runs – focus on the columns and formatting.

This means that existing tools (like our internal Perl + Python scripting suite) will still function perfectly with data captured by OCAT, while now offering the advantage of being more accessible. From a benchmarker standpoint, and having used OCAT for a little while now, it’s already clear to GN that this interface will speed-up our production time in a non-trivial fashion. Just talking about working with technicians, the training time for OCAT will be lower than for PresentMon through CMD, and that also reduces chance for technician error. From a casual user’s perspective, it’s just easier to use and get a quick, informed understanding of performance.

OCAT Bugs & Waiting for Final Analysis

Now, unfortunately, we’ve been running into bugs. Our team has been in communication with AMD over these issues for a little while now, as we’ve been trying to troubleshoot and isolate root causes. So far, Battlefield 1 throws an error with the newest patch, GTA  V fails to hook, and UWP has some known issues as well. We were able to successfully execute Metro: Last Light tests, rather boringly, but that just wasn’t enough to justify pushing forward with publishing performance results. The end goal is to analyze overhead – of which there should be nearly none – and the quality of the data output. That will wait until some more bug fixes are in place.

A huge production concern would be if game patches or new game releases require manual updating by AMD, since that basically kills pre-launch use by media, but we don’t yet know if that’ll be the case. PresentMon generally won’t break or require manual updates for new games, but we are unsure of OCAT’s pickiness at this time.

A Promising Addition

In the meantime, OCAT thus far has proven exceptionally promising and stands as an exciting addition to the PresentMon tool. We already trust our PresentMon data, understand it, and agree with its implementation. Getting it easier to use hasn’t been a trivial task, and AMD’s trying to take charge on that goal. We hope to see OCAT mature rapidly for deployment in testing sometime shortly. As a value-add, AMD tells GN that OCAT will eventually have its source code opened, further ensuring that vendors are playing fair when providing benchmarking utilities.

Editorial: Steve “Lelldorianx” Burke
Video: Andrew “ColossalCake” Coleman

Last modified on December 08, 2016 at 8:30 am
Steve Burke

Steve started GamersNexus back when it was just a cool name, and now it's grown into an expansive website with an overwhelming amount of features. He recalls his first difficult decision with GN's direction: "I didn't know whether or not I wanted 'Gamers' to have a possessive apostrophe -- I mean, grammatically it should, but I didn't like it in the name. It was ugly. I also had people who were typing apostrophes into the address bar - sigh. It made sense to just leave it as 'Gamers.'"

First world problems, Steve. First world problems.

We moderate comments on a ~24~48 hour cycle. There will be some delay after submitting a comment.

Advertisement:

  VigLink badge