How Does This Help Me?

By using wide-reaching, industry-known benchmarking tools – to include real-world tests (Metro) and synthetic tests (3DM) – you're able to compare graphics performance to our own metrics to help isolate variables. As our test bench will almost certainly be a different configuration than your system, comparing 3D Mark scores and standardized in-game performance of identical graphics cards (to our bench) will help you understand variance introduced by the CPU; you can then account for this variable (delta in FPS) when looking at our benchmarks in the future, ultimately determining if your GPU can run a new game at high settings.

Other than that, it's good for troubleshooting, ensuring performance is as advertised, burning-in refurbished products, and testing overclocks for stability. We've previously written guides for benchmarking GPUs, but this is updated for Maxwell, impending AMD products, and in-game FPS testing. Our previous article relied more heavily upon synthetic analysis.

Required Tools

For purposes of this slimmed-down test approach, you'll need the following software:

• FPS logging utility, ideally FRAPS. MSI Afterburner can also be used.
• FRAFS, the free analysis tool that presents FRAPS benchmark files in an easy-to-read format.
• A game, ideally one with a built-in benchmark (Metro 2033, GRID: Autosport, Civilization, etc.).
• Synthetic tools (3D Mark's free version, Valley, or Heaven).

Basic Procedure – Real-World Testing

We won't go through the extensive isolation detail of our in-house test cases, but can cover the basics. It's important to ensure that the system has the latest drivers installed for purposes of testing, as our website (and many others) always update drivers before a new test pass.

Once the drivers are updated, launch FRAPS and navigate to the “FPS” tab. Checkmark the three logging options and set the time. 30 seconds, 60 seconds, and 120 seconds are all reasonable. We'd encourage a 30-60 second timer for most testing procedures.

Launch a game for testing. In an ideal environment, you use a real game that has a built-in, in-game-rendered benchmark; Metro 2033 and GRID Autosport are good examples of games that offer this. By using an in-game, preset benchmark, we ensure a realistic view of GPU performance while eliminating tester-induced variance exhibited from haphazard manual benchmarking. If the tester is performing a manual benchmark, it is critical to the benchmark's reproduction that the same map, location on the map, and game events are experienced in each test pass. By introducing complex combat elements and dynamic events, the tester runs the risk of invalidating results.

Once a benchmark route or in-game bench is started, press the FRAPS benchmarking hotkey to log FPS and frame latency. Conclude the test, then navigate to the folder that contains this benchmark data. Drag the frametime file to FRAFs for analysis.

This presents three key metrics: Average FPS, 1% low, and 0.1% low. Because minimum and maximum FPS are useless outliers, we rely more heavily on 1% and 0.1% lows as our indicators of frame dips and lag. 1% time low FPS means the average of the lowest framerates 1% of the time, a noticeable dip percentage when gaming.

Frametimes show the latency between each frame delivered. It is important that these are as close as possible, even if they're slow; consistency matters most, as the human eye will primarily detect stutter and frame output lag when frames are delivered with inconsistent gaps between them.

We recommend repeating these tests 3 times for parity. If results are vastly different, there is a disparity in the benchmark, game, or the method of implementation by the tester. Test results should be considered invalid if there is a delta greater than 5% between each test.

Synthetic Testing Software

Synthetic tests offer an important, replicable approach to hardware testing. Although scores may appear somewhat nebulous at first, they are combined from a number of realistic framerate logs (in 3D Mark, these are physics, game, and combined tests) that create a reliable portrait of performance. Synthetic tests not only offer a stable platform for comparative analysis system-to-system, but also are trustworthy in their isolation of variables.

Note that some synthetic tests are more “real-world” than others, like 3DMark and Valley, both of which effectively act like extremely demanding games. Other benchmarks, like FurMark, abuse the GPU computationally and don't necessarily resemble anything “game-like” in nature.

Also note that some synthetic software does not adequately invoke the boost clock. Both AMD and nVidia have implemented clock throttling for TDP reasons – if the GPU isn't under enough load to make use of a higher clockrate, they'll automatically down-clock to preserve power and extend the GPU's overall life. These are variables to consider when selecting test platforms. You want something demanding enough to invoke all the available power from the GPU, but not something that doesn't understand how the boost clock should work (FurMark has had issues with this in the past).

What Next?

Post your results! Let us know your CPU and GPU below, then post your FPS results for various games as we continue to benchmark them. This provides a community pool of information that can be used in future troubleshooting and benchmarking attempts, ultimately enabling gamers to self-troubleshoot when a game isn't performing as expected.

- Steve “Lelldorianx” Burke.