CPU Test Methodology
Our usual testing methodology is defined further below, per typical copy/paste of methods between reviews, but we have a few explicit items to address with Ryzen.
During testing, GamersNexus discovered that Windows would occasionally engage core parking or other power saving features that could impact framerate in some games, specifically with Ryzen. This was not always detectable, but some games reacted more than others (our biggest observable difference was ~4-5%). As such, all tests were conducted in performance mode. We further discovered that SMT toggling improves performance in nearly all gaming use cases, and so tested with SMT both enabled and disabled. When asked about this, AMD supplied that it is likely on the ISV optimization side.
We’ve been using the newest (and correct) version of AMD’s chipset drivers since we began testing, and we’ve been on ASUS’ version 5704 (latest) BIOS for the Crosshair Hero VI. ASUS was good to provide us this update immediately when we reached out, though AMD officially distributed an older EFI version to reviewers; this older version is detrimental performance in some ways, as it was not the production-ready EFI. In speaking with other editors, MSI boards went through similar EFI updates that drastically changed performance, in some cases. A few toggles are broken in various EFIs we've looked at here, which can potentially skew results if the tester does not validate that the toggles work as advertised (performance modes, for instance, can toggle threads in some extreme cases). It is relatively easy to avoid unintentional corruption of test data so long as the tester rigorously validates thread count, clock-rate, and voltages prior to each major test sequence.
Fortunately, we had already retrieved the correct EFI from ASUS directly, and so did not have to test multiple EFI versions – though we did have to retest the platform and CPU multiple times for other validation. EFI version matters a lot in these very early days of Ryzen. This is an entirely new architecture – not some updated on a long-known design, like KBL was – and vendors were not prepared. Early EFI distributions were based on older uCode. Results will vary between test environments in greater ways than usual.
This part applies to all motherboards, CPUs, and platforms: For testing, we disable any performance bias options made available through EFI. ASUS often offers Cinebench or 3DMark performance bias configurations (boosting performance in specific applications). We disable these for fairness across platforms. All stock CPU options are left enabled for benchmarking, like Boost on Intel and AMD.
A clean image is used for all testing. We have Windows images on multiple SSDs, each labeled for the appropriate CPU architecture. We pull an SSD with the correct label from the shelf (e.g. Skylake, Broadwell-E, Ryzen) and use that for testing. Note also that Windows HPET can affect performance.
Because of how Ryzen’s boosting reacts to temperatures, we put the R7 1800X under the same liquid cooling we’ve used for all our major CPU reviews: the Kraken X62. This eliminates unpredictable thermal throttles during test.
Game Test Methodology
NVIDIA 376.33 drivers were used for all benchmarking. Game settings were manually controlled for the DUT. All games were run at presets defined in their respective charts. All other game settings are defined in respective game benchmarks, which we publish separately from GPU and CPU reviews. Our test courses, in the event manual testing is executed, are also uploaded within that content. This allows others to replicate our results by studying our bench courses.
Windows 10-64 was used for testing.
Each game was tested for 30 seconds in an identical scenario, then repeated three times for parity.
Average FPS, 1% low, and 0.1% low times are measured. We do not measure maximum or minimum FPS results as we consider these numbers to be pure outliers. Instead, we take an average of the lowest 1% of results (1% low) to show real-world, noticeable dips; we then take an average of the lowest 0.1% of results for severe spikes.
Core Components (Unchanging)
- NZXT 1200W Hale90v2
- For DDR4 platforms: Corsair Vengeance LPX 32GB 3200MHz*
- For Ryzen DDR4: Corsair Vengeance LPX 3000MHz clocked to 2933MHz (See Page 2)
- Premiere & Blender tests do not exceed 8GB DRAM. Capacity is a non-issue for our testing, so long as it is >16GB
- For DDR3 platforms: HyperX Savage 32GB 2400MHz
- Intel 730 480GB SSD
- Open Air Test Bench
- Cooler #1 (Air): Be Quiet! Dark Rock 3
- Cooler #2 (Cheap liquid): Asetek 570LC w/ Gentle Typhoon fan
- Cooler #3 (High-end): Kraken X62
Note: fan and pump settings are configured on a per-test basis.
AM4 Platform:
Used for R7 1800X, R7 1700X, R7 1700.
Z270 Platforms:
- Gigabyte Aorus Gaming 7 (primary)
- MSI Gaming Pro Carbon (secondary - for thermal validation)
- i7-7700K (x2) samples from motherboard vendors
Both used for the 7700K.
Z170 Platform:
- MSI Gaming M7
- i7-6700K retail
Z97 Platform:
- Gigabyte Z97X G1 WIFI-BK
- i7-4790K
Z77 Platform:
- MSI GD65 Z77
- i7-2600K
Dx12 games are benchmarked using PresentMon onPresent, with further data analysis from GN-made tools.
Note: We'd like to add the i5, i3, and FX CPUs, but this was enough for now. We'll add those as we expand into coverage of Zen or i5 Kaby Lake products.
Thermal Test Methodology
Thermal measurement on Ryzen is not necessarily trivial, as nearly all software is incorrect or inaccurate. We will discuss precisely why this is the case on the next page, which contains thermal testing. We decided to push the bulk of the thermal methodology to that page, since people don’t usually read this methodology page. That will hopefully increase visibility to the information.
Power testing is simply done at the wall. We do not presently tap into the rails, and openly identify this as our weakest point in current test methodology. This is something we will eventually work toward revamping. For today, we use wall meters to determine a power delta in A/B tests.
