RAM Test Methodology Pt. 2 - The Tools & Procedures

Where RAM goes to die. From our Kingston HQ tour last year.

I wrote a full-feature test case for this bench that was followed to the 't,' so each test was executed under identical conditions (beyond our one variable that changes). For trade reasons, we aren't releasing the start-to-finish test case spreadsheet, but I'll walk through most of it here. If you'd like to replicate my results, this should provide adequate tools and information to do so (please post your own below!).

Some standard benchmarking practices will go undetailed. Starting with a clean image, for instance, doesn't need much explanation.

We ran the following tools in the order they are listed:

Synethetic:

• Euler 3D.
• MaxxMem.
• WinRAR.
• Handbrake.
• Cinebench.
• Audacity LAME (honorable mention; was not significant enough to use).

Real-World

• Shogun 2 Benchmark (load time).
• Shogun 2 Benchmark (FPS).
• Adobe Premiere encoding pass.
• Adobe After Effects live RAM preview pass.

Each test was conducted numerous times for parity. The shorter tests were run 5 times (Euler 3D, MaxxMem, Cinebench), the longer tests were run 3 times - because we do have publication deadlines (WinRAR, Handbrake, Premiere, After Effects), and the tests that produced varied results were run until we felt they were producing predictable outcomes (Shogun 2, Cinebench). All of these were averaged into a single subset of numbers for the tables.

Shogun 2 and Cinebench exhibited some performance quirks that threw the numbers off at first, but after a good deal of experimenting, I figured out why. Shogun 2 had just been installed, and its first launch of the bench was significantly slower than all subsequent launches (nearly 16%, in fact). My understanding is that the game unpacked some files upon first launch (maybe dX), and so these results were thrown out. The game continued to produce confusing load times, sometimes varying by nearly 11% on identical configurations. This was resolved by rebooting, verifying the game cache, rebooting, and starting the test again. After all of this, the results were almost perfectly consistent over a span of 15 tests and retests (within 1%, easily in margin of error).

Cinebench's performance declined heavily after each subsequent test. Severely. The first OpenGL single-channel pass reported 111FPS, the second 108, the third 106, the fourth 105, and then it flatlined around there. This was concerning. The test did this after numerous reboots and after changing to a dual-channel config. I eventually realized that the performance degradation only existed when the application continued testing without being terminated between tests. By this, I mean that the variance subsided when I started closing the program after each pass, then reopening it. This method was used for another 10-15 passes, all of which produced results within 1% of each other.

RAM was purged between all tests. No active memory management programs were present outside of the default, clean Windows installation.

STARS Euler 3D: CFD Simulation Performance 

Euler 3D is best-used when testing CPUs and memory. The program is freely available through Casealab.

Figure 1: AGARD 445.6 Mach Contours at Mach 0.960. Source.

Euler 3D performs heavy floating-point and multi-threaded tasks that leverage resources in a similar manner to heavy-duty compiling and simulation. This is done using Computational Fluid Dynamics (CFD) and measures output "as a CFD cycle frequency in Hertz." Caselab defines their software's test sequence with great specificity, stating:

"The benchmark testcase is the AGARD 445.6 aeroelastic test wing. The wing uses a NACA 65A004 airfoil section and has a panel aspect ratio of 1.65, a taper ratio of 0.66, and a 45 degree quarter-chord sweep angle. This AGARD wing was tested at the NASALangleyResearchCenter in the 16-foot Transonic Dynamics Tunnel and is a standard aeroelastic test case used for validation of unsteady, compressible CFD codes. Figure 1 shows the CFD predicted Mach contours for a freestream Mach number of 0.960.

The benchmark CFD grid contains 1.23 million tetrahedral elements and 223 thousand nodes. The benchmark executable advances the Mach 0.50 AGARD flow solution. Our benchmark score is reported as a CFD cycle frequency in Hertz."

A lot of these words are fairly intimidating, but most of them are explaining that the test case is simulating a specific aircraft component at a specific speed. Simulation like this is a computation-intensive task that tends to demand high-speed components and multithreaded processing. CFD uses a computer model to calculate fluid flow (air, in this case). My understanding is that the test uses a complex matrix to calculate fluid flow at multiple points, then iterates (over time) with change in flow at surrounding points. Assuming this is correct, it means that we have to store values and access them upon each iteration, so RAM speed becomes important.                                                                                                  

I should note that I'm not an expert in CFD -- not anywhere remotely close -- but after speaking with GN's resident physics engineer (Tim "Space_man" Martin), that was our educated speculation.

Regardless, you don't need to know how it works to know that a higher frequency is better. We ran the test with settings of "14 / 4," where 14 is indicative of the count of passes and 4 is the active thread-count.

A higher frequency rating is better in this test.

Anyone performing simulation or computation-intensive compiling tasks will benefit from these numbers; if you're in these categories, use this benchmark to judge how much multi-channeling matters to you.

MaxxMem: Memory Copy, Read, Write, Latency, & Bandwidth Performance

MaxxMem is another synthetic test. I won't spend as much time here since it's a bit easier to understand.

NOTE: These aren't our results. Just a program screenshot.

MaxxMem tests memory bandwidth by writing to and reading from memory blocks that are sized in powers of 2, starting at 16MB and scaling up to 512MB. MaxxMem executes numerous passes and averages the result, though we did our own multiple passes to ensure accuracy. MaxxMem attempts to address CPU cache pollution concerns upon reads/writes, which can become a problem with some real-world tests that will skew results if improperly tested. The test uses "an aggressive data prefetching algorithm" to test theoretical bandwidth caps during read/writes.

WinRAR Compression: Large File Compression Speed Test 

WinRAR is a pretty standard test for memory and CPUs. Fast memory enables rapid swapping during I/O to ensure the disk isn't hit too frequently once the process initializes. This is very much a 'real-world' test, but I've listed it as 'synthetic' since I'd imagine most users don't regularly perform large file compression.

Our tests were conducted with heavy compression settings and with 22GB of files, output as a WinRAR archive.

Server architects, web servers, database servers, and computers that perform regular log file archival, indexing, or similar tasks will directly be represented by this test.

Handbrake, Audacity, & Cinebench 

These two just get quick mentions. Handbrake is a multithreaded transcoding program that has one video format / codec as input and another as output. We put a 9GB AVI recorded with FRAPS into Handbrake and spat out an h.264 (5.1 high profile @ 1080) MP4 at a continuous 60FPS with a 20Mbps datarate. In theory, this is CPU- and memory-intensive and will see the benefits of faster memory.

Audacity gets a small, honorable mention. I thought that using a 4096Kbps high-quality WAV file (recorded with our field reporting equipment) and asking LAME to convert it to MP3 would be pretty abusive, but it wasn't. The output times were almost identical -- within 0.5% of each other. I do not feel comfortable publishing the results in an official fashion, but both single- and dual-channel performance hovered around 18.5 seconds for processing time. I feel like audio compression or conversion could be pretty intensive on memory, but I am not enough of an audio expert to know without more extensive research.

Cinebench is a pretty fun synthetic test to use. We relied upon the OpenGL benchmark, since that was the only one that would have any amount of impact from RAM differences. It runs a video and logs the FPS - pretty straight-forward stuff. This is more dependent upon the GPU and CPU than anything, but had some RAM variability.

Impact of Memory Speed on Adobe After Effects Live Preview & Premiere Encoding

I'm responsible for maintaining our entire YouTube channel here at GamersNexus (and do a good deal of hobbyist mountain bike video editing in my free time), so the video editing tests were of most interest to me. Premiere is an easy one, the test settings consisted of:

We rendered out a complex 10s clip that featured four [email protected] videos shot at 24- & 28Mbps. The clips had color correction and other post-processing effects applied and were rendered at maximum depth.

To test the performance of dual-channel vs. single-channel memory configurations with After Effects, we put together another [email protected] set of videos (down-scaled into quadrants) at 20-28Mbps. Two of the videos had color and exposure correction effects applied. We then previewed the video with RAM (RAM Preview) to check for RAM Preview stuttering or low framerates (expected). A video camera was set up to monitor the RAM Preview info panel for live FPS metrics. We used an external video camera so the results would be independent of the screen capture. You can view the test in video form on page 1.

Testing Platform

We have a brand new test bench that we assembled for the 2013-2014 period! Having moved away from our trusty i7-930 and GTX 580, the new bench includes the below components: 

For this test, the CPU was configured at 4.2GHz (instead of our usual 4.4GHz) with a vCore of 1.265V. The RAM was set to 2400MHz and run in bank 0 or mismatched banks.

Please continue to the final page to view the dual-channel vs. single channel memory platform benchmark results.