This project follows our GTX 1080 Hybrid, RX 480 Hybrid, and GTX 1060 Hybrid series articles. These articles are meant to achieve three primary objectives: (1) Show the disassembly process and the PCB, (2) learn about the card's potential limitations as related to the default cooling solution, and (3) discover if additional overclocking is possible through increased thermal headroom. We generally conclude, as one would expect, that there becomes a power or VBIOS/voltage limitation once the thermals have been resolved.
Still, the GTX 1080 DIY Hybrid project granted us an additional +100MHz on the OC just by switching to liquid, and stabilized our clock-rate to a perfectly flat (ideal) line, as opposed to the spiky mess of the reference cooler.
We're using an EVGA Hybrid cooler ($100) again, which is an Asetek supplied CLC that we've previously torn down. The coldplate's copper protrusion and densely packed microfins assist in controlling thermals, with a somewhat unimaginative (but effective) impeller pushing the liquid around.
We've found these coolers to outperform most mainstream-ready alternatives for GPU cooling. CPU CLC coldplates are shaped in a manner that benefits an IHS with specifically located hotspots, but does not work quite as effectively for the flat silicon found on a video card.
Game Test Methodology
We tested using our GPU test bench, detailed in the table below. Our thanks to supporting hardware vendors for supplying some of the test components.
NVidia's 372.54 drivers were used for game (FPS) testing on the Titan X Pascal. The 368.69 drivers were used for other devices. Game settings were manually controlled for the DUT. All games were run at presets defined in their respective charts. We disable brand-supported technologies in games, like The Witcher 3's HairWorks and HBAO. All other game settings are defined in respective game benchmarks, which we publish separately from GPU reviews. Our test courses, in the event manual testing is executed, are also uploaded within that content. This allows others to replicate our results by studying our bench courses. In AMD Radeon Settings, we disable all AMD "optimization" of graphics settings, e.g. filtration, tessellation, and AA techniques. This is to ensure that games are compared as "apples to apples" graphics output. We leave the application in control of its graphics, rather than the IHV. In NVIDIA's control panel, we disable G-Sync for testing (and disable FreeSync for AMD).
Windows 10-64 build 10586 was used for testing.
Each game was tested for 30 seconds in an identical scenario, then repeated three times for parity.
Average FPS, 1% low, and 0.1% low times are measured. We do not measure maximum or minimum FPS results as we consider these numbers to be pure outliers. Instead, we take an average of the lowest 1% of results (1% low) to show real-world, noticeable dips; we then take an average of the lowest 0.1% of results for severe spikes.
|GN Test Bench 2015||Name||Courtesy Of||Cost|
|Video Card||This is what we're testing!||-||-|
|CPU||Intel i7-5930K CPU||iBUYPOWER
|Memory||Corsair Dominator 32GB 3200MHz||Corsair||$210|
|Motherboard||EVGA X99 Classified||GamersNexus||$365|
|Power Supply||NZXT 1200W HALE90 V2||NZXT||$300|
|SSD||HyperX Savage SSD||Kingston Tech.||$130|
|Case||Top Deck Tech Station||GamersNexus||$250|
|CPU Cooler||NZXT Kraken X41 CLC||NZXT||$110|
Video Cards Tested
- MSI GTX 1060 Gaming X ($290)
- NVIDIA GTX 1060 FE ($300)
- AMD RX 480 8GB ($240)
- NVIDIA GTX 1080 Founders Edition ($700)
- NVIDIA GTX 980 Ti Reference ($650)
- NVIDIA GTX 980 Reference ($460)
- NVIDIA GTX 980 2x SLI Reference ($920)
- AMD R9 Fury X 4GB HBM ($630)
- AMD MSI R9 390X 8GB ($460)
- And more
Thermal Test Methodology
We strongly believe that our thermal testing methodology is the best on this side of the tech-media industry. We've validated our testing methodology with thermal chambers and have proven near-perfect accuracy of results.
Conducting thermal tests requires careful measurement of temperatures in the surrounding environment. We control for ambient by constantly measuring temperatures with K-Type thermocouples and infrared readers. We then produce charts using a Delta T(emperature) over Ambient value. This value subtracts the thermo-logged ambient value from the measured diode temperatures, producing a delta report of thermals. AIDA64 is used for logging thermals of silicon components, including the GPU diode. We additionally log core utilization and frequencies to ensure all components are firing as expected. Voltage levels are measured in addition to fan speeds, frequencies, and thermals. GPU-Z is deployed for redundancy and validation against AIDA64.
All open bench fans are configured to their maximum speed and connected straight to the PSU. This ensures minimal variance when testing, as automatically controlled fan speeds will reduce reliability of benchmarking. The CPU fan is set to use a custom fan curve that was devised in-house after a series of testing. We use a custom-built open air bench that mounts the CPU radiator out of the way of the airflow channels influencing the GPU, so the CPU heat is dumped where it will have no measurable impact on GPU temperatures.
We use an AMPROBE multi-diode thermocouple reader to log ambient actively. This ambient measurement is used to monitor fluctuations and is subtracted from absolute GPU diode readings to produce a delta value. For these tests, we configured the thermocouple reader's logging interval to 1s, matching the logging interval of GPU-Z and AIDA64. Data is calculated using a custom, in-house spreadsheet and software solution.
Endurance tests are conducted for new architectures or devices of particular interest, like the GTX 1080, R9 Fury X, or GTX 980 Ti Hybrid from EVGA. These endurance tests report temperature versus frequency (sometimes versus FPS), providing a look at how cards interact in real-world gaming scenarios over extended periods of time. Because benchmarks do not inherently burn-in a card for a reasonable play period, we use this test method as a net to isolate and discover issues of thermal throttling or frequency tolerance to temperature.
Our test starts with a two-minute idle period to gauge non-gaming performance. A script automatically triggers the beginning of a GPU-intensive benchmark running MSI Kombustor – Titan Lakes for 1080s. Because we use an in-house script, we are able to perfectly execute and align our tests between passes.
Titan X (Pascal) Temperature Testing – Peak Average Thermals
This first chart looks at delta T values, a critical means of comparative analysis that allows for control over potential variance. In our endurance charts, we switch back to non-delta values (straight reads from the GPU diode) to get a better understanding of throttle points on the core.
Our Hybrid mod brings us down to 19.85C from 59.4C (delta T over ambient), or a reduction of ~40C for the load temperature. Idle temperatures are a little lower than the 1080 for a few reasons: (1) the die size is 471mm^2 on the GP102 chip, whereas the GP104 chip is 314mm^2, and this extra surface area helps dissipate heat; (2) for liquid testing, we've improved our implementation by keeping the baseplate on the Titan X Hybrid, not done for the 1080 Hybrid.
Keep in mind that these results are for the out-of-box product, so there's not even overclocking and we've already dropped thermals by 40C.
Titan X (Pascal) Endurance Results – Throttle Point Discovered
59.4C delta T is pretty warm – the hottest on the bench, actually – and that puts the GPU Diode value in the 84-85C range. The GTX 1080 throttles at around 83C, as we've extensively shown, and uses normal Boost 3.0 functionality to down-clock the card along the volt-frequency curve, reducing thermals by reducing performance temporarily. Once the GPU has become satisfied with its new resting temperature, it will attempt again to increase the clock-rate. This cycle repeats ad infinitum while under load, and is a normal part of GPU functionality.
What is sub-optimal, however, is heavy throttling that results in drastically reduced clock-rates. Let's look at that.
The above is an endurance chart for the stock Titan XP, before applying our liquid cooling solution.
During our endurance test, we plotted the Titan XP with its stock cooler as throttling at around 84C. Every hit to 84C caused an immediate drop in clock-rate, and the clock-rate got stuck around 1544MHz, but sometimes would spike to 1670MHz or drop as low as the 1400s. The spec calls for a 1531MHz boost, on paper, and the card mostly achieves this. The chart makes it pretty clear that our clock-rate is spiking hard, and it's a result of thermals – not power limit. We occasionally warmed to 85C or 86C, but the card mostly throttles hard to keep itself at 83C. And, as we'll find in a moment, the card spec calls for well under its actual operating potential.
Here's a look at a small cross-section of raw data to show what's going on.
|Time (s)||Core MHz||GPU Diode|
You can see that the clock has a range of more than 100MHz (in this sample of data, we've got a range of 189.5MHz, with a high of 1657.5MHz and low of 1468MHz). This clock-rate swing presents itself in 0.1% and 1% low values for gamers, but for production, it'll mostly manifest as an overall loss of efficiency and slow-down in render times. Because the card is so fast already, though, it might not be apparent that the slow-down exists – at least, not until after fixing the reference design. Then it's more obvious.
Let's put that into perspective.
|Time (s)||Core MHz||GPU Diode (C)|
(Above table: Some raw data from the Hybrid endurance run).
So, the original chart plots us as hovering in the 1468-1657+ range with an 83-84C diode, averaging at around 1531MHz. With our Hybrid mod, we brought the Titan X Pascal up to nearly 1800MHz – and that's with absolutely no overclock at all. Again, this is running stock, which means that the card's spec sheet is under its actual potential performance, and that the cooler is “stealing” speed from the chip. On average, we're moving from 1531MHz with the stock cooler to an average of about 1784MHz with the liquid cooler. We've improved the clock-rate of the stock card by more than 200MHz just by fixing nVidia's poorly performing air cooler.
The only reason we're still seeing that spiky frequency plot is because the card is now choking on power, not thermals. We've resolved the thermal constraint and are now hitting power constraints, which can be resolved simply by increasing the power limit of the card. Of course, applying an OC will re-create the spiked performance, but fixing the cooler and increasing the power limit (with no OC) will flatten overall clock-rate.
There is an impact on overclocking, too. As with the GTX 1080 FE, we were able to extract additional performance from the chip by keeping it more reasonably cooled. It would be possible to cool-down the reference (and only) Titan XP by increasing fan RPM to 100%, but then you're creating a realistically undesirable scenario by dumping noise into the environment. We found the ~70% fan speed range tolerable from a user perspective and from a power perspective.
Titan X (Pascal) Overclocking – Stock Air Cooler
Here's the stepping table for the stock card:
|Peak Core CLK||Core Clock (MHz)||Core Offset (MHz)||Mem CLK (MHz)||Mem Offset (MHz)||Power Target (%)||Peak vCore||Fan Target (%)||5m Test||60m Endurance|
We're approaching 4000RPM, here. Not quite – but close, in the ~3500RPM range. We found stability on the stock card at 1911MHz core (+175MHz) and 1363.5MHz memory (actual clock, comes out to ~10.9GHz effective). Voltage could not be increased with the tools available at this time, so that column is fully under the card's control.
Titan X (Pascal) Hybrid Overclocking – GN's Liquid Cooler
Here's the stepping results for our Hybrid:
|Peak Core CLK||Core Clock (MHz)||Core Offset (MHz)||Mem CLK (MHz)||Mem Offset (MHz)||Power Target (%)||Peak vCore||Fan target||5m Test||60m Endurance|
We were able to overclock higher and with a greater sustained average clock-rate than before, thanks to the improved thermal solution. Still, the Titan XP chokes hard on power availability, and this is demonstrated by the fact that increasing the memory clock beyond 450MHz will actually decrease FPS, because the power is eaten from the core. We actually gained more FPS by dropping the memory from its stable +600MHz value to a +450MHz value, ultimately used on both devices. That seems to be the point beyond which diminishing returns come into play.
We landed at 2012MHz peak, or about 1974MHz resting clock with the Hybrid card. The stock card, with a loud and high RPM fan, was getting stuck at about +175MHz core offset, for a total of 1987MHz peak, or 1911MHz average.
We've managed to decrease thermals, noise, and increase the overclock by about 63MHz. That's not a huge OC add, but there's more to it than just the offset numbers – we've also got to look at stability, and now that thermals are not throttling the card, it's down to just a power limit.
FPS Difference (Briefly)
We'll talk about this more in our imminent Titan X (Pascal) review, now that we've had a generous reader loan us the card (thanks, Sam!), but this condensed FPS OC results chart will provide a quick preview of what's to come:
(Note: Hitting CPU limit at 1080p.)
Mirror's Edge Catalyst, 4K/High:
|Avg FPS||1.0% low||.1% low|
|Titan XP Air 1531MHz||60.3||48.3||43.7|
|Titan XP Air 1911MHz||69.67||53.33||45.67|
|Titan XP Hybrid 1974MHz||73.67||57.33||50.67|
We see a change of ~2% in Shadow of Mordor at 1080p and at 4K, moving just from 91FPS AVG to 93FPS AVG, but the 0.1% low values have been improved from 62FPS to 71FPS as a result of the more stable clock-rate (a change of 14.5%). Our testing methodology is picking up the clock-rate instability and reflecting a real impact, albeit one which is generally not observable with this particular title. Looking at GTA V, we see a performance swing of nearly 4% in the averages at 4K. 1080p doesn't really show any real change, but the 0.1% lows are marginally improved – again imperceptibly.
Mirror's Edge Catalyst posts the biggest swings, with a change of about 5% at 1440p and about 6% at 4K High.
Production workloads would be interesting to test, but we might not be able to get to those before we have to return this card to its rightful owner.
Conclusion: Titan XP Reference Cooler Should be Replaced
If you buy a Titan X (Pascal) GPU, we firmly believe that the cooler should be immediately replaced. Operating at ~83~85C for long uptimes – how this card is meant to be used – isn't going to be great for clock-rate stability. The Titan X should have been shipped with a stock CLC or with a better air setup, we think, and the FE cooler is inadequate for a chip of this power.
Very interesting results, for this one. We'll try to research further if we're able to procure a permanent sample.
Editorial: Steve “Lelldorianx” Burke
Video: Andrew “ColossalCake” Coleman