Console Testing Methodology
As usual, we’re making constant incremental improvements to the software we built and use to analyze console framerates. If nothing else, PUBG was an excellent benchmark of the tool, because it’s the only console game we’ve tested so far that isn’t nearly constantly at its framerate limit. The framerate cap in this case is 30FPS, but it takes a few minutes of walking around and looting to stabilize. Adapting the tool to work with lossy frames makes our lives a lot easier, since we don’t have to capture at an insane bitrate or store huge files. It’s tricky, though, because the tool calculates framerate by comparing the pixels of consecutive frames and detecting duplicates--and if a frame is visibly compressed, it might not be seen as a duplicate.
This is a visual representation of a real unique frame. White pixels represent areas that are different enough from the previous frame to be considered unique. Sometimes there are a lot of white pixels, and sometimes there are very few--it’s hard to predict.
This is an example of what we have to filter for. This frame should definitely be counted as a duplicate, but the white pixels reveal that there are technically differences; however, the white pixels are isolated and scattered throughout the frame, which isn’t what we’d expect from most truly unique frames.
This is what the first example looks like when we ignore everything except white pixels that are surrounded by other white pixels. It does ignore some valid differences, but leaves plenty to work with. Applying this same filter to the second example, on the other hand, results in nothing but black.
PUBG Xbox One X Benchmarks & Framerate Analysis
From the instant the game was launched, it was clear what we were in for. The player character is displayed in the main menu screen, but it takes a second after their appearance for clothing and textures to follow. Even once everything has settled down, when new items are equipped they also take time to load in. This texture pop-in is constant, and every change of location--the starting area, the plane, wherever the player walks--temporarily turns the world into a muddy brown mess. More than it’s supposed to be, even.
Pop-in issues are a recurring theme for the Xbox One X version of PUBG. One oddity with the game is that pop-in and LOD scaling often seems to draw completely new, or incorrectly scaled, objects in place of the lower LOD objects. You can see this in the trees in the distance, where it appears that the mesh changes completely as we approach them. This is also highly visible in a scene where we’re crashing through fences with a buggy. If we slow down time – which isn’t hard, seeing as we’re running below 30FPS – you’ll see in the video that it appears the static mesh fence is being replaced with a dynamic particle mesh version of the fence. This would be done for destruction, as tracking dynamic particle meshes for every object across the entire game would likely cost too much in resources, and so swapping out upon impact could save resource cost. It appears as if the dynamic particle mesh is scaled incorrectly, though; the replacement is larger than the static mesh, and we rubberband through it as the fence breaks apart.
Above: Normal map issues on the underside of the parachute.
Texture pop-in is probably the most common issue, though. The parachute is an easy, first-seen example: Dropping into the map, the parachute takes several seconds to load its textures fully, and is exhibiting a normal map issue on the underside of the parachute. This seems to happen mostly when the camera is at glancing angles to the parachute, revealing the normal map artifacting underneath. This specific set of issues is particularly embarrassing for PUBG, seeing as the parachute is an object only seen once in the game – at the very beginning – and then it’s gone for the rest of the match.
Texture pop-in issues can basically be seen everywhere: We see them on cars, the ground, the plane, bridges while skydiving, buildings, tree alpha textures, and more. There are also problems with environment pop-in, like when we observed parts of the world failing to draw, which seems like a culling bug of some kind. Speaking of occlusion bugs, there’s an occlusion culling issue with fast camera panning, particularly noticeable from the plane and pre-airdrop. Panning the camera quickly enough reveals a draw delay where the plane once occluded the land below. You can see white space draw beneath the plane. This isn’t really a big deal as it’s not a part of the game where it’d matter, but just further illustrates the incomplete nature of PUBG on Xbox One X.
It’s not just textures, of course. Playing on a Fiber connection, we’ve also noticed severe rubberbanding that would indicate a server-side network or synchronization issue. This is partly visible when driving through destructible objects, as the static meshes are swapped with particle meshes, but is even more visible in occasional combat scenarios. In one example, we punched a lagged-out player to death, then attempted to enter a garage with a motorcycle. The kill counter updated as 0 kills, we rubberbanded in and out of the garage for a moment, and then it ticked-over to 1 kill. Although less significant because it is pre-combat, we also experienced rubberbanding when approaching the picnic tables. It seemed to become more of an issue as more players joined. Similarly, in our full 12-minute recording pass, we also noticed the fluctuating player count impacted even the ground texture resolution, with the ground texture’s higher LOD unloading at one point.
Other oddities also exist, like walls that vanish and re-appear.
Performance is disproportionately bad for first few minutes of every match. If there’s any place in the game that should really overwhelm the Xbox, it’s the pre-match free-for-all where 100 players scramble to fire as many weapons as possible in close proximity. Assets load in, players all leap and squirm around en masse, explosions go off everywhere. Every time a molotov explodes, the framerate tanks; with a good fireball in view, the framerate can dip to 12FPS. Seeing an explosion during gameplay generally means that an enemy is within throwing distance, which is exactly the wrong time for performance to drop. Watching smoke grenades go off in real combat had barely any impact, which makes this seem like a molotov-specific issue, but we’d have to find and throw one in the course of a match to see if it still happens outside of the hectic pre-match environment.
After a countdown, everyone is abruptly teleported into a featureless brown tube which slowly resolves into a plane. For gameplay, performance doesn’t matter until boots hit the ground, but it’s still hard to accept the amount of time it takes to load trees and houses as you dive towards them at 209km/h. In our first run, framerate dropped to around 15FPS between parachute deployment and hitting the ground. That was also just about enough time for the parachute’s texture to fully load.
The first couple minutes after landing are the most dangerous. Our average framerate stayed around 25FPS in this timeframe, and that potentially makes parachuting into popular areas and competing for resources more difficult. After that, framerates generally stayed around 30 if we stayed in one place or walked, but dropped if we drove too quickly into new areas on a vehicle or smashed fences. That might not seem too bad, but again, most console games do their utmost to never drop below their framerate goal for even a fraction of a second--especially when that goal is only 30FPS. Surprisingly, one-on-one shooting and punching had little impact on performance compared to the dips seen while driving a vehicle. LOD distance is obnoxiously low on foliage and trees and there’s a clearly visible line of grass that advances before the player.
Network problems and general jankiness were far more pressing problems than frames per second during our time with the game. On more than one occasion, it was possible to walk straight up to an enemy and punch them, either because they were helplessly frozen by lag or incapable of shooting at close range with a controller. Lag-induced rubber banding happened multiple times. On our highest-scoring run, we only encountered two other players, both of whom chose to just drive away from us rather than deal with a fight. Eventually, we hit a rock at 90MPH and ran over ourselves with our own bike, but that can’t be blamed on the game. Watching this death frame by frame shows our character hitting the rock, flipping upside down, teleporting back towards the rock in a standing pose without the bike, equipping his gun, sliding back towards the river again, and then abruptly dying. The camera then pans up to an invisible hill and plummets again as the ragdoll falls through the bottom of the river.
We did do one test run at 1080p, and framerates still dropped below 30FPS in the early parts of the game. We may do more precise comparisons in the future if we do a PC benchmark, but for now we can say that running at 1080p definitely doesn’t solve the problems we’ve described.
PUBG on the Xbox One is a bizarre juxtaposition between the idea of pro gamers who do precision mental calculus every time they fire a shot, and the reality of a game where it’s entirely possible to win fights by charging in throwing punches like an angry gorilla. It’s also an extremely ugly game right now, and playing at 4K doesn’t do much other than multiply the number of muddy brown pixels. Still, it’s not fair to judge it on appearance alone--it’s not like the PC version is winning any beauty contests, and it blows past two million concurrent players on a daily basis.
The Xbox doesn’t seem like an ideal platform for the long minutes of tension broken up by sniper duels and brief, frenetic firefights that make PUBG fun, but that’s still a personal opinion. In its current early-access state, PUBG on Xbox One is technically playable, but it’s hard to build an argument for it vs. the mouse-and-keyboard PC version or the console version of Fortnite. The game is a laggy, stuttery mess, and the graphics don’t do anything to justify the frame drops.
We’ll be doing PC tests soon. In the Xbox version’s favor, we didn’t see any hackers (a notorious problem on PC), and matchmaking took barely any time at all despite what is presumably a smaller player base. We’ll also probably do more testing with the console version as it nears 1.0 release.
Editorial, Testing: Patrick Lathan
Host: Steve Burke
Video: Andrew Coleman