How We Test Laptops

The laptop-review process at PCMag.com carries on core traditions that date back to the establishment of PC Labs in 1984: We compare each system to others in its category on the basis of price, features, design, and in-house performance tests.

To evaluate performance, we use a suite of software-based benchmark tests and real-world applications and games, carefully chosen to highlight the strengths and weaknesses in the tested PC's mix of components. That evaluation ranges from the CPU and the memory subsystem to the machine's storage hardware and graphics silicon.

In some cases, we make use of standardized tests created by established benchmark developers. We've also created our own tests, where needed. We also regularly evaluate new benchmark solutions as they hit the market, and overhaul our testing procedure as needed to ensure that we can accurately reflect the effects of the latest technologies.

Our laptop testing breaks down into three rough classes of testing: productivity testing, graphics testing, and a battery-life trial. Here's a breakdown of each.

Productivity Testing

PCMark 10

Our first task is evaluating a laptop's everyday productivity performance using UL's PCMark 10 benchmark, which simulates real-world productivity and content-creation workflows. (UL, or Underwriters Labs, acquired Futuremark, the maker of the long-running PCMark and 3DMark benchmarks.)

We use PCMark 10 to assess overall system performance for office-centric tasks such as word processing, spreadsheet work, web browsing, and videoconferencing. The test generates a proprietary numeric score; higher numbers are better and are primarily meaningful compared to one another.

We run the main test suite supplied with the software, not the Express or Extended version. Note that all else being equal, a higher screen resolution will suppress a system's performance on PCMark 10. (The more pixels to push, the more resources required.)

PCMark 8 Storage

We then assess the speed of the laptop's main boot drive using another UL benchmark, PCMark 8. This test suite has a dedicated PCMark 8 Storage subtest that reports a proprietary numeric score...

As with PCMark 10, higher numbers are better. The results from laptops with cutting-edge solid-state drives (SSDs) tend to cluster together closely on this test.

Cinebench R15

Next in line is Maxon's CPU-crunching Cinebench R15 test. We run this test at the All Cores setting. Derived from Maxon's Cinema 4D modeling and rendering software, this test is a CPU horsepower test. It is fully threaded to make use of all available processor cores and threads. Think of it as an all-out processor deadlift.

Cinebench stresses the CPU rather than the GPU to render a complex image. The result is a proprietary score indicating a PC's suitability for processor-intensive workloads, when used with software that is fully threaded.

Handbrake 1.1.1

Cinebench is often a good predictor of our Handbrake video-editing trial. This is another tough, threaded workout that's highly CPU-dependent and scales well as you add cores and threads.

In this test, we put a stopwatch on test systems as they transcode a standard 12-minute clip of 4K video (the open-source Blender demo short movie Tears of Steel) to a 1080p MP4 file. We use the Fast 1080p30 preset in version 1.1.1 of the Handbrake app for this timed test. Lower results (i.e., faster times) are better.

Adobe Photoshop CC Photo Editing Test

Our final productivity test is a custom Adobe Photoshop image-editing benchmark. Using an early 2018 release of the Creative Cloud version of Photoshop, we apply a series of complex filters and effects (Dust, Watercolor, Stained Glass, Mosaic Tiles, Extrude, and multiple blur effects) to a PCMag-standard JPEG image. (We use a script executed via an Actions file of our own making.) We time each operation and, at the end, add up the total execution time. As with Handbrake, lower times are better here.

The Photoshop test stresses CPU, storage subsystem, and RAM, but it can also take advantage of most GPUs to speed up the process of applying filters. So systems with powerful graphics chips or cards may see a boost.

Graphics Performance

Judging graphics performance requires using tests that are challenging to every system yet yield meaningful comparisons across the field. We use some benchmarks that report proprietary scores and others that measure frames per second (fps), the frequency at which the graphics hardware renders frames in a sequence, which translates to how smooth the scene looks in motion.

Synthetic Tests: 3DMark and Superposition

The first graphics test is UL's 3DMark. The 3DMark suite comprises a host of different subtests that measure relative graphics muscle by rendering sequences of highly detailed, gaming-style 3D graphics. Many of these tests emphasize particles and lighting.

We run two different 3DMark subtests, Sky Diver and Fire Strike, which are suited to different types of systems. Both are DirectX 11 benchmarks, but Sky Diver is suited to laptops and midrange PCs, while Fire Strike is more demanding and made for high-end PCs to strut their stuff. The results are proprietary scores.

Also in our graphics mix is another synthetic graphics test, this time from Unigine. Like 3DMark, the Superposition test renders and pans through a detailed 3D scene and measures how the system copes. In this case, the rendering action happens in the company's eponymous Unigine engine, offering a different 3D workload scenario than 3DMark. This provides a second opinion on the machine's graphical prowess.

We present two Superposition results, run at the 720p Low and 1080p High presets. The scores are reported in frames per second, higher frame rates being better. For lower-end systems, maintaining at least 30fps is the realistic target, while more powerful computers should ideally attain at least 60fps at the test resolution.

Real-World Gaming Tests

The synthetic tests above are helpful for measuring general 3D graphics aptitude, but it's hard to beat full retail video games for judging gaming performance. Far Cry 5 and Rise of the Tomb Raider are both modern, high-fidelity titles with built-in benchmarks that illustrate how a system handles real-world video games at various settings.

These games are run on both the moderate and the maximum graphics-quality presets in the benchmarking utility. (Those presets are Normal and Ultra for Far Cry 5, Medium and Very High for Rise of the Tomb Raider.) We test by default at 1080p if possible and (if the laptop's native display resolution is higher or lower) at the screen's native resolution to judge performance for a given system.

These results are also provided in frames per second. Far Cry 5 is a DirectX 11-based game, while Rise of the Tomb Raider can be flipped to DirectX 12 mode, which we do for this benchmark.

Battery Life Testing

Finally, we perform a video-playback-based battery rundown test, which supports all operating systems, to estimate in a relative sense how long the laptop will last away from a power outlet. Our rundown test involves playing a looped 720p version of Tears of Steel (mentioned earlier), saved on the laptop's storage drive in the MP4 format. If the file will not fit on the system's local storage, we run the video from an SD card or a USB memory key.

Before we start the test, we turn on the system's power-saving mode, crank down the brightness of the screen to 50 percent, boost the audio volume to 100 percent, and disable adaptive screen brightness. Wireless radios, keyboard backlighting, and any case lighting are turned off. We then start the test and run it for as long as a fully charged battery lasts, usually to the point that the system hibernates at under 5 percent battery capacity (as prescribed in the Critical Battery Action we also set in Power Options). If the laptop has more than one battery, we will perform a separate, second rundown test with both batteries installed.

Evaluating battery performance is difficult, since results can vary widely based on the types of tasks performed. (Heavy gaming off the battery, for one thing, will deliver much shorter times.) But in general, we consider a system to offer all-day computing if it lasts more than eight hours on our battery-rundown test.

Special Cases: macOS, Chromebooks, Mobile Workstations

We don't run all of the above tests on every laptop. We only run Far Cry 5 and Rise of the Tomb Raider on laptops that are specifically designed for gaming, equipped with a dedicated graphics processor. And we don't use PCMark, 3DMark, or Superposition for testing Apple laptops, since these tests have no macOS version. To evaluate some specialized subsets of laptops, such as workstations, Chromebooks, and ARM-based systems, we supplement our standard tests.

Chromebooks

With Chromebooks, for instance, of the above tests we perform only the battery rundown test, since that's the only one that's compatible with Chrome OS. Instead of the 720p file, we use a different, lower-resolution source file (a DVD rip of the full Lord of the Rings trilogy, looped) stored (if possible) on the Chromebook's internal storage.

We then run the benchmarks CrXPRT and WebXPRT, from Principled Technologies, to help us make comparisons among Chromebooks. (Because WebXPRT is a web-based test that can run on nearly any PC, we also use it to test laptops with ARM processors.) These are single-click tests without settings to tweak, and they report back proprietary scores that are meaningful only relative to one another.

Workstation Laptops

With workstation laptops, we run all of the above tests and supplement them with a few workstation-specific measures. These specialized tests include the multimedia rendering tool POV-Ray (for a ray-tracing simulation). We also run the SPECviewperf 13 suite, loading three "viewsets" for the apps Creo, Maya, and SolidWorks, to gauge how the workstation machine handles the manipulation of relevant files in these three seminal workstation programs. The POV-Ray results are reported as time to completion of the test task, and the SPECviewperf results are reported in frames per second.

Posted in: Tips