Beyond the Headlines: Don’t be fooled by misleading security tests

By Alejandro Borgia, senior director, product management, Symantec Corporation

For customers looking for the best security technology for their needs, it’s critical to have access to reliable and meaningful information so they can make smart decisions to stay protected in today’s complex threat landscape.  As an industry, we face a challenge in that testing security products is a highly technical and time-consuming process, and there are shortcomings in the approach of some tests that result in data that is misleading at best. 

Last week, PC Magazine published an article titled “Microsoft Outperforms Symantec in Antivirus Test,” which shared the results of a recent on-demand file-scanning test performed by AV-Comparatives. Although the article explains a shortcoming of the test related to the treatment of false positives, a much bigger issue with the test, and others like it, is that the cited detection rates are misleading and not representative of real-world product efficacy. These types of file scanning tests are run in artificial environments that cripple all modern protection features. The latest Norton security products (and some other security products) employ multiple, complementary protection technologies in order to block threats. Classic fingerprint-based protection, network intrusion prevention, behavior-based protection, and Insight reputation-based security are four distinct and highly complex security layers in Norton products which all work together to detect and block malicious attacks before they can reach an end user.

Think about this analogy – these layers of protection are like the multiple layers of protection found in modern automobiles. Today’s cars don’t just have lap seat belts—they have dozens of features and systems that work together to maximize passenger safety, including shoulder harnesses, head restraints, multiple airbags, crumple zones and more. Auto safety tests attempt to simulate real-life crashes as closely as possible, and measure the impact on crash test dummies. In contrast, imagine a test that evaluated a car’s safety by first disabling every safety feature except for the car’s lap seatbelts — the tester removed shoulder harnesses, disabled all the airbags, etc.  Such a test may conclude that the car was extremely dangerous to drive. Yet such a conclusion would be entirely flawed since it evaluated only a single safety feature, explicitly ignoring all of the protection afforded by the car’s overall safety system.

Similarly, testing security products with just a file scan, as seen in the recent AV-Comparatives report and similar types of tests, is misleading. Such a testing approach does not accurately represent the real-world threat conditions seen today – and also does not accurately represent the level of protection provided by a security product. Putting forward conclusions from such flawed testing creates confusion and does a disservice to consumers.

Symantec has long been an advocate of independent “whole product” or “real-world” tests that most closely represent the threat environment and utilize all of the proactive technologies provided with a product.  Symantec consistently ranks at the top of real-world tests performed by independent testers. The Norton products have received recognition for industry-leading protection in real-world, unsponsored tests conducted by independent testing labs such as AV-TEST and Dennis Technology Labs.  These labs evaluate each product in realistic infection scenarios that typical users might experience.

We look forward to the day when all published tests are real-world tests. In the meantime, readers need to beware of artificial tests that show misleading product comparisons.

[edit: changed links to point to the company websites]