Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Risk

7/19/2019
10:00 AM
Brian Monkman
Brian Monkman
Commentary
Connect Directly
LinkedIn
RSS
E-Mail vvv
100%
0%

The Problem with Proprietary Testing: NSS Labs vs. CrowdStrike

Why apples-to-apples performance tests are the only way to accurately gauge the impact of network security products and solutions.

When a company is researching options for network security tools, it needs to be able to weigh a number of factors, including cost, effectiveness, compatibility, integration with current or third-party tools and platforms, and — perhaps most important — impact on network performance. To evaluate all options, however, there has to be some way to compare different solutions accurately. The recent legal conflict between CrowdStrike and NSS Labs illustrates why it is such a challenge for organizations to evaluate different vendors and products.

NSS Labs and CrowdStrike announced in late May that after two years, they had reached a confidential settlement to end their legal battle. CrowdStrike initially sued NSS Labs in February 2017 over negative test results that were published in an NSS Labs endpoint protection report. NSS Labs gave the CrowdStrike Falcon platform a "Caution" rating. CrowdStrike, however, maintained that the test was faulty. In late 2018, NSS Labs fought back with a lawsuit of its own against CrowdStrike, Symantec, ESET, and the Anti-Malware Testing Standards Organization (AMTSO), claiming that the vendors and AMTSO conspired to prevent NSS Labs from testing their products.

Following the settlement, NSS Labs retracted its 2017 assessment of CrowdStrike's Falcon platform and issued a corrective statement, saying that "NSS's testing of the CrowdStrike Falcon platform was incomplete and the product was not properly configured with prevention capabilities enabled. In addition to the results having already been acknowledged as partially incomplete, we now acknowledge they are not accurate and confirm that they do not meet our standards for publication."

The CrowdStrike-NSS altercation is only one legal battle between a testing lab and a vendor. The NSS Labs lawsuit against Symantec, ESET, and AMTSO is still ongoing, even in the wake of the settlement with CrowdStrike, and it is likely that similar lawsuits are in the pipeline.

What's Wrong with Network Security Testing
The situation involving NSS Labs and CrowdStrike is a perfect example of what's wrong with network security testing. In a Dark Reading column earlier this year, I described network security testing as the Wild West, noting that proprietary testing methods conducted under uniquely optimized conditions create a chaotic scenario in which everyone plays by their own rules and customers are left struggling to sort it all out.

The fact that NSS Labs retracted its rating of CrowdStrike's Falcon platform highlights one of the primary issues with closed or proprietary network security testing standards. Without visibility into the testing protocols and standards used, there is no way for organizations to objectively determine whether an NSS Labs assessment is right or wrong.

The flip side of the coin is that closed and proprietary testing standards also create the potential for vendors to game the system or gain an advantage over rival companies. Vendors frequently impose conditions on how their products can be tested and demand that testing labs design environments and tests that focus on areas in which the product excels, minimizing scenarios in which the product does not perform well.

In the end, however, whether the testing lab makes a mistake or misconfigures the product, or whether the vendor influences the testing unfairly to make its product look better, the result is that the testing can't be trusted. And untrustworthy testing is useless when organizations are trying to compare various products to make a purchasing decision.

The Importance of Open Security Testing Standards
Testing labs don't want to be in the position of trying to negotiate with vendors for the privilege of testing their products, nor do they want to cave to demands that make those products look better. Vendors don't want to have their products tested in scenarios that don't pit them against rivals on an even playing field. Customers don't want test results that can't be trusted, or assessments with questionable accuracy that can't be verified because the testing methodology is proprietary. The answer is to adopt open network security testing standards.

NetSecOPEN (like AMTSO, a vendor-led organization) was formed in 2017 to address issues and challenges related to the current proprietary testing environment. We believe that the network and security industries must go beyond simply validating test results and, instead, establish new tests that are open, transparent, and created through a cooperative, collaborative effort. Ultimately, apples-to-apples performance tests that accurately portray the effectiveness and impact of network security products on network performance are essential, and the result of such testing is a win-win-win for testing labs, vendors, and — most of all — customers.

Related Content:

 

Black Hat USA returns to Las Vegas with hands-on technical Trainings, cutting-edge Briefings, Arsenal open-source tool demonstrations, top-tier security solutions, and service providers in the Business Hall. Click for information on the conference and to register.

Brian Monkman is executive director of NetSecOPEN, a nonprofit, membership-driven organization with a goal of developing open standards for testing network security products. A 25-year network security veteran, he has extensive experience in technical support, sales ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Oldest First  |  Newest First  |  Threaded View
username007
50%
50%
username007,
User Rank: Strategist
7/19/2019 | 2:16:06 PM
Agreed bt don't trust
As outlined in this article it is necessary for secuity "things" to be open and transparent. Hiding behind proprietary testing methods diminishes the value of those results. To make this more fun the author of this post included a link to a resource that is a redirect from mimecast over to the site claiming to be more transparent. Yay for being hypocrytical. 
REISEN1955
100%
0%
REISEN1955,
User Rank: Ninja
7/19/2019 | 2:24:44 PM
Re: Agreed bt don't trust
Gartner - I have never liked their power as well, a bunch of Greenwich CT young pups running stupid Magic Quadrant campaigns that have no bearing in the real world.  Snotty types. 
tdsan
100%
0%
tdsan,
User Rank: Ninja
7/19/2019 | 3:44:30 PM
Collaboration with the vendor is key
We believe that the network and security industries must go beyond simply validating test results and, instead, establish new tests that are open, transparent, and created through a cooperative, collaborative effort. 

I believe this section is the most import aspect of testing - collaboration. By collaborating with the testing teams and the OEM, we can alleviate some of the misconfigurations associated with the test.

T
Mobile Banking Malware Up 50% in First Half of 2019
Kelly Sheridan, Staff Editor, Dark Reading,  1/17/2020
7 Tips for Infosec Pros Considering A Lateral Career Move
Kelly Sheridan, Staff Editor, Dark Reading,  1/21/2020
For Mismanaged SOCs, The Price Is Not Right
Kelly Sheridan, Staff Editor, Dark Reading,  1/22/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment:   It's a PEN test of our cloud security.
Current Issue
IT 2020: A Look Ahead
Are you ready for the critical changes that will occur in 2020? We've compiled editor insights from the best of our network (Dark Reading, Data Center Knowledge, InformationWeek, ITPro Today and Network Computing) to deliver to you a look at the trends, technologies, and threats that are emerging in the coming year. Download it today!
Flash Poll
How Enterprises are Attacking the Cybersecurity Problem
How Enterprises are Attacking the Cybersecurity Problem
Organizations have invested in a sweeping array of security technologies to address challenges associated with the growing number of cybersecurity attacks. However, the complexity involved in managing these technologies is emerging as a major problem. Read this report to find out what your peers biggest security challenges are and the technologies they are using to address them.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2019-3697
PUBLISHED: 2020-01-24
UNIX Symbolic Link (Symlink) Following vulnerability in the packaging of gnump3d in openSUSE Leap 15.1 allows local attackers to escalate from user gnump3d to root. This issue affects: openSUSE Leap 15.1 gnump3d version 3.0-lp151.2.1 and prior versions.
CVE-2019-3694
PUBLISHED: 2020-01-24
A Symbolic Link (Symlink) Following vulnerability in the packaging of munin in openSUSE Factory, Leap 15.1 allows local attackers to escalate from user munin to root. This issue affects: openSUSE Factory munin version 2.0.49-4.2 and prior versions. openSUSE Leap 15.1 munin version 2.0.40-lp151.1.1 a...
CVE-2019-3693
PUBLISHED: 2020-01-24
A symlink following vulnerability in the packaging of mailman in SUSE SUSE Linux Enterprise Server 11, SUSE Linux Enterprise Server 12; openSUSE Leap 15.1 allowed local attackers to escalate their privileges from user wwwrun to root. Additionally arbitrary files could be changed to group mailman. Th...
CVE-2019-3687
PUBLISHED: 2020-01-24
The permission package in SUSE SUSE Linux Enterprise Server allowed all local users to run dumpcap in the "easy" permission profile and sniff network traffic. This issue affects: SUSE SUSE Linux Enterprise Server permissions versions starting from 85c83fef7e017f8ab7f8602d3163786d57344439 t...
CVE-2019-3692
PUBLISHED: 2020-01-24
The packaging of inn on SUSE SUSE Linux Enterprise Server 11; openSUSE Factory, Leap 15.1 allows local attackers to escalate from user inn to root via symlink attacks. This issue affects: SUSE SUSE Linux Enterprise Server 11 inn version 2.4.2-170.21.3.1 and prior versions. openSUSE Factory inn versi...