Security Testing Houses: Know the Truth!

[Petrovic]

My previous employer – not to name any names – was one who sold millions upon millions of endpoints each year. I was a part of the Office of the CTO, and my main focus was the security strategy for these endpoints. One of my many challenges was to find and test the very best in endpoint security products – technology that could prevent commodity threats from executing on the endpoint, and provide advanced threat protection (ATP).

 

While conducting my search, I was shocked to discover that I could not rely on security testing house results, because most security vendors scored 100% in tests, which those of us who work in the industry know is not possible. If it were, then everyone could take their ball home because the game would be over. Those impossible testing house results were completely at odds with the results I obtained in my independent testing, but more on that later. 

 

Fast forward 18 months and I found myself at Cylance – in large part due to the incredible results I’d obtained from Cylance’s products during my testing. My first call of duty after joining Cylance was to engage with the testing houses that publicly test security software. I will not discuss the names of any of these testing houses, but what I found in some of these places is astonishing to say the least.

 

Let me start by saying that in some, I found examples of positivity and growth, usually coupled with the willingness to change and work with new security technologies such as machine learning which have produced a paradigm shift in the security industry of late. In others, unfortunately, I encountered nothing but fraud, bias, software piracy and extortion. I know these are strong accusations, but I call it like I see it.

 

Why It’s All About the Benjamins

As with most things in life, when you dig a little deeper, it’s all about the Benjamins. Some testing houses are scared to change, because change will impact their revenue streams. They are afraid that testing and showing the true results (or lack of results) from the most well-known AV vendors will result in those vendors discontinuing their relationships with said testing house, resulting in loss of revenue. This is, in effect, fraud. Some of these testing houses refuse to show poor efficacy results so as not to impact their bottom line, and in that refusal, perpetrate a fraud against you, the consumer of that security testing report. It’s astonishing that these business practices of courting security vendors to reflect high efficacy results in exchange for payment are alive and well. To me, this is completely unacceptable. 

 

The defrauding and manipulation of the public with these tests also stems from vendors who pay so that their test results will show 100% efficacy. These reports not only deceive the buyer, but they also set up impossible standards for the entire security industry. Repeat after me: there is no such thing as a 100% efficacy rate in security. There is no single silver bullet that will provide total, unbreachable protection against every type of malware in every situation - ever!

 

If a vendor does get 100% on an anti-malware test, they either:

1) Paid for perfection, bribing the testing house to hide the negative results of their tests.
2) Tested using a statistically invalid sample set of malware like 100 samples.
3) Tested with samples not in any way reflective of real world attacks.
4) All of the above.

 

Uncovering Bias and Extortion in AV Testing Houses

Now, it goes without saying that there is always a bias of some sort when it comes to technology. If humans are involved in the testing, we will always have a percentage of bias. Some of the time that’s due to the relationship between the vendor and tester, and in other cases it’s the testing houses’ preference for a certain type of technology – even technology that is now outdated. But I get it. There’s something very comforting about the familiar. For example, I have a bias for certain car manufacturers because of my perception of them (right or wrong) as being good quality, reliable brands. Maybe there are better, newer brands of automobile out there, but because I’ve been burned a few times, I’m going to stick to what I know, thank you very much. So I always buy the same brand of car, the brand I’m used to.

 

A certain testing house pirated Cylance’s software at one point, because they believed we had unproven marketing claims. They wanted Cylance to come over to their table and pay for testing – at their testing house. They felt that the efficacy rates reported by users of our next-generation, artificially intelligent products were unproven because Cylance didn't use so-called public testing methods, which are (surprise, surprise) biased… because they can be bought.

 

Well, I have some news for you: Cylance will never ‘pay to play.’ In fact, we invite you to Test For Yourself. Personally speaking, I did so myself while at my former employer. The jaw-dropping results I obtained stood in stark contrast to the results ‘created’ at most testing houses.

 

The next ‘big whopper’ moment was when we were approached by a testing house to be included in their own analyst testing. Notwithstanding the atrocious testing methodologies commonly utilized by this place, Cylance was asked to personally pay for this testing in order to be granted various ‘rights’, such as a choice of the types of malware used in the test (which real life does not give us), an opportunity to challenge the testing results (excuse me?) and also, worst of all, to have edit rights to the final article. If we did not give in to this extortion, then the Cylance product would be forcibly tested by this testing house, purposely using bare-bones policy settings that would make our product inert. Those skewed results would then be publicly posted.

 

Once again, this is extortion, plain and simple. Buying test settings does not give the consumer accurate results. Picking and choosing which malware is to be used in the test does not help the end user. Having a glowingly positive EDITED article written about your test results does not help your client when a piece of ransomware you ‘chose’ not to test against breaks through your inadequate traditional defenses and lays waste to their company’s systems.

AGAIN I say: Test For Yourself. Res Ipsa Loquitur! (“The thing speaks for itself!”)

Article source

[straycat19]

This is a blatant advertisement for Cylance Threat Testing Framework!  They want to sell you their software and malware databases so you can do your own testing, so do you really think they want you to believe that any of the testing sites that provide this information freely are any good.  

 

NOTE TO POSTER:  Please read the crap before you post it.

[Petrovic]

20 minutes ago, straycat19 said:

NOTE TO POSTER:  Please read the crap before you post it.

 

Note: Their perspective may be different than yours

[Ghazi]

Yes in the last two Paragraphs the tilt is towards cylance but it is totally true that no product can guarantee a 100% result against all the threats released each day that most of us see in these tests. Also I( agree with author that there is money and bias involved in these tests

[Holmes]

I call bullshit cylance is full of shit I agree with stray.  Please dont fall for there bullshit.

[steven36]

[pc71520]

Cylance Marketing and nothing more...

[steven36]

One person debates the topic and the topic starter debates  them back and the rest say the same thing as the  2nd and 3rd post  witch they  all ready said it  . Saying the same thing over and over again is not adding any value to the post . This is not a poll topic and no one cares . If you agree with the person there is a thank you button .

 

All it would take is someone too build a 0day  like Stuxnet they could of wiped the internet out with it  .Its already been proved they could, we just got lucky it was designed to attack a certain kind of computer box . Because everywhere was infected with it and the Governments of the world wont even talk about it. Meaning if something bad happen  they would act like it dont exist .0day means  no Anti-malware  or Antivirus  can detect it yet even. 

 

The whole system  is rigged  when ever researcher  finds  something  and report it they have 90 days to patch it before its made public  the whole world could  be infected by then . Last patch Tuesday Microsoft patch  a hole in there browser that spread malware they knew about for a year even,, the exploit evaded security researchers computers even .   . Now they find a big mtm hole in Firefox trough add-ons  atuo updates that  Tor Browser  patched and Firefox is  fixing to patch  .For a longtime  i have auto update checking off  in mine the only time mine checks is if i check myself  .

 

The internet is full of malware ads  on just about all sites  were they be hidden  hijackers  were if you click on a page it will open a new  tab  and the adblockers and script blockers cant stop them unless you're adblocker like UBO  has them blacklisted in its list the whole site or you blacklist them yourself. Popup blockers can stop most of them better  than adblocks can  and  it gets worse all the time.