MRG Effitas Time to Detect & Remediate Assessment Project Q2 2013

[ande]

The two metrics MRG Effitas believes are most relevant are:

• The time taken to protect a system. There are two sub elements to this metric:
  • Time to detect
  • Time to remediate
• Assessment of data exfiltration. There are three sub elements to this metric:
  • Determination as to whether a data breach has occurred or not (uncorrupted data)
  • Calculation of what was breached
  • Measurement of how long the breach occurred

The Purpose of this Assessment:
It should be taken as a given that no security solution can detect 100% of in the wild malware (malware that is active at live URLs). Exhaustive testing has shown conclusively that all security products and solutions get bypassed at some point and therefore, it is sensible to investigate product efficacy under this compromised condition.
In this report, we are assessing the ability of a cohort of security products to prevent an endpoint from being infected by live, ITW malware. If the system is infected, we measure the time the products take to detect the infection, up to a maximum of twenty four hours.
In order that testing maps closely to real world scenarios, 300 early life malware samples are used, 275 of these attempt to infect the system from live URLs via Internet Explorer and 25 via a USB stick.


Security Applications Tested:

• Avast Internet Security 8.0
• AVG Internet Security 2013-2014
• BitDefender Internet Security 2013-2014
• Emsisoft Anti-Malware 8.0-8.1
• ESET Smart Security 6.0
• Kaspersky Internet Security 2013-2014
• Malwarebytes Anti-Malware 1.74-1.75
• McAfee Internet Security 2013
• Microsoft Security Essentials 4.3
• Panda Internet Security 2013-2014
• SourceFire Immunet Protect Plus 3.1
• SUPERAntiSpyware 5.5-5.6
• Symantec Norton Internet Security 2013-2014
• ThreatTrack Vipre Internet Security 2013-2014
• Trend Micro Titanium Internet Security 2013
• Trusteer Rapport Emerald Build 1302.61
• Webroot SecureAnywhere Internet Security Plus 2013

Methodology Used in the Assessment:

1. Windows 7 Ultimate Service Pack 1 64 bit operating systemi is installed on a virtual machine and all updates are applied and third party applications installed and updated according to our “Average Endpoint Specification”i
2. An image of the operating system is created.
3. A clone of the imaged systems is made for each of the security applications to be used in the test.
4. An individual security application is installed using default settingsiii on each of the systems created in 3 and then, where applicable, is updated.
5. A clone of the system as it is at the end of 4 is created.
6. Each live URL test is conducted by:
   a. Downloading a single malicious binary from its native URL using Internet Explorer to the desktop, closing Internet Explorer and then executing the binary
   b. The security application blocks the URL where the malicious binary is located.
   c. The security application detects and blocks the malicious binary whilst it is being downloaded to the desktop.
   d. The security application detects the malicious binary when it is executed according to the following criteria:
   i. It identifies the binary as being malicious and either automatically blocks it or postpones its execution and warns the user that the file is malicious and awaits user input.
7. The system under test deemed to have been infected by the following criteria:
   a. The security application fails to detect or block the binary at any stage in 6 and allows it to be executed.
8. Testing on infected systems continues for twenty four hours by the following process:
   a. A Fast/Quick scan is performed every 30 minutes to give the security application an opportunity to detect the infection.
9. Remediation performance of an application is classified in three stages determined by contrast to the clean system created in stage 4. There are three classifications:
   a. “A” = all traces of the infection were removed, “B” = the malicious code was removed, but non malicious remnants, registry entries etc. were left “C” = malicious code was not removed.
10. Testing is conducted with all systems having internet access.
11. Each individual test for each security application is conducted from a unique IP address.
12. All security applications are fully functional unregistered versions or versions registered anonymously, with no connection to MRG Effitas.
13. All testing was conducted during Q2 & Q3 2013.

Samples Used:

Test Results:

The graph below shows the average time to detect for all security applications under test:

The graph below shows the remediation level for those applications which did not detect / block the malware on initial exposure:

Note: A = full remediation, B = remediation by removal of malicious code only, C = failure to remediate.

Conclusion / Notes:

• Avast, BitDefender, Emsisoft, Kaspersky, TrendMicro, Trusteer and Webroot detected / blocked all 250 malicious binaries and so protected the system under test from infection.
• ESET, Malwarebytes and Symantec removed all traces of the infection, including any dropped files, temporary files and registry entries.
• It should be noted that both Malwarebytes Anti-Malware and SUPERAntiSpyware are complementary tools, however both tools are designed to protect against threats used in this test.
• SuperAntiSpyware was the only application which failed to detect 100% of infections within the 24 hour period.
• SUPERAntiSpyware was the only application under test which failed to detect all samples inside 24h, it missed a total of 28 samples.
• This assessment focuses on core financial malware only. ZeuS and Torpig are the two oldest pieces of Financial Mlware that are still active. (Torpig is also known as Sinowal).

i: AES includes Adobe Flash, Reader, Java, Microsoft Office 2007 & Mozilla Firefox, all fully updated.
ii: During the installation of the security application, if an option to detect PUAs is given, it is selected.


Source
PDF