MRG Effitas Time to Detect Assessment Q2 & Q3 2013

[ande]

The two metrics MRG Effitas believes are most relevant are:

• The time taken to protect a system. There are two sub elements to this metric:
  • Time to detect
  • Time to remediate
• Assessment of data exfiltration. There are three sub elements to this metric:
  • Determination as to whether a data breach has occurred or not (uncorrupted data)
  • Calculation of what was breached
  • Measurement of how long the breach occurred

The Purpose of this Assessment:
It should be taken as a given that no security solution can detect 100% of in the wild malware (malware that is active at live URLs). Exhaustive testing has shown conclusively that all security products and solutions get bypassed at some point and therefore, it is sensible to investigate product efficacy under this compromised condition.
In this report, we are assessing the ability of a cohort of security products to prevent an endpoint from being infected by live, ITW malware. If the system is infected, we measure the time the products take to detect the infection, up to a maximum of twenty four hours.
In order that testing maps closely to real world scenarios, 300 early life malware samples are used, 275 of these attempt to infect the system from live URLs via Internet Explorer and 25 via a USB stick.


Security Applications Tested:

• Avast Internet Security 8.0
• AVG Internet Security 2013-2014
• BitDefender Internet Security 2013-2014
• Emsisoft Anti-Malware 8.0-8.1
• ESET Smart Security 6.0
• Kaspersky Internet Security 2013-2014
• Malwarebytes Anti-Malware 1.74-1.75
• McAfee Internet Security 2013
• Microsoft Security Essentials 4.3
• Panda Internet Security 2013-2014
• SoftSphere DefenseWall 3.22
• SourceFire Immunet Protect Plus 3.1
• SUPERAntiSpyware 5.5-5.6
• Symantec Norton Internet Security 2013-2014
• ThreatTrack Vipre Internet Security 2013-2014
• Trend Micro Titanium Internet Security 2013
• Webroot SecureAnywhere Internet Security Plus 2013

Methodology Used in the Assessment:

1. Windows 7 Ultimate Service Pack 1 64 bit operating systemi is installed on a virtual machine and all updates are applied and third party applications installed and updated according to our “Average Endpoint Specification”ii
2. An image of the operating system is created.
3. A clone of the imaged systems is made for each of the security applications to be used in the test.
4. An individual security application is installed using default settingsiii on each of the systems created in 3 and then, where applicable, is updated.
5. A clone of the system as it is at the end of 4 is created.
6. Each live URL test is conducted by:
   a. Downloading a single malicious binary from its native URL using Internet Explorer to the desktop, closing Internet Explorer and then executing the binary
   b. The security application blocks the URL where the malicious binary is located.
   c. The security application detects and blocks the malicious binary whilst it is being downloaded to the desktop.
   d. The security application detects the malicious binary when it is executed according to the following criteria:
   i. It identifies the binary as being malicious and either automatically blocks it or postpones its execution and warns the user that the file is malicious and awaits user input.
   
7. Each USB infection test is conducted by:
   a. Downloading a single malicious binary from its native URL using a system out of the testing harness, then copying it to a USB stick.
   b. The USB stick is inserted in to the system under test and copied to the desktop using explorer and executed.
   c. The system under test is deemed to have been initially protected by the following criteria:
   d. The security application detected the malicious binary when the USB stick was inserted.
   e. The security application detected the malicious binary when it was copied to the desktop.
   f. The security application detects the malicious binary when it is executed according to the following criteria:
   i. It identifies the binary as being malicious and either automatically blocks it or postpones its execution and warns the user that the file is malicious and awaits user input.
8. The system under test deemed to have been infected by the following criteria:
   a. The security application fails to detect or block the binary at any stage in 6 or 7 and allows it to be executed.
9. Testing on infected systems continues for twenty four hours by the following process:
   a. A Fast/Quick scan is performed every 30 minutes to give the security application an opportunity to detect the infection.
10. Testing is conducted with all systems having internet access.
11. Each individual test for each security application is conducted from a unique IP address.
12. All security applications are fully functional unregistered versions or versions registered anonymously, with no connection to MRG Effitas.
13. All testing was conducted during Q2 & Q3 2013.

Samples Used:

Infection Vectors:

Test Results:

The graph below shows the average time to detect for all security applications under test:

Conclusion / Notes:

• Avast, BitDefender, Emsisoft, Kaspersky and SoftSphere detected / blocked all 300 malicious binaries and so protected the system under test from infection.
• It should be noted that both Malwarebytes Anti-Malware and SUPERAntiSpyware are complementary tools, however both tools are designed to protect against threats used in this test.
• SuperAntiSpyware was the only application which failed to detect 100% of infections within the 24 hour period.
• Whilst it is true that vast majority of infections occur when users visit malicious URLs, removable media devices also pose as a very effective delivery system for malware. We have seen this with some high profile pieces of malware in the past such as the infamous Conficker worm and various APTs. It is because of this that we chose to introduce the second infection vector to this test.
  

i DefenseWall is tested on Windows 7 32 bit
ii AES includes Adobe Flash, Reader, Java, Microsoft Office 2007 & Mozilla Firefox, all fully updated.
iii During the installation of the security application, if an option to detect PUAs is given, it is selected.


Source
PDF