AV-Comparatives File Detection Test - September 2013

File Detection Tests
 

Aims of the test and target audience The File Detection Test assesses the ability of antivirus programs to detect malicious files on a system. Whilst the test only assesses one antimalware feature of the programs, this feature is important for a security solution. This is because it can identify malware attacks from sources other than the Internet, and detect malicious files already present on the system
.

Test Procedure
 

We install each antivirus program on its own physical PC, and update the signatures. The malware sample files are then scanned using the program’s standard scanning procedure, and the number of detections is recorded. The PCs remain connected to the Internet during the test, meaning that the security programs can use any cloud features provided by their manufacturers.

Typically, more than 100,000 malware samples are used for the test. These are prevalent malicious files of all types that have been recently collected, i.e. within a period of a few weeks or months before the test is performed.

 

To ensure that the tested programs do not simply identify all unknown files as malware, a false-positive test is also conducted for every File Detection Test. Any programs with a high rate of false positives will have their award reduced by one level. For example, a program which has a detection rate high enough to earn an Advanced Award will be demoted to a Standard Award if it produces a high rate of false alarms.


Detection and False Alarm Results:

 

1.png

1.png


Antivirus Awards; (Based on detection rates and false alarms)


  Advanced+ - F-Secure, Kaspersky, Bitdefender, Emsisoft, Bullguard, Fortinet, Sophos, Trend Micro.

Advanced - Avira, G-Data, McAfee, Panda, eScan, Vipre, AVG, ESET, Avast.

Standard - AhnLab

Tested - Symantec

 

Here's the source: http://www.av-comparatives.org/wp-content/uploads/2013/09/avc_fdt_201309_en.pdf

 

 

 

 

 

 

 

-Symantec results are pretty surprising.. :(

AV-Comparatives requires products to submit to all tests.  Because Symantec declines to participate in the File Detection test, it is prohibited from joining the Real-World Protection test, a test in which Norton has always performed extremely well in the past.  So what does AV-Comparatives do?  They put Norton into the File Detection test anyway but do not enter Norton into the Real-World Test.  I guess when you make the rules, you are allowed to break the rules.

File Detection Tests
 

Aims of the test and target audience The File Detection Test assesses the ability of antivirus programs to detect malicious files on a system. Whilst the test only assesses one antimalware feature of the programs, this feature is important for a security solution. This is because it can identify malware attacks from sources other than the Internet, and detect malicious files already present on the system
.

Test Procedure
 

We install each antivirus program on its own physical PC, and update the signatures. The malware sample files are then scanned using the program’s standard scanning procedure, and the number of detections is recorded. The PCs remain connected to the Internet during the test, meaning that the security programs can use any cloud features provided by their manufacturers.

Typically, more than 100,000 malware samples are used for the test. These are prevalent malicious files of all types that have been recently collected, i.e. within a period of a few weeks or months before the test is performed.

 

To ensure that the tested programs do not simply identify all unknown files as malware, a false-positive test is also conducted for every File Detection Test. Any programs with a high rate of false positives will have their award reduced by one level. For example, a program which has a detection rate high enough to earn an Advanced Award will be demoted to a Standard Award if it produces a high rate of false alarms.


Detection and False Alarm Results:

 

1.png

1.png


Antivirus Awards; (Based on detection rates and false alarms)


  Advanced+ - F-Secure, Kaspersky, Bitdefender, Emsisoft, Bullguard, Fortinet, Sophos, Trend Micro.

Advanced - Avira, G-Data, McAfee, Panda, eScan, Vipre, AVG, ESET, Avast.

Standard - AhnLab

Tested - Symantec

 

Here's the source: http://www.av-comparatives.org/wp-content/uploads/2013/09/avc_fdt_201309_en.pdf

 

 

 

 

 

 

 

-Symantec results are pretty surprising.. :(

Here is an article on the latest results of the Dennis Technology Labs test, which looks at how well a security program actually protects a computer from becoming infected with malware in the first place.  Rather than relying solely on file detection, this test allows a program to use all of its components, such as SONAR and Download Intelligence (reputation), in a realistic attack scenario.  In the real world, an attack consists of several stages and this test allows each program an opportunity to block the malware at any point in that process using all of its defenses rather than just one.

 

http://securitywatch.pcmag.com/security-software/316877-mcafee-tanks-independent-av-test

Pretty good *real world* result, for Norton. :smileyhappy:

'The question does come up—why couldn't they adjust the product so it does what they think it should and also passes all the tests? Indeed, rumor has it that some vendors assign as many as a dozen engineers specifically to the task of ensuring good test scores. My Symantec contacts say doing so would just encourage retention of what they consider to be bad testing styles; they don't want to be enablers.' http://www.pcmag.com/article2/0,2817,2424118,00.asp

 

'Rather than focus on making its own antivirus the best in the business, Stewart said Microsoft was "doing everything we can to protect against real threats" and passing data on those threats to antivirus makers, so multiple parties can target the problems.

 

"It’s not as efficient to have one kind of weapon," she said. "Like anything you must have that diversity. It’s a weakness to just have one."

 

Previously, Microsoft would spend resources trying to improve Security Essentials' performance in tests. "We used to have part of our team directed towards predicting test results and figuring out what might be in someone’s test. There’s always a cost to that," she said. "If they are doing that work they are not looking at those threats that are affecting our customers. We always felt that was wrong. There’s something not right about that – we’re not doing the best job for our customers."

 

The company decided to stop that practice and put its effort elsewhere.
"We put half of those people on focusing on what we call prevalent threats. We developed this new telemetry to look for emerging threats - sort of an early notification system that new threats were emerging. We had this group of folks start focusing on those threats and we saw that it increased our protection service level for our customers."

 

In practice, it means Microsoft is focusing on tracking emerging threats and sharing that data within the security industry, saying that's a more meaningful way to protect customers.' http://www.pcpro.co.uk/news/security/384394/microsoft-security-essentials-is-designed-to-be-bottom-of-the-antivirus-rankings


SendOfJive wrote:

 

AV-Comparatives requires products to submit to all tests.  Because Symantec declines to participate in the File Detection test, it is prohibited from joining the Real-World Protection test, a test in which Norton has always performed extremely well in the past.  So what does AV-Comparatives do?  They put Norton into the File Detection test anyway but do not enter Norton into the Real-World Test.  I guess when you make the rules, you are allowed to break the rules.


Hi SendOfJive

 

AV-Comparatives do list that testing scenario in their Sources of Income information:

 

Sources of Income 

 

In 2003, Andreas Clementi was hoping to finance the project with donations from users. Unfortunately, this was not viable, as only two or three users made significant donations, despite an appeal. Other sources of income had to be found to keep the project going. There was much debate as to how to support AV-Comparatives without compromising its most important quality, namely its neutrality. Payment must not be allowed to have any influence on test results.

 

The solution actually turned out to be very simple: if all manufacturers pay the same fee in order for their product to be tested, none of them can be advantaged or disadvantaged. In some cases it can happen that a highly demanded – either by users or magazines - vendor is tested even if it did not apply for it. In this case, the costs will be covered either by the magazines or by other independent parties, which requested the results. [...]

 

Symantec made the decision to not participate in the File Detection Tests. Based on the above funding model, it's not inconceivable that Symantec's competitors may have funded the testing to find out why Symantec didn't want their products tested in the File Detection Test. The fact that Symantec can't beat Microsoft in File Detection tests has the potential to be used/misused for competitive advantage.

 

Given that, the simple solution is for Symantec to voluntarily rejoin the AV-Comparatives testing in 2014. This will eliminate the lingering doubt about the effectiveness of Symantec's products. Consumers will actually be able to see how well the Norton products perform in the AV-Comparatives Real-World Protection Tests.

 

 

 


elsewhere wrote:
Symantec made the decision to not participate in the File Detection Tests. Based on the above funding model, it's not inconceivable that Symantec's competitors may have funded the testing to find out why Symantec didn't want their products tested in the File Detection Test. The fact that Symantec can't beat Microsoft in File Detection tests has the potential to be used/misused for competitive advantage.
Given that, the simple solution is for Symantec to voluntarily rejoin the AV-Comparatives testing in 2014. This will eliminate the lingering doubt about the effectiveness of Symantec's products. Consumers will actually be able to see how well the Norton products perform in the AV-Comparatives Real-World Protection Tests.

Symantec elected not to participate in the file detection test, knowing that it would be precluded from participation in any of the other tests, as well.  That is a decision that Symantec has every right to make and it should be respected.  AV-Comparatives should not be testing Norton at all.  Worse, to selectively test Norton only on the one test that Symantec contends is misleading and none of the others, is highly unethical.  Let's hope that other vendors are not paying to have a competitor's product entered only into certain tests that put the product in the worst light.  That would not say much for the impartiality of the tests, or the independence of the testing lab, now would it?

Basically it is blackmail.

Symantec dropped out because of this test so they are going to continue to include them in this test figuring they will have to pay the money to be included in the tests it scores well on.

 

 

I'll stick with my gut feeling that I'm being well looked after, by Norton. Av Comparatives results are always skewed.


SendOfJive wrote:

 

[...]

 

Symantec elected not to participate in the file detection test, knowing that it would be precluded from participation in any of the other tests, as well.  That is a decision that Symantec has every right to make and it should be respected.  AV-Comparatives should not be testing Norton at all.  Worse, to selectively test Norton only on the one test that Symantec contends is misleading and none of the others, is highly unethical.  Let's hope that other vendors are not paying to have a competitor's product entered only into certain tests that put the product in the worst light.  That would not say much for the impartiality of the tests, or the independence of the testing lab, now would it?


Hi SendOfJive

 

In addition to File Detection tests, AV-Comparatives has, on two occasions this year, tested Norton products in their Performance Tests. On both of these occasions, the additional testing costs were borne by a magazine publisher.  Both times, the Norton products achieved AV-Comparatives highest rating, ‘Advanced+’ (see May 2013 results below):

 

AV-Comparatives - Performance Test - Symantec.png

 

Link:  http://www.av-comparatives.org/performance-tests/

 

Does this address your concerns about the impartiality of the tests and the independence of the testing lab?

This was AV-Comparatives' rationale for requiring participation in certain tests, such as the File Detection test (emphasis mine):

 

In order to ensure that our test results give a complete and accurate picture of a product’s capabilities, AV-Comparatives has strict rules about which tests every product must take part in, and which tests are optional. This is to ensure that the aggregated results best reflect the actual protection the tested products provide to their customers.

 

As Symantec Norton declined to participate in the File Detection Test, we will unfortunately not be able to provide our readers with any test results for Symantec Norton in 2012. Symantec Norton have the option of officially returning to the tests, but only in accordance with the standard conditions, namely that results of all compulsory tests will be published.

 

Apparently, the "strict" rules are waived in cases where a private third-party is offering money to do selective testing of a non-participating product; and I guess AV-Comparatives feels that customers won't be too badly misled about a product's "actual protection" if the "aggregated results" of several required tests are missing when the limited-scope testing is bought and paid for.  It doesn't appear to me that AV-Comparatives is hewing to its commitment to readers to provide "a complete and accurate picture of a product’s capabilities" in the case of Norton, where it seems to be far too willing to disregard its own guidelines and requirements for product inclusion in selected tests.


SendOfJive wrote:

 

This was AV-Comparatives' rationale for requiring participation in certain tests, such as the File Detection test (emphasis mine):

 

In order to ensure that our test results give a complete and accurate picture of a product’s capabilities, AV-Comparatives has strict rules about which tests every product must take part in, and which tests are optional. This is to ensure that the aggregated results best reflect the actual protection the tested products provide to their customers.

 

As Symantec Norton declined to participate in the File Detection Test, we will unfortunately not be able to provide our readers with any test results for Symantec Norton in 2012. Symantec Norton have the option of officially returning to the tests, but only in accordance with the standard conditions, namely that results of all compulsory tests will be published.

 

Apparently, the "strict" rules are waived in cases where a private third-party is offering money to do selective testing of a non-participating product; and I guess AV-Comparatives feels that customers won't be too badly misled about a product's "actual protection" if the "aggregated results" of several required tests are missing when the limited-scope testing is bought and paid for.  It doesn't appear to me that AV-Comparatives is hewing to its commitment to readers to provide "a complete and accurate picture of a product’s capabilities" in the case of Norton, where it seems to be far too willing to disregard its own guidelines and requirements for product inclusion in selected tests.


If Symantec thought that AV-Comparatives was behaving as unscrupulously as you’ve described above, then wouldn't it be within Symantec's rights to purchase a standalone Real-World protection test for their Norton products from AV-Comparatives? Surely a refusal to conduct that test would confirm that suspicion. Only Symantec themselves can confirm or deny this. Please also keep in mind that AV-Comparatives has their reputation to uphold.

 

With regards to the “strict rules” and “aggregated results”, please consider the following. The AV-Comparatives web site has an interactive product comparison feature that allows consumers to track how well each vendor’s product has performed in each of their tests over time:

 

http://chart.av-comparatives.org/chart2.php

 

AV-Comparatives has every right to make certain tests compulsory because, if they didn't, it would quickly make their interactive product comparison feature worthless. This feature relies on a continuous series of test results over time in order to provide value. If vendors were allowed to pick and choose which tests their products participated in, then the test results of products over time would quickly end up with gaps in their time series. As a result, consumers would not be able to make direct comparisons between vendor's test results for a particular time period (ie. Vendor A participated in a November 2013 test but Vendor B chose not to). To illustrate, AV-Comparatives has time series File Detection test results for Symantec that date back to 2004:

 

AV-Comparatives - File Detection Test - Symantec.png

 

As per your post above, AV-Comparatives stated that they wouldn't test Symantec's Norton products in 2012 and, as you can see, they honoured that commitment (note the missing data for 2012 in the graph above). So, at what point in time in that gap in the graph above did the Norton product’s File Detection capability fall from an Advanced rating to a Tested rating. You can now see how non-participation by vendors impacts on the AV-Comparatives interactive product comparison feature.

 

Right now, the pressing issue is Symantec's potential participation in AV-Comparatives testing for 2014. This decision has to be made as soon as possible; otherwise Symantec will miss their run. AV-Comparatives is the only testing facility that, as far as I’m aware, offers Real-World protection testing results on a monthly basis.

 

Given Symantec's emphasis on Real-World protection testing, shouldn't Symantec be testing their Norton products in as many Real-World protection tests as possible? What are your thoughts on this?

 

(Edit: Clarity in the second last paragraph above).

 

 

 

 


Elsewhere wrote:
If Symantec thought that AV-Comparatives was behaving as unscrupulously as you’ve described above, then wouldn't it be within Symantec's rights to purchase a standalone Real-World protection test for their Norton products from AV-Comparatives? Surely a refusal to conduct that test would confirm that suspicion. Only Symantec themselves can confirm or deny this. Please also keep in mind that AV-Comparatives has their reputation to uphold....

AV-Comparatives would not permit Symantec to "purchase" participation in a single test - that is not allowed.  It requires vendors to participate in all of the compulsory tests.  That is the point.  Symantec has a quarrel with the File Detection test methodology and is willing to forgo placement in the Real World and other tests rather than submit to a test that it feels is providing misleading results.

 

So what does AV-Comparatives do?  They completely disregard their stated reasons for insisting on participation in all compulsory tests, and go ahead and test Norton anyway - but only in selected tests, not in all of the compulsories!  Given AV-Comparatives' testing regulations, it certainly should not be selectively testing Norton, against Symantec's wishes, in tests that Symantec feels are unfair.  If AV-Comparatives feels that it is somehow obligated to publsih test results for Norton even if Symantec has elected not to participate, then it should stick with its stated principles and publish results for all mandatory tests.  Of course, that would mean that participation would no longer be voluntary for anyone.

 

I don't think Symantec needs to do anything or prove anything  regarding AV-Comparatives.  Symantec has a legitimate argument with one of the tests and so has opted out of all tests, as AV-Comparatives' participation requirements demand.  AV-Comparatives needs to follow its own rules and respect that.  Or change its rules.


SendOfJive wrote:

Elsewhere wrote:
If Symantec thought that AV-Comparatives was behaving as unscrupulously as you’ve described above, then wouldn't it be within Symantec's rights to purchase a standalone Real-World protection test for their Norton products from AV-Comparatives? Surely a refusal to conduct that test would confirm that suspicion. Only Symantec themselves can confirm or deny this. Please also keep in mind that AV-Comparatives has their reputation to uphold....

AV-Comparatives would not permit Symantec to "purchase" participation in a single test - that is not allowed.  It requires vendors to participate in all of the compulsory tests.  That is the point.  Symantec has a quarrel with the File Detection test methodology and is willing to forgo placement in the Real World and other tests rather than submit to a test that it feels is providing misleading results. [...]


The assumption that AV-Comparatives would not permit Symantec to purchase participation in a single test is still largely speculation. AV-Comparatives has already set a precedent that would enable Symantec to do so. Symantec needs to advise whether or not they actually attempted to purchase a Real-World protection test after the recent ad-hoc Symantec/Norton product tests conducted by AV-Comparatives and advise what the outcome was.

 


SendOfJive wrote:
[...]  If AV-Comparatives feels that it is somehow obligated to publsih test results for Norton even if Symantec has elected not to participate, then it should stick with its stated principles and publish results for all mandatory tests.  Of course, that would mean that participation would no longer be voluntary for anyone [...]

 

That’s the core issue here – participation in all independent tests should be mandatory for all vendors.

 

One of the biggest issues that's present across the whole antivirus software (AV) industry is the exaggerated claims made by each vendor’s product advertising about how effective their products are at protecting users. If their products effectiveness actually matched their advertised claims, then the global cost of cybercrime should be negligible. Instead, what we actually find is that the annual cost of cybercrime is this:

 

The findings are both eye-opening and deeply concerning. According to the report, some 1 million-plus adults become cybercrime victims every single day and, if you break that down, it equates to a staggering 12 victims per second. [...]

 

And what an impact that turns out to be, with the global price tag of consumer cybercrime now topping US$113 billion annually – enough to host the 2012 London Olympics nearly 10 times over –  while the cost per cybercrime victim has shot up to USD$298: a 50% increase over 2012. [...]

 

Source:  http://www.symantec.com/connect/blogs/cybercrime-takes-its-toll

 

Try writing the text “$113 billion” as a number on a piece of paper. Then tell me why AV software vendors should have the option of NOT participating in independent product testing. The onus is on the AV software vendors themselves to prove that their software works as per their advertised claims.  If the global cost of cybercrime tops US$113 billion annually, then it also highlights the fact that, under the current funding arrangements, the independent AV software testing facilities are seriously under-funded. That’s a win for cybercrime...

 

Given the above, consumers rely on the test results from AV software testing facilities like AV-Comparatives to separate the facts from the fiction. These test results allow consumers to make an informed decision about the product they will eventually choose to use to protect them, whether it be a free or paid-for product. Symantec now needs to make a decision about where they stand here; knowing, of course, that the clock is ticking...

 

PS I would like to remind everyone reading this thread that this is a public discussion. If you would like to share your views on this topic, then please do so now. As I mentioned earlier, there is a time limit in play here.

 

 

 


elsewhere wrote:

These test results allow consumers to make an informed decision about the product they will eventually choose to use to protect them, whether it be a free or paid-for product. Symantec now needs to make a decision about where they stand here; knowing, of course, that the clock is ticking...


It is Symantec's position that the methodology used in the File Detection test does not accurately reflect the protection capabilities of the Norton product and, therefore, consumers relying on those test results would not be able to make an informed decision about Norton.  That is where Symantec stands - they will not participate in that test.  That is their decision, and has been for some time.

 

Further, if Symantec, or any vendor, could simply purchase participation in only the tests that they chose, we would not be having this discussion.  Symantec is barred from participation in any tests unless it consents to full participation in all of the compulsory ones.

elsewhere

 

<< participation in all independent tests should be mandatory for all vendors. >>

 

Then you need to speak to your representatives in Washington .... if you are in the USA <s> Who else can make things mandatory?

 

Do you really want that level of interference? (And I'm not a tea drinker ..... )


SendOfJive wrote:

 

[...]

 

It is Symantec's position that the methodology used in the File Detection test does not accurately reflect the protection capabilities of the Norton product and, therefore, consumers relying on those test results would not be able to make an informed decision about Norton.  That is where Symantec stands - they will not participate in that test.  That is their decision, and has been for some time.

 

Further, if Symantec, or any vendor, could simply purchase participation in only the tests that they chose, we would not be having this discussion.  Symantec is barred from participation in any tests unless it consents to full participation in all of the compulsory ones.


Again, I would suggest that given the extenuating circumstances I mentioned earlier, Symantec would rightly have the option to purchase a standalone test. We would need Symantec to post an update in this thread that advises whether or not they actually chose to pursue this option.

 

Symantec’s position regarding the File Detection test has been largely refuted in the following post:

 

http://community.norton.com/t5/Norton-Internet-Security-Norton/Poor-results-in-latest-AV-Comparitives-test/m-p/953445/highlight/true#M237716

 

I’d suggest that everyone interested in this issue read all of that thread. Symantec’s decision to not participate in the AV-Comparatives testing is proving to be untenable.  Given the upcoming deadline for participation in next year’s testing, wouldn't now be the appropriate time to review that decision?

 

 

 

Tell it to AV-Comparatives:

 

Antivirus Testing Lab Wants to Know What's Important to You

I have a question . Symantec Exit From AV-C test at 2011 ? yes ? Why AV-C use name and logo Symantec at their test after 2011 ? I believe they Furious from Symantec . and try Retaliation exit Symantec from their test . because Symantec is most valid Security company and when exit AV-C users loss trust this site . must attention webroot same Symantec at end year 2012 exit from AV-C test . Also I believe AV-C Furious from Trend Micro Brand Validation . before Exit Symantec and webroot from their test , always trend micro at end table with poor detection . but after it they move trend micro to up table with good detection and use trend micro Credit for Authentication Small and new AV Companies same kaspersky .