AMTSO: Reviewing the Reviews

In January of this year, Symantec became a founding member of the Anti-Malware Testing Standards Organization (AMTSO). Symantec and about 40 of the leading academics, reviewers, testers and vendors in the security industry joined forces to form the organization dedicated to helping improve the objectivity, quality and relevance of anti-malware technology testing. Since then, we’ve met several times and shared numerous spirited debates, but something we have agreed upon since our inception is the charter of the organization:

·         Providing a forum for discussions related to the testing of anti-malware and related products;

·         Developing and publicizing objective standards and best practices for testing of anti-malware and related products;

·         Promoting education and awareness of issues related to the testing of anti-malware and related products;

·         Providing tools and resources to aid standards-based testing methodologies; and,

·         Providing analysis and review of current and future testing of anti-malware and related products.

Now that the first two official documents for the AMTSO have been adopted and published (http://www.amtso.org/documents/cat_view/13-amtso-principles-and-guidelines.html) the next step will be to put these principles into practice.  AMTSO is now working on a process to Review the Reviews.  This will be a process whereby a published review can be critiqued to see where it did or did not follow the AMTSO principles and guidelines – and for the latter give specific feedback as to why.  This is understandably a potentially very controversial thing to do.   I’d like to share my personal views on how this process should be conducted. 

The two main concerns people have about “reviewing the reviews” are that the process will seem to be AV vendors “whining” about losing a review and that a thorough analysis of a review cannot be done in a timely enough manner to blunt the damage caused by a problematic review.  To address the former several specific steps have been outlined.  For the second point this process will be explicitly understood not to attempt to prevent any damage from such a review, but rather to help improve future reviews. The general outline of the “Review Analysis” would be as follows.

·         Process must be impartial

·         Process must be transparent

·         Process is analytical only

·         Review analysis is not intended nor will it be able to “unring the bell.”   

Process must be impartial 

Steps should be taken to ensure that first there is a consensus that the review should be analyzed.  It is important that an entity not simply request the analysis because they lost, or did not like something that was done.  Two specific steps should be included.  First, specific examples of violations of the Principles and Guidelines must be identified.  Second, multiple entities must agree with those violations.  These two steps should help ensure that there is solid ground for the request. 

Process must be transparent 

The process of analyzing the review must be as open and visible as possible.  The entities requesting the analysis as well as their arguments for why should be made public.  The entity that conducted the review must have an opportunity to respond.  Other interested parties – both supporting and criticizing – must have an opportunity to comment. 

Process is analytical only. 

The output of this process should be as objective as possible, and should not contain subjective statements (such as “This was a poorly conducted test.”).  The output should be specific language regarding where the review in question is and is not in compliance with the AMTSO Principles and Guidelines – and why.  It may contain recommendations about specific steps which can be undertaken to gain compliance.   

Analysis cannot “unring the bell.”

This analysis will naturally take some time, but the larger goal is to improve future reviews, as opposed to responding quickly enough to give press quotes while a story is “hot.”  The committee review should happen in a timely manner, but this process should not be counted on to mitigate damage of a bad review. 

So what do you think about our goals for Review Analysis? I’m very interested in your feedback regarding my opinion of what I’d like to see happen, so please leave your comments and questions in the section below.

Message Edited by Sondra_Magness on 11-13-2008 02:10 PM