The Independent Testing Labs

Who Are They?

The big independent testing labs do large scale testing of popular anti-malware products. They have dozens of employees and work 24/7 to determine the best performers.

AV-Test and AV-Comparatives do large scale testing that includes on-demand scanning and on-access scanning. On-demand means they run the program's scanner on a large set of samples and record the results. On-access means they test the software's ability to catch existing and zero-day threats when the software isn't completely updated with the newest virus signatures and program updates. This requires checking files for behavior that might be malware.

Overall, they don't limit the sample size and try to include every threat known and unknown as of the time of testing. This can mean millions of samples.

Virus Bulletin awards it's "VB100" award to any product that detects all of the threats on the ITW list. But they do it for multiple operating systems and that list changes periodically. Whenever possible, we report the results for Windows XP Pro but it will be the same platform for all of the reviews.

AV-Test releases results based on percentage for both on-demand and on-access scanning.They also throw in a "false alarm" rating using terms such as "almost none", "few", and "many". If a product detects 98% of known threats, it gets a "98%" rating for that scan. The same is true for on-access or "blocking" of threats. AV-Test only releases results to certain publications so you won't find them on their web site. You can usually see them in PC Mag.

Update: For 2012, PC Mag has a nice explanation on how they rate antivirus products. Though we have some differences, the goal is the same: to put more weight on the independent testing lab results than we do on our own subjective opinions. You can read about it here: PC Mag How we interpret...

AV-Comparatives uses a different system. They use the terms, "Standard", "Advanced", and "Advanced +" to rate performance. They also give a word-based rating of false positives, "none", "few, and "many". You can read AV-Comparative's reports yourself at their web site. These reports explain the ratings in more depth.

Virus Bulletin tests using the ITW, or "In The Wild" virus list as compiled by WildList.org. They list every known virus that is believed to be actively spreading around the world. Virus Bulletin also does on-demand and on-access testing but a product will get what is known as a "VB100" award if it detects all of the ITW threats.

How Best-Security-Software.com Does Reviews

For my reviews, I consult AV-Test, AV-Comparatives, and Virus Bulletin--three of the most well respected independent testing labs. There are others that we could consider but I believe that these three provide enough information.

There are a couple of things to know about the big labs' testing approach: First, each lab has its own testing methods and criteria for who participates. Because of this, we can't get results for every every product from each of the labs; they just don't test them all.

Secondly, this site focuses on suites that include antivirus, antispyware, and firewall components. All three independent testing labs test for antivirus performance but not necessarily for the other parts. They even state that if they test a particular antivirus product, that doesn't imply the same results for a suite that includes that component.

So, the reviews DO reflect the same antivirus engines (although not necessarily the exact same version number--still they're very close) but don't project onto the other features of the suites. That said, most antivirus engines includes antispyware nowadays.

Just to get it clear in your head, consider all these variables that the independent testing labs must contend with:

• Millions of viruses, Trojans, spyware, rootkits, rogue programs, exploits, adware, and so on, exist today
• Hundreds of new threats and variations of existing threats appear every day
• There are multiple operating systems in use: Windows 2000, Server 2003, Server 2007, XP, Vista, and Windows 7 to name a few
• There are dozens of browsers in multiple versions in use on all of those various operating systems. Browsers are the major source of security exploits
• Only a fraction of users keep all of their software up to date--one of the first steps to keeping you computer secure
• Testing in labs is never quite the same thing as the real world though AV-Test is trying to simulate this to some degree

I could go on but I hope you see that it's impossible to write security software that offers 100% protection from all threats all of the time. And the independent testing labs can't test every possible combination; it's just too vast.

What this means is that I can't name an all-around guaranteed winner for an Internet security suite--and no one else can either. I'll give you the labs' results and you'll have to decide which one is best for you.