Compressed File Scan Looping - Concerns With Disable Recommendation

I have an issue in NIS 2009 where a full system scan shows 650,000-750,000 items scanned, when it should be more like 150,000 (or somewhat less using Insight).

Tech support’s advice was to turn off compressed file scanning, as I could be getting tripped up by the “scans 10 levels deep” property of the Norton on-demand scanner.   I’ve seen the same advice in these forums, although one expert seemed to vehemently disagree that it is normal and that it is “good enough” to let realtime protection handle any potential infection.  (He suggested that the poster do a clean reinstall.  But do note that this is happening on both of my pc’s.)  I also note that the help files strongly recommend leaving compressed-file scanning – the default – enabled to ensure infection detection.  This seems distinctly at odds with the prevailing advice from tech support and forum experts.

Anyway, I have some specific questions on this issue that weren’t really addressed by either tech support or the previous threads.  Discomfort with this is the one remaining issue that may possibly keep me from proceeding with purchase of licenses for my NIS 2009 trials.  I need to make a decision soon, so I  would greatly appreciate any input on these specific questions:

1.  I have watched the full-system scans, and saw that the “looping” was clearly all in one XP folder: \Windows\Installer.  So why couldn’t I just make this folder a scan exception instead of entirely disabling compressed-file scanning against Help’s advice?

2.  When I do an on-demand custom scan on soley this folder (from either the main GUI or the context menu) it finishes in about a minute.  It reports about 450 files scanned and zero skipped.  Why the discrepancy between this and the full-system scan?  Why no looping?  (BTW, all other parts of this post assume a full-system scan.)

3.  Why does the scan eventually finish (with an extremely large number of files listed)
instead of entering an endless loop?  Does it just “say”, “OK, scanned this one 10 deep, giving up and moving on?” 

4. Does the inflated number of items reported actually represent something, or is it an aberration?

6.  Let’s say I don’t mind letting the scan run for an hour and scan ~700,000 items.  Can I trust the results from the rest of the scan despite the looping in Windows\Installer?


7.  One thing that makes me the most uncomfortable is that the problem is only intermittemt (again, on both pc’s).  Why would that be?  Sometimes if I run another full scan very soon after one of these “inflated” scans, the count drops down to anything from 100,000 - 240,000 items (similar to what I got when I disabled compressed-file scanning as a test at the direction of tech support, who then declared the issue solved.)  The number listed as “skipped items” has ranged anywhere from 350 to 50,000 in these cases.  (BTW, Windows\Installer has about 150 compressed files, including a large number related to MS Office 2003.  Those tend to be the larger ones. The largest is 112mb.)


Thanks much for any input.  

Message Edited by Ardmore on 05-05-2009 11:52 PM
Message Edited by Ardmore on 05-05-2009 11:54 PM
Message Edited by Ardmore on 05-05-2009 11:56 PM

Good questions.  Windows Installer accounts for about 200,000 of the 600,000 files reported on my full scans as well.  I have no idea why.  But I can answer your last question about a second scan in quick succession being smaller and faster:  Norton will skip files that have not changed since the last scan.  So if you immediately rescan, the speed is impressive.  But the catch is that once you get new virus definition updates all those skipped files will be rescanned the next time.  This is because of the possibility that the files might contain malware that can only be found with the latest signature definitions and therefore could have been missed in earlier scans.  So after an initial scan your next scan could be either short or lengthy depending upon whether your definitions updated in the interim.  This would explain the apparent intermittance that you said you were most uncomfortable with.  Hopefully I’ve explained this correctly and this will solve that issue for you.

Message Edited by SendOfJive on 05-05-2009 10:55 PM

Thanks for the thoughts.  My understanding from the Help files is that files which are unscanned because they are unchanged from the last scan will be classified as Skipped.

The last time I did two consecutive full-system scans, I went from 650,000 items down to 105,000, yet Skipped was only listed at 400.  Is it possible that the deeply-compressed items are only considered one file apiece in the Skipped tally?  Even if the answer is “yes,” that still leaves open what I observed in question 2, where a scan soley of the problem-folder took less than a minute and reported zero files Skipped.

Then of course there are the other nagging questions…   But this is a start – thanks.  And BTW, where were you able to find out how many full-scan “items” a particular folder accounts for?  Are there more-detailed logs somewhere?


Ardmore wrote:
  And BTW, where were you able to find out how many full-scan “items” a particular folder accounts for?  Are there more-detailed logs somewhere?


I only know that Windows Installer accounts for so many files from having actually watched the scan.  This folder takes forever and when Norton finally finishes with it, I'm 200,000 or so files closer to the end.  :smileytongue:

Along with SendOfJive's answer I wanted answer each point:

 

1)  Not really looping but scanning each branch of the uncompressed file tree to the end or the limit of extraction (you can change this in the settings - 2GB or 10 levels deep).  When it reaches the end of that branch, Norton moves back to the branch point and continues onto the next files / branch.

 

2)  A context manual scan will invoke the uncompressing of the files.  A prefigured Full System Scan (from scan now) will do the same.  However an different scan from the Scan Now menu does not (not sure on the reasoning of this other than speed).

 

3)  Endless loop?  Not sure why you think Norton enters an loop, endless or otherwise.  The display may look that way but Norton scans one file and then the next until end of branch, end of size limit (compressed file setting) or end of depth (compressed file level).

 

4)  The number IS the total number of items the Norton scanned (registry keys, memory items, regular files and extracted files from compressed files.

 

5)  There is no 5

 

6)  Yes you can trust the entire scan.

 

7)  SendOfJive's answer is correct on this point.  This is a "temporary" Insight trust; definitions are the same, the files has not changed since I scanned last therefore I do not need to scan again.  The skipped number is the number of files that have not changed.  If either tests fail then Norton scans the file again. 

 

Hopes this helps.  Look forward to hearing your decision or if you have any more questions.

dbrisendine refers to an Insight scan.

 

Have you run Norton Insight  a few times to see how low the percentage needing scan drops, and how low it more less fixes on, unless you have a lot of changing files?

 

I see mine is settled on Trusted 96 / Scan needed 4% !

If you skip one CAB file, you are actually skipping hundreds, maybe thousands, of internal files.

dbrisendine -

 

This helps, but I'm still unclear on some of the questions, which happen to be key ones.

 

First, on #7:  Per the scan summaries, the most that the number of unscanned items has ever increased by between two consecutive full system scans is 50,000 (often far less), even when the second scan reports that the total full-scan count dropped by, say 500,000.  So I still have trouble understanding how your explanation accounts for this major discrepancy.   Could it be that each compressed file is counted as only one file when skipped?  (BTW, I don't have my home pc with me here, so I can't give exact counts at the moment.)

 

On #2:  If a context manual scan uncompresses the files, why does blaze through the folder in under a minute without the problems encountered in the manual scan, and then just report a scan number in the hundreds (roughly the same number of files in the folder, counting a compressed file as 1 file).  If this is simply a  much more efficient way of scanning (which would not be inutitive to me), then why wouldn't the full scan use it, too?

 

On #1:  You didn't address the issue of why I couldn't just set Windows\Installer as a scan exception instead of completely disabling compressed-file scanning.  And on a related note, the question of whether it is indeed OK to depend on realtime scanning to detect any possible compressed-file infection when the Help files cite the importance of leaving compressed-file scanning enabled.

 

Finally, to huwyngr's point about Insight "coverage":  Extrapolating, I can tell that Insight calls my C partition over 80% trusted.  So I can't understand why the unscanned files are often as low as 700.

 

Thanks.  Wish I had my pc here to double-check on all my "statistics," but I'm pretty sure they're in the ballpark.  When I get back to the pc on Sunday night I'll have 2-3 days left on the trial!

 

EDIT:  Re the Insight question, is it possible that the "% Trusted Meter" goes by % of file *space* instead of % of files? 

Message Edited by Ardmore on 05-08-2009 12:31 AM

#7)  One skipped uncompressed files = 1 file skipped

 

#1)  You can turn that off if you want.  It will skip scanning all compressed files.  AutoProtect scans every file when accessed anyway.  If you want to leave this on for the other files in C: and skip the Windows\Installer folder, this is also OK also.  Personal preference.  I would think that this is here as a warning to those who turn off the scan or exclude it and then, like a good MS customer, turn off the AV when installing large MS programs.  Sometimes this could be a problem.

 

#2)  This one bothers me in that I have never seen that happen before.  Reading over your original concern and this previous one carefully, I have to say I can't answer that one because I have never seen or heard of that happening.  I don't know if this is from changing the scan rules to include or exclude compressed files, exclude / include folders or a difference between having protected folders / files showing or not.

 

Sorry about not being able to answer the last point but I simply don't know cause I've never seen or heard of that happening. 

 

Personally, I don't exclude any files from scanning and have been very happy with the scanning speed of NIS2009 (800K to 1.1M in 1:50 to 2:10).  I know that not scanning in the compressed files is acceptable because I have never seen NIS2009 miss scanning an accessed file as it comes out of the compressed shell.

 

I also think that you have misunderstood the Insight percent and what it represents.  A good discussion on the subject is here:

http://community.norton.com/t5/Norton-Protection-Blog/Norton-Insight-A-solution-to-performance-improvement-without/ba-p/20642

 

Also, The Nis2009 Help file has a good amount of detail on Insight also. 

 

HI Ardmore, you know I had a problem sorta like this. in feb2009 I had done a rebuild of my Windows xp pro system and my NIS2009 was counting around 500,000 files with the Scan Performance Profiles set to Full Scan.  Then after the big 16.5.0.135 update my copy of norton started counting 250,000, I didn't change a single setting yet it started countting files strange. you can read about that here

 

I never really got a good asnwer to why it does this but maybe some day I'll find out why.

 dbrisdendine -

 

Thanks for tackling my followups questions.  After your latest reply, it's only question #2 that still leaves me--and apparently you, too-- somewhat uncomfortable.  As for my other questions:   While some aspects still aren't  crystal clear to me (or to others such as jarrycanada, apparently), your  responses  -- plus input here from other folks -- have helped allay my concerns..

 

Those  context scans were all done with the compressed-file scanning still at the default "enabled" for manual scans.  So it remains kind of perplexing and unsettling that the context scan  blazes through the "problem" folder (Windows\Installer) so fast, with no difficulty -- even more perplexing given that you say it is decompressing the files.  Fyi, I've never added anything to the scan exceptions, only inquired about doing it. (I've left the default System Restore folder exception in place.)  Also, I always have Windows Explorer set to show all files, including protected.

 

Other than this lingering concern I'd now be at a definitely-buy point.  I'd still give it a good 80% chance that I'll convert the trials to purchases, but it sure would be good to have a Symantec person weigh in on this, as one helpfully did (don't have his name handy) on my more-straightforward issue as to what Firefox "malicious sites" settings could/should be used in tandem with Norton Safe Search.  It would be welcome even if it can't happen until after my trials expire (although sooner would be best  :smileywink:).

 

 

Message Edited by Ardmore on 05-08-2009 06:26 PM

I've been following this thread with interest, as much to learn as to offer anything.

 

I am still confused by your term "loop", which to mean would be circling back on itself.  I 've tried to replicate what you describe in #2, and have seen nothing I would call looping.

 

I am also not sure what you mean by "context" scanning.

 

It seems to me that NIS gives you an option of scanning, everything, even those items not in a position to generate a virus.  .txt files and files without dot ids and numerous other files are neither executable not the kind of file that would catalyze a malware process.  So why scan them at all?  Because a malware component could be hidden as an innocent file then primed by being renamed.  For example:  Lilolme.txt could be renamed to tckngbmb.exe.  So it good to check all files on occasion.  Why not check all files all the time?  Because it's too time consuming.  Norton has another line of defense that would check tckngbmb.exe if and when it was launched.  It may even scan it whenever it gets created by being renamed.

 

My impression is that NIS full scan forces a scan of everything regardless of what your settings were.  And that anything else depends on individual settings.

Glad we could help, some.  I will see if I can dig on the #2 point.  If I get conclusive info then ---------------------> YOU.

 

Anyway, new things are coming.  And, I for one will be trying to get some of the nagging things ironed out in NIS2010 (not that I have that much influence but I think you understand).  Stay in touch and let us know how Norton works for you.

mijcar -

 

I believe the OP meant the right click context scanning of files (from inside explorer, for example).  I'm sure they will correct me if I'm wrong but that is what I meant in my answers.


dbrisendine wrote:

mijcar -

 

I believe the OP meant the right click context scanning of files (from inside explorer, for example).  I'm sure they will correct me if I'm wrong but that is what I meant in my answers.


If that is so, then I can report I got two different responses depending on my settings.

 

If I included compressed files, then an original collection of 978 files was expanded into 376,000 files, all scanned.

If I excluded compressed files, then the same collection was expanded into 1071 files.  But wait?!  How can 978 yield 1071 if there was no expansion?  Pure strange math and English, my dear Watson:

The original aggregate consisted of 978 files AND 72 folders AND .... take a deep breadth .... the containing original folder itself, which is not counted in the properties listing.  978 + 72 + 1 = 1071.

 

Hopefully, the math above will also help explain other numerical "discrepancies".

mijcar -

 

If what you report refers to a context-menu scan, then you apparently initiated the scan at a very high level, e.g. root C.  I never tried that, but will be interested in doing so when I get back to my pc's with NIS on Sunday night.

 

While that may prove a helpful diagnostic, I doubt it would solve the mystery of why my context-menu scans of the Windows\Installer folder complete in under a minute and produce an item count that's only in the hundreds when the compressed files in that folder have many, many thousands of items.

 

BTW, as dbrisendine suggested, "looping" was probably not an appropriate descriptor for me to use.  That's just the word that came to mind when I saw many of the same filenames from Windows\Installer going by over and over during the "high count" full system scans.

Now I understand; the "looping" is scanning the same file in different branches of the compressed file's tree.  What you don't always see is the middle of the file path due to screen width constraints.  In the compressed files, when it expands this and scans the file path may be longer than the string can display in the window, so Norton truncates the path string to fit showing the file it is scanning and usually the start of the path.  Since you are scanning MS installer packages, there will be the same support files in every package; hence it looks like Norton is 'looping' on certain files.

 

Wow, glad I read this thread and last post again.


Ardmore wrote:

mijcar -

 

If what you report refers to a context-menu scan, then you apparently initiated the scan at a very high level, e.g. root C.  I never tried that, but will be interested in doing so when I get back to my pc's with NIS on Sunday night.

 



I initiated the scan at the point C:\windows\installer (aka my computer\c:windows\installer\ )

 

 

OK, back and here are the results (compressed-file scanning enabled for all).  Turns out mijcar's math does indeed help explain some of the aparent discrepancies. 

 

First I did a Live Update followed by context scan of the Windows\Installer folder.  The scan took 6 minutes and showed about 500,000 items scanned.  This was the first time I ever had a result over 1000 for a context scan of that folder.

 

Then I immediately repeated the exact same context-menu scan, and it finished within seconds.  The result was 450 total items, which is the number of files (with each compressed file counting as 1) plus folders in Windows\Installer.


Then I ran a full system scan, and it showed 149,177 items scanned -- vs. the approx 700,000 I have had on some full system scans.  450 items were listed as skipped -- clearly the Windows\Installer folder.  (There were also some other categories including trusted, but I won’t go into that detail here, even though they obviously are very beneficial in reducing scan time.)

So, everything seems to be falling in place.  Apparently any kind of scan, including context, skips files which are unchanged from any prior scan (again of any type, including context) --assuming no interim update of the virus definitions, of course.  This must be the reason behind the consistently quick, lower-count results I got from context scans of this folder until tonight’s first run.  Lingering mystery solved!

I’ve already converted the NIS trials to purchases, BTW.  I’m really impressed by NIS 2009.  It’s an admirably well-thought-out, well-implemented program that’s full-featured yet really system-friendly.  This all assumes that the threat protection is highly effective, but I have no doubt that it is.

----

On a sidebar issue: dbrisendine, re your post about filenames packed into compressed files:   It actually didn’t appear that anything but the top-level filenames showed during the lengthier context-menu scan of Windows\Installer referenced here.  Now, I can’t say whether this observation would also apply to a full-system or custom scan.  I did try going straight to a full-system scan when I got around to my second NIS pc, but since it doesn’t have Office the Windows\Installer compressed files are much smaller and probably have fewer layers of compression.  So it was hard to draw any conclusions from that.  But I’ll do a full-system scan on the pc with Office sometime after the virus defintions are updated, and report back if the names I see zipping by look any different than in the long context-menu scan.  Maybe that will indeed turn out to be the case, because I honestly can’t confirm that I saw any filenames *repeating* in the long context-menu scan even though as you know I’ve definitely observed that phenomenon before, in full-system scans.

EDIT:  Oops, still getting used to the forums.  I thought "solved" was global for the thread, not to indicate one post as the accepted solution.  I sure didn't work through the issue by myself (!), but don't see any way to cut the "solved" from my own post.  Sorry:smileytongue:
Message Edited by Ardmore on 05-11-2009 02:46 AM

Not to worry about the 'Solved" thing; you did work out the 'math' on your own.  I personally don't worry over the number of files scanned at any particular time as definitions are nice but heuristics are better.  Knowing that AutoProtect does what it should and the different actions about compressed files usually helps settle any fears about missing a virus.  And even though I have not seen them yet, I feel the next versions will be even more impressive.

 

Ask your questions anytime; this is place for answers.  Glad your here.