Stats.norton.com - DNS/nameserver issue

Archive
1

This is a repeat post of a topic I posted last month but there is a problem with display of screenshot images in that post.  I can see the images if I am logged in to the community but otherwise they appear as "Image in moderation".  So I'll try again.

The issue started on May 10th when I noted that Norton's Product Maintenance task frequently failed to complete.  I know from an issue some years ago (also DNS related) that some Norton background tasks need to access the stats.norton.com server.   On checking I found that my ISP's DNS was regularly failing to resolve stats.norton.com.    Further investigation revealed that stats.norton.com is now hosted by Microsoft's Azure cloud services and access managed by Azure Traffic Manager.  My ISP diagnosed that the issue occurs when its DNS attempts to refresh its cached records from the trafficmanager.net authoritative nameservers via IPv6 but everything works fine if refreshed via IPv4.

Full details to follow in my next post.

2

A further related discovery is from this Traffic Manager FAQs web page (which is dated June 3rd, later than the date in May when this issue started):

https://learn.microsoft.com/en-us/azure/traffic-manager/traffic-manager-faqs

It includes this section:

Does Traffic Manager support IPv6 endpoints?

Traffic Manager doesn't currently provide IPv6-addressable name servers. However, Traffic Manager can still be used by IPv6 clients connecting to IPv6 endpoints if the client's recursive DNS server supports IPv4. A client doesn't make DNS request directly to Traffic Manager. Instead, the client uses a recursive DNS service. An IPv6-only client sends requests to the recursive DNS service via IPv6. The recursive service must then be able to contact the Traffic Manager name servers using IPv4. Traffic Manager responds with the DNS name or IP address of the endpoint.

 

The statement that Traffic Manager doesn’t provide IPv6-addressable name servers conflicts with reality.  All four trafficmanager.net nameservers do have IPv6 addresses, as follows:

Tm1.dns-tm.com has the IPv6 address   2603:1061:0:f00::f0

Tm2.dns-tm.com has the IPv6 address   2620:1ec:8ec:f00::f0

Tm1.edgedns-tm.info has the IPv6 address  2a01:111:4000:f00::f0

Tm2.edgedns-tm.info has the IPv6 address  2620:1ec:bda:f00::f0

If Microsoft’s understanding is that they don’t support IPv6 nameservers for Traffic Manager, maybe those nameservers are not properly configured for IPv6 operation and that explains why, when addressed via IPv6, they return Authority records and not Answer records when asked to resolve stats.norton.com.trafficmanager.net.

Yet again,  Norton Support need to pursue this with Microsoft Azure Support.

3

As promised, here are the full details of the DNS/nameserver problem impacting access to stats.norton.com.

  1. The Norton domain stats.norton.com is now hosted by Microsoft Azure cloud services.   Access to the Azure hosts is managed by Azure’s Traffic Manager system.    Therefore, the DNS record for stats.norton.com is now an alias (CNAME) that points to stats.norton.com.trafficmanager.net.
     
  2. Stats.norton.com.trafficmanager.net is a further CNAME that resolves to one of two end-points, either mue1-bit-ipv6-pip.eastus.cloudapp.azure.com, with an IP address of 52.188.144.172, or muw1-bit-ipv6-pip.westus.cloudapp.azure.com, with an IP address of 13.64.142.149.
     
  3. To ensure effective traffic management between the two endpoints, the CNAME record for stats.norton.com.trafficmanager.net has a short Time-to-Live (TTL), requiring downstream DNS servers to regularly refresh their cached copy of the record by reference to the trafficmanager.net nameservers.
     

Using the Powershell command Resolve-dnsname, this screenshot shows the correct resolution of stats.norton.com.   The lookup is fully resolved to the westus Azure server and the authority section is that for Microsoft Azure host:

However, the DNS lookup often fails to fully resolve, resulting in this response.  Stats.norton.com.trafficmanager.net is not resolved and the authority section is for trafficmanager.net, not the Azure endpoint.

The failure is due to the downstream DNS server failing to correctly refresh its stats.norton.com.trafficmanager.net record from a trafficmanager.net nameserver if accessed via IPv6.  Trafficmanager.net publishes 4 nameservers, as per:

Resolving those names reveals that each nameserver has an IPv4 and an IPv6 address, as per this example of tm2.edgedns-tm.info:

Attempting to resolve stats.norton.com.trafficmanager.net via the IPv4 address works correctly but fails via the IPv6 address, as shown in this screenshot of attempts with the addresses for nameserver tm2.edgedns-tm.info:

Via IPv4, the correct Answer section is returned.  Via IPv6, only the Authority section is returned.  The same problem is seen with any of the four nameservers.

Further confirmation of  the issue of the trafficmanager.net nameservers not returning the correct response if queried via IPv6 can be seen in these tests using the NSLOOKUP command with the -debug switch.  (Test results are shown in text form as the the responses are too lengthy to be captured in a screenshot.)

1.   Look up stats.norton.com.trafficmanager.net from nameserver tm2.edgedns-tm.info via its IPv4 address 13.107.206.240:

PS C:\WINDOWS\system32> Nslookup -debug stats.norton.com.trafficmanager.net 13.107.206.240
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 1, rcode = REFUSED
        header flags:  response, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        240.206.107.13.in-addr.arpa, type = PTR, class = IN

------------
Server:  UnKnown
Address:  13.107.206.240

------------
Got answer:
    HEADER:
        opcode = QUERY, id = 2, rcode = REFUSED
        header flags:  response, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        stats.norton.com.trafficmanager.net.home, type = A, class = IN

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 3, rcode = REFUSED
        header flags:  response, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        stats.norton.com.trafficmanager.net.home, type = AAAA, class = IN

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 4, rcode = NOERROR
        header flags:  response, auth. answer, want recursion
        questions = 1,  answers = 1,  authority records = 0,  additional = 0

    QUESTIONS:
        stats.norton.com.trafficmanager.net, type = A, class = IN
    ANSWERS:
    ->  stats.norton.com.trafficmanager.net
        canonical name = muw1-bit-ipv6-pip.westus.cloudapp.azure.com
        ttl = 30 (30 secs)

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 5, rcode = NOERROR
        header flags:  response, auth. answer, want recursion
        questions = 1,  answers = 1,  authority records = 0,  additional = 0

    QUESTIONS:
        stats.norton.com.trafficmanager.net, type = AAAA, class = IN
    ANSWERS:
    ->  stats.norton.com.trafficmanager.net
        canonical name = muw1-bit-ipv6-pip.westus.cloudapp.azure.com
        ttl = 30 (30 secs)

------------
Name:    stats.norton.com.trafficmanager.net

In the final two parts of the response, both questions get the Answer record correctly resolving stats.norton.com.trafficmanager.net to the canonical name (CNAME) muw1-bit-ipv6-pip.westus.cloudapp.azure.com.

2.  Look up stats.norton.com.trafficmanager.net from nameserver tm2.edgedns-tm.info via its IPv6 address 2620:1ec:bda:f00::f0:

PS C:\WINDOWS\system32> Nslookup -debug stats.norton.com.trafficmanager.net 2620:1ec:bda:f00::f0
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 1, rcode = REFUSED
        header flags:  response, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        0.f.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.f.0.a.d.b.0.c.e.1.0.0.2.6.2.ip6.arpa, type = PTR, class = IN

------------
Server:  UnKnown
Address:  2620:1ec:bda:f00::f0

------------
Got answer:
    HEADER:
        opcode = QUERY, id = 2, rcode = REFUSED
        header flags:  response, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        stats.norton.com.trafficmanager.net.home, type = A, class = IN

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 3, rcode = REFUSED
        header flags:  response, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        stats.norton.com.trafficmanager.net.home, type = AAAA, class = IN

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 4, rcode = NOERROR
        header flags:  response, auth. answer, want recursion
        questions = 1,  answers = 0,  authority records = 1,  additional = 0

    QUESTIONS:
        stats.norton.com.trafficmanager.net, type = A, class = IN
    AUTHORITY RECORDS:
    ->  trafficmanager.net
        ttl = 30 (30 secs)
        primary name server = tm1.dns-tm.com
        responsible mail addr = hostmaster.trafficmanager.net
        serial  = 2003080800
        refresh = 900 (15 mins)
        retry   = 300 (5 mins)
        expire  = 2419200 (28 days)
        default TTL = 30 (30 secs)

------------
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 5, rcode = NOERROR
        header flags:  response, auth. answer, want recursion
        questions = 1,  answers = 0,  authority records = 1,  additional = 0

    QUESTIONS:
        stats.norton.com.trafficmanager.net, type = AAAA, class = IN
    AUTHORITY RECORDS:
    ->  trafficmanager.net
        ttl = 30 (30 secs)
        primary name server = tm1.dns-tm.com
        responsible mail addr = hostmaster.trafficmanager.net
        serial  = 2003080800
        refresh = 900 (15 mins)
        retry   = 300 (5 mins)
        expire  = 2419200 (28 days)
        default TTL = 30 (30 secs)

------------
Name:    stats.norton.com.trafficmanager.net

In this response, the final two questions receive Authority records rather than Answer records, as confirmed by the record counts in the response headers.

 

 

Not all DNS providers are impacted by this issue, both Google DNS and Cloudflare DNS consistently resolve stats.norton.com OK if queried via their IPv4 addresses.   Perhaps the explanation is that they always default to refreshing their cached records via IPv4 and ignore IPv6 nameserver addresses.  My current workaround is to set my router to use Google DNS instead of my ISP DNS.  Previously I bypassed DNS with a mapping for stats.norton.com in my Windows hosts file.

After feedback from my ISP, I have raised a support case with Norton support.  A recent update is that the case has now been escalated from "L2" to "L3", but the first level call agents were not able to give me any further details of the investigation.  Ultimately, I would think Norton need to raise this with Microsoft as the issue would appear to be with the Azure Traffic Manager nameservers.

I have received an email from Norton Support saying they tried unsuccessfully to contact me by phone and that no voicemail was available.    That doesn't make sense as I have voicemail and there is no missed call on my incoming call log.    I contacted Support again, checked that they do have my correct phone number and booked a call-back but again no call was received.    If a Norton forum moderator is monitoring this topic, I'd be grateful if they could pursue it with Norton support to obtain more details.