Handling Stale Resolver Caches After Cutover

Problem Statement

The TTL was 60 seconds, the change went out four hours ago, every public resolver you check returns the new address — and a steady trickle of requests is still arriving at the old origin. The DNS is not broken. Something in the path is holding the previous answer for longer than the TTL said it would: a resolver applying its own floor, a browser with its own cache, a long-lived process that resolved once at start-up and never asked again. This page sits under DNS Propagation Tracking.

Caching layers between the zone and the client Five layers — authoritative zone, recursive resolver, operating system stub, browser cache and long-lived application process — each with its own respect for the TTL. Five Caches, Not One LAYER RESPECTS TTL? TYPICAL TAIL Authoritative zone it sets it none Recursive resolver mostly, with a floor up to 5 minutes OS stub resolver usually seconds to minutes Browser cache its own policy about a minute Long-lived process often never re-resolves until restart
The last row is the one that produces traffic days after a cutover, and it is not a DNS problem at all.

When to Use This Approach

  • Traffic is still reaching the old origin well past the point where the TTL says it should have stopped.
  • You need to decide when the legacy origin can safely be switched off.
  • Public resolver checks all return the new address but the origin logs disagree.
  • API clients or server-to-server integrations are involved, where long-lived processes are common.

“Propagation” Is the Wrong Word

DNS does not propagate. Nothing is pushed anywhere. An authoritative server changes its answer, and every cache in the world continues serving what it already holds until that entry expires and somebody asks again. The word “propagation” implies a wave moving outwards on a schedule you can watch, and that mental model produces exactly the wrong expectations: that the change is progressing, that it will complete, and that the remaining stragglers are behind but catching up.

The accurate model is expiry. Each cache holds an answer with a countdown attached, and the countdown started when that cache fetched it — not when you made the change. A resolver that fetched thirty seconds before your edit will serve the old answer for almost the full TTL; one that fetches thirty seconds after will never see it at all. There is no ordering, no wave, and no completion event.

Once you think in expiry terms, the behaviour in this page stops being mysterious. A resolver returning a longer TTL than you publish is applying its own floor, which is a fact about their configuration and not about your change. A client that never re-asks will never expire anything, because expiry requires a subsequent question. And nothing you do to the zone now affects an answer that has already been handed out.

# Not "has it propagated" but "what is each cache still holding, and for how long"
for r in 1.1.1.1 8.8.8.8 9.9.9.9; do
  printf '%-12s ' "$r"; dig +noall +answer @"$r" example.com A | awk '{print $5, "ttl", $2}'
done

Step-by-Step Instructions

1. Confirm the Zone Is Actually Serving What You Think

Before investigating caches, rule out the trivial cause: a record that was never changed, or a second authoritative server still serving the old zone.

# Ask every nameserver in the delegation, not just the one you edited
for ns in $(dig +short NS example.com); do
  printf '%-28s ' "$ns"; dig +short @"$ns" example.com A | tr '\n' ' '; echo
done

Any disagreement here is a zone propagation fault, not a caching one — resolve it with the technique in Verifying DNS Propagation with dig Across Resolvers.

2. Measure the Stale Traffic at the Old Origin

Quantify before acting. The shape of the tail tells you which layer is responsible.

# Requests per minute still landing on the legacy origin, with their user agents
awk '{print substr($4,14,5)}' /var/log/nginx/access.log | uniq -c | tail -20
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

A tail dominated by browsers decays within minutes. A tail dominated by one API client’s user agent, flat over hours, is a long-lived process that resolved at start-up.

3. Identify the Layer From the Decay Curve

# Remaining TTL each public resolver is counting down — a floor shows up as a value above yours
for r in 1.1.1.1 8.8.8.8 9.9.9.9 208.67.222.222 64.6.64.6; do
  printf '%-16s ' "$r"; dig +noall +answer @"$r" example.com A | awk '{print $2}'
done

A resolver returning a TTL higher than the value you publish is applying a minimum of its own, and no amount of waiting on your side shortens it.

Decay shape identifies the responsible layer Two decay curves on the legacy origin: a fast exponential fall from resolver and browser caches, and a flat plateau from long-lived processes that never re-resolve. Read the Shape, Not the Volume caches expiring — normal flat plateau — a process, not a cache t+0 t+2h t+8h A plateau never resolves on its own — somebody has to restart something.
A decaying tail is a waiting problem; a flat tail is a communication problem with whoever runs the client.

4. Keep the Legacy Origin Serving Correctly Meanwhile

Do not switch the old origin off to force the issue. While the tail exists, the old origin should either serve the same content or redirect — never fail. The cheapest safe posture is a minimal redirect service on the legacy address.

# Legacy origin during the tail: redirect everything, keep the carve-outs alive
server {
    listen 443 ssl;
    server_name example.com;
    location = /google1a2b3c.html { root /var/www/verification; }
    location ^~ /.well-known/      { root /var/www/verification; }
    location / { return 301 https://$host$request_uri; }   # to the new origin's address
}

5. Chase the Plateau Directly

A flat tail needs a human. Identify the client from its user agent or source addresses, contact the owner, and ask them to restart or to configure a resolution TTL. Java applications with the historical networkaddress.cache.ttl default and connection pools built at start-up are the usual suspects.

Worked Example

A logistics API moved origins with a 60-second TTL. Browser traffic on the legacy origin fell to zero within eleven minutes, exactly as expected. A flat 40 requests per minute continued for three days.

The user-agent breakdown showed a single client library and four source addresses, all belonging to one freight partner. Their integration was a long-running JVM service that resolved the hostname once when its connection pool was created — with the JVM’s cache configured to never expire — and had last been restarted five weeks earlier.

Three things followed:

  1. The legacy origin was kept alive as a redirect service rather than switched off on the planned date.
  2. The partner restarted their service during their own change window, and the plateau went to zero within a minute.
  3. The integration guide gained a note about resolution caching, and the migration runbook gained a step: identify long-lived clients before cutover and schedule their restart.

The DNS change had been correct throughout. The failure was of expectation, not of configuration.

Verification

# 1. Every authoritative nameserver agrees on the new address
for ns in $(dig +short NS example.com); do dig +short @"$ns" example.com A; done | sort -u

# 2. Public resolvers return the new address, and their TTL is counting down
for r in 1.1.1.1 8.8.8.8 9.9.9.9; do
  printf '%-12s ' "$r"; dig +noall +answer @"$r" example.com A | awk '{print $2, $5}'
done

# 3. Legacy origin traffic is decaying, not flat — the number that decides switch-off
awk '{print substr($4,14,2)}' /var/log/nginx/access.log | uniq -c | tail -12

Do not switch the legacy origin off until check three has read zero for a full day, and preferably a full week for anything with server-to-server integrations. The cost of keeping a redirect service running is trivial next to the cost of a partner’s integration failing silently — the same reasoning applied to rollback windows in Planning DNS Rollback Windows Around TTL.

Decommissioning gates for the legacy origin Four gates before switching off the legacy origin: tail at zero for 24 hours, no server-to-server clients remaining, verification assets re-homed, and certificates still valid. Before You Switch the Old Origin Off Tail at zero for 24 hours a week if APIs No S2S clients partners confirmed restarts done Tokens re-homed verification in DNS nothing left on it Certs valid redirect still TLS renewal path alive A legacy origin that fails TLS is worse than one that is switched off — the error is unrecoverable. Keep renewals running for as long as the host answers at all.
Certificate renewal on the legacy host is the gate teams forget, and it fails weeks later without warning.

FAQ

Why do some resolvers hold the record longer than my TTL? Because many apply a minimum lifetime of their own, typically between 30 and 300 seconds, and some prefetch popular records on their own schedule. You cannot shorten another operator’s floor; you can only plan for it.

Can I force a resolver to drop a cached record? A few large public resolvers expose a cache-flush form for a single name, which is useful for spot checks. It does not scale to the thousands of resolvers in the path, so it is a diagnostic convenience rather than a remedy.

How long should the legacy origin keep answering? Until traffic to it has been zero for at least a day — a week where server-to-server integrations exist. Running a small redirect service costs almost nothing next to a partner integration failing silently.

What causes a completely flat tail that never decays? A process that resolved the hostname once and cached the result for its lifetime. Long-running JVM services, connection pools built at start-up and some HTTP client libraries all behave this way, and only a restart or a configuration change fixes it.

Does lowering the TTL further help once the change has already gone out? No. Resolvers holding the old answer are counting down the TTL that was in force when they fetched it; a lower value now applies only to fetches made from this point on. TTL reduction is a preparatory step, never a remedial one.

Should I identify stale clients by IP address or by user agent? Start with the user agent, because it usually names the library or the integration and therefore the owner. Fall back to source addresses when the agent is generic — a small set of addresses arriving at a constant rate is a strong signal of a single long-lived process rather than a population of visitors.

Can I use the old origin’s logs to prove the tail is real? Yes, and it is the only reliable evidence. Public resolver checks tell you what a handful of resolvers currently return; the origin’s access log tells you what is actually still arriving, which is the number that decides whether the host can be switched off.

Is a plateau ever legitimate? Occasionally — a monitoring probe or a health check configured against the old address will produce one, and it is harmless. It is still worth chasing, because a probe pointed at a host you are about to decommission will start alerting the moment you do.

What if the client is a third party who will not restart? Then the legacy address has to keep answering for as long as they persist, which turns a technical question into a commercial one. Keeping a small redirect service alive indefinitely is usually cheaper than the alternative, and it is the reason to hold the legacy domain rather than release it.

How does this interact with anycast or load-balanced origins? It does not change the caching behaviour, but it does change what “the old origin” means. If both old and new addresses are served by the same anycast network, the tail may be invisible in logs you are reading — check at the layer that actually terminates the connection.

Related

← Back to DNS Propagation Tracking