DNS, DHCP, and Name Resolution Questions
How names and addresses are served and kept working on a network. DNS: resolution flow (stub, recursive, authoritative, root/TLD referrals), host-side resolver configuration and why one machine or process resolves differently from another, record types and their pitfalls (including apex aliasing, MX, SRV and CAA use), zones, delegation and glue, caching, TTL and negative caching, split-horizon and private zones, reverse DNS, zone transfers (AXFR/IXFR, TSIG), DNSSEC and key rollover, resolver and authoritative fleet design, forwarding versus running recursion, encrypted DNS (DoH/DoT), truncation, TCP fallback and EDNS, DNS-layer attacks (cache poisoning, amplification, spoofed-source floods), DNS health from the user's point of view, and DNS changes, mail and registrar migrations and outages. DHCP: address assignment and leases, scopes and sizing, reservations, relay across VLANs including relay agent information (option 82), redundancy and failover, and rogue or exhausted-scope failures. Includes diagnosing resolution failures that masquerade as wider outages. Excludes the layered network fault-isolation method and general packet capture, Active Directory-integrated DNS on domain controllers, DNS-based service registries for microservices, load-balancing algorithm design and global traffic steering, incident-command process and SLO or error-budget design, and generic scripting or infrastructure-as-code tooling.
A security team wants to block clients from using public encrypted DNS, while users argue it protects their privacy. How do DoH and DoT change what an enterprise can see and enforce, and what would you propose?
Sample Answer
Direct answer
Encrypted DNS protects the query from everyone on the path, and that includes the enterprise. DoT (DNS over TLS) wraps DNS in TLS on its own TCP port, 853 (RFC 7858). DoH (DNS over HTTPS) carries DNS messages inside ordinary HTTPS requests (RFC 8484). Once a device talks to a public encrypted resolver, the company can no longer see the names it looks up, apply DNS-based blocking, or log queries per client. I would not fight this with a blanket ban alone. I would run an enterprise resolver that itself speaks encrypted DNS, point managed devices at it by policy (so users get privacy from the local network and the company keeps visibility on its own devices), block the unmanaged routes at the network edge, and be explicit with staff about what the company can see.
What each side sees and can enforce
| Plain DNS (UDP/TCP 53) | DoT (TCP 853) | DoH (TCP 443) | |
|---|---|---|---|
| Query names visible to the network | Yes | No (encrypted) | No (encrypted) |
| Easy to identify as DNS | Yes, by port | Yes, by port 853 | No, looks like web traffic to a firewall that only reads ports |
| Block with a firewall rule | Port rule works | One rule: deny outbound TCP 853 | Port rule would block all web traffic; needs destination lists or TLS inspection |
| Enterprise DNS filtering, logging, response policy (rewriting or refusing the answer for chosen names) | Works | Works only if devices use the enterprise resolver | Works only if devices use the enterprise resolver |
| What remains visible | Everything | Destination IP, volume, timing | Destination IP, volume, timing, and usually the server name the client announces in the TLS handshake (SNI, server name indication, normally sent unencrypted so the server knows which site is wanted) |
RFC 8484 states this plainly: filtering or inspection systems that rely on unsecured transport of DNS will not function in a DoH environment. The protocol encrypts and authenticates the server, but it does not tell the resolver who is a managed device.
The two positions are both partly right
- Security team: DNS logs and DNS firewalls (blocking known-bad domains, spotting command-and-control beaconing (malware contacting its operator's server at regular intervals), alerting on newly registered domains) only work if the queries pass through resolvers the company controls. A browser that silently sends its lookups to a public DoH service bypasses all of it.
- Users: On a coffee-shop or home network, plain DNS is readable and forgeable by anyone on the path, and encrypted DNS fixes that. The privacy gain is real against the local network and the ISP. It does not hide queries from the resolver operator, so it moves trust to a different party rather than removing it.
What I would propose
Steps 1 to 3 do the enforcing. Steps 4 to 6 make the setup smoother for clients and defensible to staff.
- Run an enterprise resolver that supports encrypted DNS. Clients on the corporate network and on VPN use it, over DoT or DoH, so the connection is protected from the LAN while the company keeps logging and filtering at the resolver. The resolver should also forward upstream over TLS. An Unbound example that passes
unbound-checkconf(the forwarder address is a documentation placeholder, which stands in for the vetted upstream you choose):
server:
interface: 127.0.0.4
tls-cert-bundle: /etc/ssl/certs/ca-certificates.crt
forward-zone:
name: "."
forward-tls-upstream: yes
forward-addr: 192.0.2.53@853#dns-upstream.example.net
Reading the block, line by line:
server:opens the global section.interface: 127.0.0.4is the address Unbound listens on for client queries (a loopback placeholder here; on a real resolver it is the address your clients are pointed at).tls-cert-bundlenames the file of trusted certificate authorities (CAs) that Unbound uses to check the upstream's TLS certificate. Without it the upstream cannot be authenticated.forward-zone:withname: "."means "the root", so every name that is not answered locally is forwarded rather than resolved by walking down from the root servers.forward-tls-upstream: yesmakes Unbound speak DoT (TLS) to the upstream instead of plain port 53.forward-addr: 192.0.2.53@853#dns-upstream.example.nethas three parts: the upstream's IP address,@853the port to connect to, and#dns-upstream.example.netthe name that the upstream's certificate must carry. The#namepart is what stops someone else answering on that IP: Unbound completes the TLS handshake only if the certificate is valid for that name.
unbound-checkconf printed unbound-checkconf: no errors in <file>, naming the file it checked. It checks syntax only; it does not contact the upstream.
- Configure managed endpoints by policy, and lock the settings. Firefox has a
DNSOverHTTPSenterprise policy withEnabled,ProviderURLandLockedkeys, so you can point it at the enterprise DoH endpoint and stop users changing it. Chrome'sDnsOverHttpsModepolicy takesoff,automatic(DoH first, with fallback to insecure) orsecure(DoH only, with no fallback, so resolution fails if the DoH server is unreachable), sosecurewith the enterprise template is the managed-device setting. An operating-system or browser setting that the user can change is not enforcement. - Block the unmanaged routes at the network edge. Deny outbound TCP 853 except from your resolvers (one firewall rule). For DoH, deny outbound connections to the public DoH providers you identify by address list or secure web gateway category, and refuse their hostnames at the enterprise resolver so a client that uses the system resolver to find them fails. A secure web gateway is a proxy that filters web traffic by site category, which is how "public DoH provider" can be a category you block. In the lab, an Unbound resolver configured with
local-zone: "dns.google." always_refuseand the same forcloudflare-dns.com.returned:
dns.google: status: REFUSED
cloudflare-dns.com: status: REFUSED
- Let clients discover the enterprise's own encrypted resolver. DDR (Discovery of Designated Resolvers, RFC 9462) lets a client that knows only the plain resolver address ask that resolver for the name
_dns.resolver.arpa.with record type SVCB (service binding, a record that says how to reach a service: target host name, protocol and port). The answer points to the same operator's encrypted endpoint. With a local-data entry in Unbound, the lab returned:
$ dig @127.0.0.1 _dns.resolver.arpa. SVCB +noall +answer
_dns.resolver.arpa. 300 IN SVCB 1 dns.corp.example. alpn="dot" port=853 ipv4hint=192.0.2.53
Read it as: priority 1, target host dns.corp.example., alpn="dot" the protocol (DoT), port=853, and ipv4hint an address to try without another lookup (all values are illustrative). The security rule is that for verified discovery the client must check that the encrypted resolver's certificate lists the IP address of the plain resolver it started from (RFC 9462 requires an iPAddress entry in the certificate's subjectAltName). The reason is that the plain query can be forged by anyone on the path, and a certificate naming that IP shows the owner of that address designated the encrypted endpoint. Advertising DDR means clients upgrade to your resolver rather than to a public one.
5. Give guests and personal devices a separate network with no promise of DNS visibility, so you are not trying to inspect devices you do not manage.
6. Write down the privacy rules. Say what the resolver logs, how long they are kept, who may query them, and that encrypted DNS is on for staff against outside observers. This turns a dispute into a documented trade-off.
Why not just block everything
Blocking 853 is cheap and reliable. Blocking DoH is a moving target: any HTTPS server can answer DNS queries, so address lists age, and TLS inspection (a gateway that decrypts and re-encrypts traffic using a certificate the company's devices trust) breaks pinned applications (apps that accept only their own expected certificate and so reject the gateway's) and adds cost and legal exposure. Block what is cheap (853, known provider hostnames and addresses), and rely on managed-endpoint policy for the rest. Some devices, such as unmanaged phones or applications that ship their own resolver, will still get through, and the answer to that gap is endpoint management, not more DNS rules.
Performance and privacy trade-offs
- Performance: A TLS or HTTPS connection costs extra round trips once, then queries reuse the connection, so steady-state latency is close to plain DNS. Enterprise resolver capacity must cover many long-lived TLS sessions. Cache hits at the enterprise resolver are faster than a cold lookup to a public service.
- Privacy: Employees' lookups are visible to the company on managed devices, which is a policy and legal decision, not just a technical one. Public encrypted DNS hides queries from the local network but hands them to the provider.
- Failure mode: With Chrome's
securemode, an outage of the enterprise resolver means no resolution at all, whileautomaticfalls back to insecure DNS. Choose consciously and monitor the resolver.
What is a delegation in DNS, and when is a glue record needed? Give an example where a domain's nameservers sit inside the domain itself and explain what breaks if the parent holds no glue for them.
Sample Answer
Direct answer
A delegation hands a part of the DNS tree to another set of servers: the parent zone publishes NS (name server) records at the zone cut (the point in the name tree where the parent's data ends and the child zone's data begins), naming the servers that are authoritative for the child zone. A glue record is an address record (A or AAAA) that the parent also carries for a name server whose name sits inside the delegated zone itself. It is needed to break a circle: to find ns1.example.parent a resolver must first ask the servers of example.parent, whose only server is ns1.example.parent. Without glue in the parent, a resolver that does not already have the address cached cannot reach the child zone at all.
Delegation and referral
The parent does not answer questions below the cut. It returns a referral: a response with no answer section, the child's NS records in the authority section, and the authoritative-answer (AA) flag not set (the AA flag is a bit in the reply header that a server sets only when it owns the zone the answer comes from, so a referral from the parent does not carry it). The resolver then repeats the question to one of those servers. The child zone's own NS records and address records are the authoritative copy; whatever the parent holds is a pointer.
The example, executed with BIND
The parent zone (called parent in a lab) delegates example.parent to two servers whose names are inside example.parent:
example.parent. IN NS ns1.example.parent.
example.parent. IN NS ns2.example.parent.
ns1.example.parent. IN A 192.0.2.1
ns2.example.parent. IN A 192.0.2.2
The last two lines are the glue. Asking the parent's server for a name below the cut without recursion (dig @127.0.0.1 +norec +noall +authority +additional www.example.parent A) returned:
example.parent. 3600 IN NS ns1.example.parent.
example.parent. 3600 IN NS ns2.example.parent.
ns1.example.parent. 3600 IN A 192.0.2.1
ns2.example.parent. 3600 IN A 192.0.2.2
The NS records came back in the authority section (the first two lines: who is responsible) and the glue in the additional section (the last two lines: extra records offered to save a lookup), so the resolver can query 192.0.2.1 immediately.
With the two glue lines removed, named-checkzone (BIND's zone-file checker) still reports the zone as loaded. A program's exit status is the number it returns to the shell, where 0 means success and non-zero means failure, and here it is 0. The run printed:
$ named-checkzone parent /z/parent-noglue.zone ; echo "exit=$?"
zone parent/IN: example.parent/NS 'ns1.example.parent' has no REQUIRED GLUE address records (A or AAAA)
zone parent/IN: example.parent/NS 'ns1.example.parent' (out of zone) has no addresses records (A or AAAA)
zone parent/IN: example.parent/NS 'ns2.example.parent' has no REQUIRED GLUE address records (A or AAAA)
zone parent/IN: example.parent/NS 'ns2.example.parent' (out of zone) has no addresses records (A or AAAA)
zone parent/IN: loaded serial 1
OK
exit=0
The REQUIRED GLUE lines are the ones that matter here; the checker also prints a second, (out of zone) warning per name server, and with the two glue lines restored only those second warnings remain. Every line is a warning and the run ends with OK and exit=0, so a pipeline that checks only the exit status passes a zone whose delegation cannot be followed. The referral for www.example.parent carried the same two NS records, with the header showing AUTHORITY: 2 and ADDITIONAL: 1 (the EDNS option pseudo-record only: EDNS, Extension Mechanisms for DNS, lets a message carry extra options such as a larger size limit, and dig counts that option record in the additional section), so no address records at all. With the glue present the same query showed ADDITIONAL: 3, the two glue records plus that one. The delegation is published and silently broken: that is why a warning-only check is dangerous in a pipeline that treats exit status 0 as success.
What breaks without glue
A resolver with a cold cache gets the referral, has two NS names and no addresses, and must resolve those names, which sends it back to the same parent and the same referral. It cannot make progress and returns SERVFAIL (a server failure response) to the client. Resolvers that already hold the server addresses from earlier lookups keep working until those cached records expire, so the failure looks intermittent and depends on who asks.
When glue is and is not needed
- Needed (in-domain name servers): the NS name is under the delegated zone, as above. RFC 9471 says an authoritative server must include all available glue for such names in a referral, and must set the TC (truncated) flag (a header bit meaning "this reply was cut off to fit the UDP size limit") if they do not fit, so that the client retries over TCP.
- Not needed (out-of-domain name servers): if the NS names are in a different zone, for example
ns1.dnsvendor.net, the resolver finds their addresses through that other zone. Parents often still include such addresses as an optimisation, but they are not required.
Operating it
- With a registered domain, the glue is held in the parent (the TLD, top-level domain, registry), and you manage it through the registrar, which calls them host records or glue records. Changing a name server's IP means updating the glue at the registrar as well as the A record in the zone.
- Keep the child's own A records for its name servers identical to the glue. A mismatch sends different resolvers to different addresses.
- Verify with
dig +norec @PARENT-SERVER child.zone NSand read the additional section.
Pitfall
Putting a domain's name servers inside the domain is legal and common, but it makes the registrar glue a hard dependency. Pairing one in-domain server with one out-of-domain server from a different provider removes the dependency and also gives provider redundancy.
You are paged: customers in one region cannot reach api.example.com. How do you isolate whether the cause is DNS, the network or the service itself?
Sample Answer
Direct answer
Split the path into three stages and test each one independently of the others: name to address (DNS), address reachable (network), and service healthy (the application). The fastest discriminator is to bypass DNS: if the request works when you give the client the address by hand but fails by name, the fault is in name resolution. If it fails both ways, test the path to the address and then the service. For a one-region problem, the first question is which resolvers those customers use and whether they get a different answer from the resolvers that work.
Ordered checks
1. Define the symptom. Collect from a real affected user, or a probe in the region: what error (NXDOMAIN, SERVFAIL, timeout, connection refused, TLS error, HTTP 5xx)? Which resolver and which IP did the client use? Which networks are affected? Different error types point at different stages, and the table at the end maps them.
2. Bypass DNS with the known-good address. Take the address from a working region (or from the service's own inventory) and force the name to it:
$ curl -s --resolve api.example.com:8080:127.0.0.2 http://api.example.com:8080/health
ok
--resolve host:port:address makes curl skip DNS and connect to that address with the right Host header and certificate name. From an affected location:
- Works by address, fails by name: the problem is DNS. Go to step 3.
- Fails by address too: the problem is the network or the service. Go to step 5.
3. Compare resolvers (DNS is suspect). Ask the region's resolver, then ask other resolvers and the authoritative servers directly, with recursion off so you read what each one holds:
$ dig @regional-resolver api.example.com A
$ dig @other-resolver api.example.com A
$ dig +short NS example.com
$ dig @ns-a.example.com api.example.com A +norecurse
$ dig @ns-b.example.com api.example.com A +norecurse
What the outputs look like in a lab (real runs against local test servers; the second server holds an older copy of the zone, the third has no copy):
$ dig @127.0.0.1 api.example.com A +noall +answer # resolver
api.example.com. 60 IN A 192.0.2.10
$ dig +noall +answer @127.0.0.2 api.example.com A +norecurse # authoritative, current
api.example.com. 60 IN A 192.0.2.10
$ dig +noall +answer @127.0.0.3 api.example.com A +norecurse # authoritative, stale
api.example.com. 60 IN A 192.0.2.99
$ dig @127.0.0.4 api.example.com A +norecurse | grep status # lame: no copy of the zone
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 8624
$ dig +short @127.0.0.2 example.com SOA
ns-a.example.com. hostmaster.example.com. 7 3600 900 1209600 60
$ dig +short @127.0.0.3 example.com SOA
ns-a.example.com. hostmaster.example.com. 6 3600 900 1209600 60
Healthy means every authoritative server prints the same answer and the same SOA serial (the version number, the 7 and the 6 above). A different address plus a lower serial is the stale server. status: REFUSED is the signature of a lame server, one that is listed for the zone but will not answer for it. Read the results in this order:
- All authoritative servers agree and answer correctly, but the regional resolver does not: the resolver's cache or forwarding path is wrong. Look at the TTL (time-to-live, how long a cache may keep an answer) in its answer.
- One authoritative server answers differently or not at all: a stale secondary or a lame server (one that does not answer for the zone). Compare SOA serials across them.
- No authoritative server answers from the affected region, but they do from elsewhere: the path to the authoritative servers is the issue, such as anycast routing (one address announced from many sites, so the network sends each client to a nearby one, possibly a failed one) sending that region to a failed node, or a firewall dropping UDP/TCP 53.
Reading the TTL. A lab run showed what a cache can hide. A resolver cached api.example.com (TTL 60 seconds), then the authoritative server was stopped. Steps 1 and 2 ran about four seconds apart, with the server still up; the server was stopped before step 3:
1. warm the resolver cache for api
api.example.com. 60 IN A 192.0.2.10
2. same name from resolver (cached, TTL counting down)
api.example.com. 56 IN A 192.0.2.10
3. a name not in cache
status: SERVFAIL
4. ask the authoritative server directly
;; communications error to 127.0.0.2#53: connection refused
The cached name still resolved with its TTL counting down while an uncached name returned SERVFAIL. So "it resolves for me" is not proof that DNS is healthy, and "it fails for those customers" may depend on whether their resolver had the name cached. A TTL that counts down confirms you are reading a cached copy.
4. Check the record types and the TTL interplay. Ask for A and AAAA separately (dig api.example.com AAAA). A client that prefers IPv6 and gets an AAAA record pointing at an unreachable address stalls even though A is fine. Follow any CNAME chain (a CNAME is an alias that points one name at another, so a chain runs name to name to address) with dig +noall +answer. Each link is cached on its own TTL, so a change to the final address shows up after that A record's TTL, while a change to where the CNAME points shows up after the CNAME's TTL. In the lab, www was a CNAME with TTL 300 pointing at api, whose A record had TTL 60:
$ dig @127.0.0.1 www.example.com A +noall +answer # resolver, first lookup
www.example.com. 300 IN CNAME api.example.com.
api.example.com. 43 IN A 192.0.2.10
$ dig @127.0.0.1 www.example.com A +noall +answer # three seconds later
www.example.com. 297 IN CNAME api.example.com.
api.example.com. 40 IN A 192.0.2.10
The two clocks count down separately, so the shortest TTL in the chain decides how soon the resolver must rebuild the chain, and the longest decides how long a change to that link can stay hidden. A failed lookup is held for the negative-caching time (a resolver remembering that the name did not exist), the lower of the SOA TTL and the SOA MINIMUM field (RFC 2308). If region-specific answers are used (geo or latency steering, or split-horizon DNS, where different clients get different answers depending on where they ask from), check that the region's resolver is being matched to the intended pool. After you fix a record, affected users only recover as caches expire. With a TTL of 3600 seconds, the last cache can hold the bad answer for up to an hour, which is why a TTL is lowered before a change, not during it.
5. DNS is fine, so test the network. Connect to the resolved address from the affected region:
- Test the TCP connection and time it:
curl -s -o /dev/null -w "%{time_connect} %{http_code}\n" https://api.example.com/health. A timeout in connect with a healthy DNS answer is a routing, firewall or load-balancer reachability problem. The first number is the seconds spent connecting and the second the HTTP status:
0.000053 200 # healthy: connected at once, service answered 200
0.000000 000 # broken: curl gave up (here after 3 s with -m 3), no connection, no status; curl exits with code 28
- Trace the path in both directions with
tracerouteormtrtoward the address, and compare with a region that works. Each line is one hop with its address and round-trip time. A hop that prints*is one that did not answer. In the lab:
$ traceroute -n -m 3 -w 1 -q 1 127.0.0.1 # healthy: the destination answers
1 127.0.0.1 0.012 ms
$ traceroute -n -m 3 -w 1 -q 1 198.51.100.7 # an address on a link where nothing answers
1 *
2 *
3 *
A path whose replies stop at a provider boundary (good hops, then only * lines) points at an upstream or peering fault, with the caveat that some routers never answer traceroute, so confirm with a real connection test.
- Check whether the failure is size-dependent, which suggests an MTU (maximum transmission unit, the largest packet a link carries without splitting it) or fragmentation fault: small requests work, large responses stall.
6. Network is fine, so test the service. Hit the health endpoint on the specific backend in that region, bypassing the load balancer if possible, then check load balancer target health, certificate validity, recent deployments and error rates for that region. If one backend pool or one region's load balancer fails health checks, the cause is the service even though DNS and the network are healthy.
Mapping symptoms to stages
| What you see | Most likely stage | Next move |
|---|---|---|
| NXDOMAIN (the name does not exist) from the regional resolver, correct answer elsewhere | DNS: stale cache or missing record at one authoritative server | Query each authoritative server directly, compare SOA serials |
| SERVFAIL (the resolver could not get a usable answer) for uncached names | DNS: resolver cannot reach or validate the authoritative servers | Query the authoritative servers from the resolver's network; check DNSSEC validity if signed |
| Resolves, but connection times out, and by-address also times out | Network | mtr to the address from the region |
| Resolves, by-address works, by-name fails after the cache expires | DNS record change or removal | Check the zone change history and TTLs |
| Connects, then 5xx or TLS error | Service | Backend health, certificate, deployment |
| Resolves to a different IP in the region than elsewhere | Geo steering or split-horizon answers | Check the steering configuration and the answer from each resolver |
Mitigate and verify
Fix the stage you found, for example restore the record, repair the stale secondary (rndc retransfer example.com on BIND), withdraw the broken anycast node, or fail the region over. Then verify from the outside: the same dig against the regional resolver returns the correct answer, the curl by name succeeds from the affected region, and error rates for that region return to baseline. Record the TTL of the old answer so you can tell customers when the last cached copy expires.
You migrated an organisation's mail to a new provider by changing MX records, but some senders keep delivering to the old hosts for hours. Why does that happen, and how would you have planned the cutover to avoid lost mail and allow a safe rollback?
Sample Answer
Direct answer
Senders keep using the old hosts because a mail sender looks up the MX record (MX = "mail exchanger", the DNS record that names the server that accepts mail for a domain) through its own resolver, and that resolver is allowed to keep the answer for the record's TTL (time-to-live, the number of seconds a cached answer may be reused). The TTL clock starts when each resolver cached the record, not when you edited the zone, so the last stragglers disappear one full old TTL after your change. To avoid lost mail you lower the TTL ahead of time, make the new provider ready before the switch, keep the old host accepting and relaying mail long enough, and keep a rollback that does not strand mail in the wrong place.
Why it happens (shown, not asserted)
This lab uses an authoritative server (the one holding the zone) and a separate caching resolver standing in for a sender's resolver. The MX TTL is 3600 seconds.
$ dig @127.0.0.1 example.com MX +noall +answer # port 53: the sender's caching resolver, which now caches it
example.com. 3600 IN MX 10 mx.old-host.example.com.
# admin edits the zone: MX now points at mx.new-host, serial bumped, zone reloaded
$ dig -p 5353 @127.0.0.1 example.com MX +noall +answer # port 5353: the authoritative server holding the zone
example.com. 3600 IN MX 10 mx.new-host.example.com.
# five seconds later, same sender resolver
$ dig @127.0.0.1 example.com MX +noall +answer
example.com. 3594 IN MX 10 mx.old-host.example.com.
The authoritative server already says "new", yet the resolver still hands out "old" with a TTL counting down (3594). Every resolver that cached the record at a different moment expires at a different moment, so mail trickles to the old host for up to one old TTL. Three things can stretch the tail further. The first affects every migration. The second applies only to senders whose resolver operator has raised a minimum. The third applies only when the migration also moves the domain's name servers.
- The TTL that matters is the one the record carried when it was cached. Lowering it on cutover day does nothing for caches already holding the long one.
- Some resolvers enforce a minimum TTL. Unbound has a
cache-min-ttlsetting (a floor on how short a cached TTL may be) whose default is 0 (disabled), but operators can raise it, so a few senders will hold answers longer than you published. - If you also moved nameservers, the delegation itself is cached by resolvers. A live query to the .com registry servers for example.com returned NS records with TTL 172800 (48 hours) and a DS record with TTL 86400 (24 hours). The NS records are the delegation, the list at the parent zone of which servers answer for your domain; the DS record, which exists only if the zone is DNSSEC-signed, is a fingerprint of your signing key. You cannot lower those, because the registry sets them. A resolver that cached the old delegation just before you changed it keeps sending queries to the old name servers for up to 48 hours, so a nameserver move needs a 48-hour overlap where both providers serve identical data.
Cutover plan
Assume the current MX TTL is 86400 s (24 hours) and the target is 300 s.
| When | Action | Check |
|---|---|---|
| T-7 days | Build the new provider: every mailbox, alias, group and shared mailbox exists. Publish the new provider's DKIM (DomainKeys Identified Mail, a signing key in DNS) selectors (a selector is the label that picks which key, as in selector1._domainkey.example.com). Add the new provider to SPF (Sender Policy Framework, the TXT record that lists who may send as your domain) next to the old sender. | swaks --to user@example.com --server mx.new-host.example.com (swaks is an SMTP test tool that sends one message straight to a named server) delivers to a real mailbox |
| T-7 days | Configure the old host's relay (accept, then forward onward without keeping a copy) and test it for a test address only. Keep storing real mail on the old host: until the MX changes, users still read the old mailboxes, so relaying their live mail early would send it to a store they are not reading. | Send a message to the test address via the old host, find it in the new mailbox |
| T-2 days, 09:00 | Lower the MX TTL from 86400 to 300. | dig @<authoritative-ns> example.com MX +noall +answer shows 300 |
| T-1 day, 09:00 | Confirm that 24 hours (one old TTL) have passed since the TTL was lowered, so no resolver still holds a copy cached under the 86400 TTL. | Queries via several public resolvers show a TTL no higher than 300 |
| T0, 09:00 | Change MX to the new host and, at the same moment, switch the old host's relay on for every address, then run a final incremental copy of mailbox contents from the old store to the new provider so nothing delivered before the switch is left behind. Remove the old MX, or leave it at a worse preference number as described in the paragraph on coexisting MX records. | Authoritative answer shows the new host, plus the old host at preference 20 only if you chose to keep it |
| T0 to T+7 days | Old host keeps accepting and relaying. Watch its inbound connection count fall, and the new provider's rise. | Old host's inbound count per hour trends to near zero |
| T+7 days | Decommission the old host once a full day shows only spam and no real senders. Raise the TTL back (for example 3600). | Final audit of the old host's logs |
Why seven days: RFC 5321 section 4.5.4.1 says a sending server MUST delay retrying a destination after an attempt has failed, that the retry interval SHOULD be at least 30 minutes, and that the give-up time generally needs to be at least 4-5 days. A sender whose message is queued against the old host can therefore legitimately try it for days. Seven days covers that.
Both MX records can coexist during the transition. RFC 5321 section 5.1 says lower preference numbers are tried first and equal-preference hosts must be randomised. Putting the old MX at preference 20 and the new at 10 sends nearly everything to the new host while leaving a safety net, because every host in the list delivers to the same place.
Rollback that does not lose mail
The reason for relaying from the old host is that mail lands in the new provider's mailboxes whichever server the sender picked. Rollback is then three actions. First, set the MX back to the old host (it propagates in about 5 minutes because of the 300-second TTL). Second, reverse the relay so the old host stores mail instead of forwarding it. Third, bring back the mail already in the new provider's mailboxes: everything delivered after T0 is stored there and not in the old store, so either set the new provider to forward its inbound mail to the old host for the rollback window or export those messages and import them into the old mailboxes. Skipping the third step leaves recent mail in a store nobody is reading. Decide the trigger in advance, for example "deferral or bounce reports from known partners in the first hour, or the new provider's inbound count not rising within 30 minutes of the change". A deferral is a temporary failure reply (an SMTP 4xx code) after which the sender queues the message and retries.
What actually loses mail
| Failure | Result |
|---|---|
| Old host shut down or answering 5xx (permanent failure) for mailboxes it no longer holds | Senders get a non-delivery report and the mail is gone for good |
| Old host accepts mail into a store nobody reads | Silent loss, the worst case |
| New provider missing an alias or group that the old host had | Bounces that start only on cutover day |
| TTL lowered on cutover day instead of two days earlier | Hours of mail to the old host, which is tolerable only because of the relay |
Trade-offs
A shorter TTL costs more queries to your authoritative servers during the window, which is negligible for one MX record. Leaving the old host as a relay for a week costs a server and patching for a week, which is cheap against lost customer mail. A single sharp cutover with no overlap is only acceptable when both stores are the same system.
A secondary DNS server has stopped picking up changes made on the primary. How do zone transfers work, what are the usual reasons they fail, and how would you debug and then lock transfers down?
Sample Answer
Direct answer
A secondary (also called slave) keeps a copy of a zone by comparing the serial number in the zone's SOA record (start of authority, the zone's header record) with the primary's, then pulling changes over TCP with a zone transfer. A secondary that stops updating is almost always one of four things: the serial did not increase, the secondary never heard or accepted the NOTIFY (the primary's "something changed" message), TCP port 53 is blocked, or the transfer is rejected by an access list or TSIG (transaction signature) check. Debug by comparing serials on both servers, then testing a transfer by hand from the secondary's own host, and lock transfers down by allowing them only to holders of a TSIG key.
How a zone transfer works
- Serial comparison. The SOA record holds SERIAL, REFRESH, RETRY and EXPIRE (RFC 1035 section 3.3.13). The secondary asks the primary for the SOA every REFRESH seconds. If the primary's serial is greater (compared with sequence-space arithmetic on an unsigned 32-bit number, not plain subtraction; the next paragraph shows what that means), the secondary pulls the zone. A failed refresh is retried after RETRY seconds. If nothing succeeds for EXPIRE seconds, the secondary stops answering for the zone.
What "greater" means (RFC 1982). The serial is a 32-bit counter, so it runs from 0 to 4294967295 and then wraps to 0. A seriali1counts as newer thani2wheni1 > i2andi1 - i2is less than 2^31 (2147483648), or wheni1 < i2andi2 - i1is more than 2^31. Normal case: the primary holds 2024010103 and the secondary 2024010102; the difference is 1, so the primary's is newer and the secondary pulls. Wrap case: the primary's counter has passed 4294967295 and wrapped to 0 while the secondary holds 4294967295; plain comparison says 0 is smaller, but the rule above says 0 is newer (the gap, 4294967295, is more than 2^31). Backwards case: a primary serial of 2023123101 against a secondary's 2024010102 differs by 887001, which is less than 2^31, so the primary's counts as older and is ignored. Jumps larger than 2^31 in one step (for example from 100 to 3000000000) also count as older. All of these were computed with Python against the RFC 1982 rule. This is why a date-counter serial (YYYYMMDDnn) can only be increased by small steps and a lowered serial is silently ignored. - NOTIFY. After a change, the primary sends a NOTIFY message (RFC 1996) to the servers in its notify set (the zone's NS servers plus any
also-notifyaddresses) so they do not wait a full REFRESH. A secondary that receives NOTIFY from an address that is not a known primary for the zone (not one of the addresses in that zone'sprimarieslist on the secondary) ignores it and logs an error (RFC 1996 section 3.10); it does not query the sender, so it learns of the change only when its own REFRESH timer fires. A NOTIFY from a known primary makes the secondary check that primary's SOA at once instead of waiting. NOTIFY is a shortcut. The refresh timer is the safety net. - Transfer. AXFR (full zone transfer) is defined over TCP only (RFC 5936), starts and ends with the zone's SOA record, and a general-purpose implementation is recommended to use TSIG to control it. IXFR (incremental transfer, RFC 1995) sends only the differences since the secondary's serial and falls back to a full transfer when the primary lacks that history.
- Authentication. TSIG (RFC 8945) signs each message with a shared secret and a timestamp, so the primary can accept transfers by key instead of by source address.
The lab used for the evidence below
Primary at 127.0.0.2, secondary at 127.0.0.3, BIND 9.20, zone example.com. The primary has allow-transfer { key xfer-key; }, also-notify { 127.0.0.3; }; the secondary has primaries { 127.0.0.2; } and server 127.0.0.2 { keys { xfer-key; }; }, with a key made by tsig-keygen -a hmac-sha256 xfer-key.
Debug in this order
1. Compare serials on both servers. This tells you at once whether the secondary is behind.
$ dig +short @127.0.0.2 example.com SOA | awk '{print "primary serial",$3}'
$ dig +short @127.0.0.3 example.com SOA | awk '{print "secondary serial",$3}'
dig +short prints just the SOA data, seven space-separated fields: primary server, contact mailbox, serial, refresh, retry, expire, minimum. For example ns1.example.com. hostmaster.example.com. 2024010103 7200 3600 1209600 300. The serial is the third field, which is why awk prints $3. A healthy pair prints the same number twice. With the primary at 2024010103 and a secondary that has not caught up, the lab printed:
primary serial 2024010103
secondary serial 2024010102
A lower number on the secondary means it is behind; the other steps find out why.
Equal serials but a missing record means the change was made without bumping the serial. In the lab, adding late IN A 192.0.2.30 to the primary and reloading without changing the serial left the secondary answering status: NXDOMAIN for late.example.com, because from its point of view nothing had changed. Fix: increment the serial on every change, or generate it (a date-counter such as 2024010102 or a Unix timestamp) in the deploy pipeline.
A serial that moves backwards is ignored. When the primary's serial was set to 2023123101 while the secondary held 2024010102, the secondary kept 2024010102 and never received the new record. Recovery is to raise the serial above the old value, or to follow the documented serial-reset procedure for your server software.
2. Is NOTIFY arriving, and from where? Check the secondary's log. A multi-homed primary (one with more than one IP address) may send NOTIFY from a different address than the one the secondary lists. Before the fix, the lab primary sent from 127.0.0.1 and the secondary logged:
received notify for zone 'example.com'
zone example.com/IN: refused notify from non-primary: 127.0.0.1#50034
Without NOTIFY the secondary only learns of the change at its next REFRESH, which is why this fault looks like "works, but slowly". Fix on the primary: notify-source 127.0.0.2; (and transfer-source 127.0.0.3; on the secondary for its own transfers). After the fix the log shows notify from 127.0.0.2#56137: serial 2024010103 and the transfer starts at once.
3. Is TCP/53 open both ways? UDP/53 may work while TCP is dropped, so SOA checks pass and NOTIFY arrives but the transfer hangs. In the lab, after iptables -A INPUT -p tcp --dport 53 -j DROP on the primary and a serial bump:
primary serial 2024010103
secondary serial 2024010102
$ dig +tcp +time=2 +tries=1 @127.0.0.2 example.com SOA
;; Connection to 127.0.0.2#53(127.0.0.2) for example.com failed: timed out.
The secondary logs notify from 127.0.0.2#56137: serial 2024010103 then Transfer started. and nothing further. Test from the secondary's host with dig +tcp @primary example.com SOA.
4. Is the transfer authorised? Request a full transfer by hand from the secondary's host, with the key:
$ dig @127.0.0.2 example.com AXFR +noall +comments # no key
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 3605
(flags and OPT lines omitted)
; Transfer failed.
$ dig @127.0.0.2 example.com AXFR -y hmac-sha256:xfer-key:<secret>
Without a key the lab primary returned REFUSED (a plain dig ... AXFR without +comments prints only ; Transfer failed., so add +noall +comments to see the status). With a wrong secret the transfer failed and the response carried TSIG ... BADSIG. Other TSIG failures to recognise (RFC 8945): BADKEY (key name or algorithm unknown to the server) and BADTIME (the timestamp is outside the allowed "fudge" window, the number of seconds the two servers' clocks may differ before a signature is rejected; 300 seconds is the recommended value, so the cause is clock drift between the two servers). BADSIG means the signature did not verify (usually a wrong secret). Fix clock sync with NTP (network time protocol), or correct the key name, algorithm and secret on both sides.
5. Force the retry once the cause is fixed. On BIND, rndc retransfer example.com on the secondary pulls the zone immediately. In the lab it exited with status 0 and the secondary moved from 2024010101 to 2024010105. Then re-check serials.
Lock transfers down
The first two items are the controls that decide whether a stranger can pull the zone: refuse transfers by default, and allow them only with a key. The remaining items add layers around those two.
- Set a global
allow-transfer { none; };inoptions, and open it per zone:allow-transfer { key xfer-key; };. In the lab an unsigned request was REFUSED while a signed one returned the zone, with the SOA record appearing as first and last record. - Use a key per secondary,
hmac-sha256or stronger (RFC 8945 discourages SHA-1), and rotate keys on a schedule. - Add a source-address allow-list at the network layer as well: TCP/53 from the secondaries only, and a firewall rule that lets the secondaries' source addresses reach the primary. The network rule limits who can reach the port at all, and the key limits who may pull the zone once they can, so each covers a gap the other leaves.
- Keep the primary hidden (not listed in the zone's NS records, so resolvers never query it and only the secondaries face the internet; the secondaries are the servers you publish).
- Monitor SOA serial parity across all name servers and alert when they differ for longer than REFRESH plus RETRY.
Why open transfers matter
An open AXFR hands out every hostname in the zone, including internal-looking names (vpn, jenkins, staging-db), which saves an attacker the guesswork of enumeration.
Unlock Full Question Bank
Get access to all 37 DNS, DHCP, and Name Resolution interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.