FAANG-Standard Network Engineer Interview Preparation Guide (Entry Level)

Direct answer

The core move in an incident explanation to a non-technical audience is separating three layers up front: what happened (in plain terms, no root-cause mechanism), what it meant for them (impact, in terms they already track), and what's being done about it, then deliberately leaving out anything that doesn't serve one of those three. Below is an incident where I did that under time pressure, including delivering it live to a mixed engineering-and-business audience.

What to include, what to leave out, and how to decide

• Lead with impact, not sequence. Legal, sales, and executives care about what happened TO THEM first, which customers, how long, what's the exposure, the technical timeline is useful evidence, not the headline.
• Deliberately exclude logs, stack traces, and internal service names; they add authority for an engineering audience and add nothing but confusion for this one. A useful test: if a detail doesn't change what the listener should do next, leave it out.
• Give the cause in one plain sentence with no jargon, something like "a recent configuration change made one of our systems too slow to respond to a partner service in time," rather than either omitting cause entirely (which reads as evasive) or over-explaining the mechanism.
• When delivering this live rather than in a written report, whether it's a hallway update or presenting a postmortem verbally to a room that mixes engineers and business stakeholders, pause after the impact statement for questions before moving to cause. People worried about impact can't absorb a root-cause explanation until that worry is addressed first.

Worked example

Situation: during a high-traffic sales period, our payment service began intermittently failing checkout requests for roughly ninety minutes. Legal, sales leadership, and the executive team needed an explanation quickly.

Task: explain what happened clearly enough for them to act, communicate with affected customers, assess any obligations, decide on immediate next steps, without either alarming them with irrelevant detail or minimizing the impact.

Action: I opened with impact, in the terms they track: which customers were affected, for roughly how long, and that the issue was fully resolved and being watched closely. I gave the cause in one sentence: a recent configuration change made our payment service too slow to respond to our external payment gateway in time, causing some checkout attempts to fail. I described what we did in plain terms (reverted the change, increased how long we wait before giving up on a slow response, added an automatic circuit breaker so a slow dependency can't cascade into a wider outage) and what we were doing next (a deeper review, with a fuller technical writeup available to anyone who wanted it). I left out the specific error codes, service names, and configuration parameter, none of which changed what legal, sales, or the executives needed to do next. I paused for questions right after the impact statement, before moving on, and answered a legal question about customer notification obligations directly instead of routing it back to engineering jargon.

Result: legal and sales left with a clear, accurate picture of exposure and could communicate confidently with affected customers; the executive team approved the follow-up work (the circuit breaker and review) without needing to dig into implementation detail themselves, and a fuller technical postmortem was made available separately for the engineering team that wanted the mechanism-level explanation. I learned that pausing for questions right after the impact statement, before cause, kept people from tuning out a cause explanation they weren't ready to hear yet.

Trade-offs and pitfalls

Leaving out technical detail can read as evasive if you do it silently; I said "I'm not going to walk through the technical internals here, I'm glad to share those separately" so the omission was visible on purpose rather than hidden. The other pitfall is understating severity to keep the room calm, that erodes trust the moment the real scope becomes clear later. State the honest impact even when it's uncomfortable, and let the "what we're doing about it" section carry the reassurance instead of the impact statement itself.

Situation / Task
In my role as a network engineer I must keep current on protocol changes, vendor features, and cloud networking while running day-to-day ops.

Learning routine & time allocation

• Weekly: 3–4 hours split — 1h RFCs/blogs (IETF, vendor release notes), 1h hands-on labs (vendor emulators, AWS/GCP free tier), 1–2h training/cert prep or webinars.
• Monthly: 4–8 hours for deeper study (new vendor platforms, advanced features) and cross-team brown-bag.
• Quarterly: Attend a conference or vendor summit and update runbooks.

How I validate & apply safely

• Lab first: replicate changes in an isolated VM/cloud lab or vendor sandbox.
• Automated tests: use IaC (Terraform/Ansible) and test suites to validate configs.
• Staged rollout: deploy to a canary segment during maintenance windows, monitor with KPIs (latency, error rates, flow logs).
• Change control: peer review, rollback plan, and post-change validation checklist.

Result: continuous learning that translates into low-risk, measurable production improvements.

Approach (step-by-step)

1. Confirm scope & collect quick signals

• Check monitoring/alerts (CloudWatch, Datadog): 95th/99th latency, RPS, error rate, host CPU/memory, network Rx/Tx.
• Identify if single server is affected vs. LB-target group.

2. Network checks (isolate network layer)

• Ping/traceroute from a healthy host to the server:

ping -c 5 <server-ip>
traceroute -n <server-ip>
mtr -c 100 <server-ip>

• Check open connections and retransmits:

ss -tna | grep <server-ip>
netstat -s | grep -i retrans

• Capture packets to see retransmits/RSTs/latency:

sudo tcpdump -i eth0 host <server-ip> and port <app-port> -w /tmp/capture.pcap

• Cloud: review VPC Flow Logs, ELB/NLB metrics (ActiveFlowCount, ClientTLSNegotiationErrorCount).

Short-term mitigations: remove server from load balancer target group or set LB weight to 0; route traffic to healthy instances.

3. Host checks (isolate OS/resource issues)

• CPU, memory, IO, network queues:

top -b -n1
vmstat 1 5
iostat -x 1 3
ethtool -S eth0
dmesg | tail -n50

• Disk saturation or NIC errors indicate host-level problem.

Mitigation: restart offending process/service or network interface; reboot as last resort; migrate instance (AWS: stop/start or replace with new AMI).

4. Application checks (isolate app layer)

• Application logs, thread dumps, GC pauses, database latency.
• Check application socket queues/backlog:

ss -ltnp | grep <app-port>
cat /proc/net/sockstat

• Simulate load locally (curl, hey, wrk) to reproduce.

Mitigation: restart app, scale out (add instances/autoscale), enable circuit breaker/3xx responses.

Decision logic

• If tcpdump shows packet loss/retransmits but host metrics are fine → network.
• If high CPU/IO or NIC errors on host → host.
• If OS fine but app threads blocked / long GC / DB latency → application.

Keep steps iterative, preserve captures/logs for postmortem.

Direct answer
Building reliability on top of UDP means implementing, by hand, the exact machinery TCP gives you for free: a sequence number to detect duplicates, explicit acknowledgments, and a retransmission timer. A single-bit (0/1) sequence number is enough for stop-and-wait specifically, because only one message is ever in flight at a time.

Structured elaboration (approach)
send_reliable sends the payload tagged with the current sequence bit, then blocks (with a timeout) waiting for a matching ACK; on a timeout it just resends the same packet, and on receiving an ACK for the WRONG sequence number (a stale ACK from a previous round) it keeps waiting rather than treating that as success. receive_reliable accepts a packet, immediately ACKs it (even if it's a duplicate, in case its own previous ACK was lost), and only hands NEW data (matching the expected next sequence bit) up to the caller, silently absorbing duplicates.

Worked example (code)

import socket, struct

HEADER = struct.Struct("!BB")  # (sequence bit, type: 0=DATA, 1=ACK)

def send_reliable(sock, dest, payload, timeout, max_retries=5, state={"seq": 0}):
    seq = state["seq"]
    packet = HEADER.pack(seq, 0) + payload
    sock.settimeout(timeout)
    for _ in range(max_retries):
        sock.sendto(packet, dest)
        try:
            data, addr = sock.recvfrom(4096)
        except socket.timeout:
            continue  # retransmit on timeout
        if len(data) < HEADER.size:
            continue
        ack_seq, ack_type = HEADER.unpack(data[:HEADER.size])
        if ack_type == 1 and ack_seq == seq:
            state["seq"] = 1 - seq
            return True
    return False

def receive_reliable(sock, state={"expected_seq": 0}):
    while True:
        data, addr = sock.recvfrom(4096)
        if len(data) < HEADER.size:
            continue
        seq, pkt_type = HEADER.unpack(data[:HEADER.size])
        if pkt_type != 0:
            continue
        payload = data[HEADER.size:]
        sock.sendto(HEADER.pack(seq, 1), addr)  # always ACK, even duplicates
        if seq == state["expected_seq"]:
            state["expected_seq"] = 1 - seq
            return payload, addr
        # else: duplicate, already ACKed above, loop for the real next message

This was executed against a deterministic loss-simulating wrapper (a UDP socket wrapper that drops a configurable fraction of outgoing packets using a seeded random generator, so the test is reproducible) sending 4 messages at loss rates of 0%, 30%, and 60%. At every loss rate tested, all 4 messages were delivered exactly once, in the original order, confirming both the retransmit-on-timeout path and the duplicate-suppression path work correctly under real, repeated loss.

Trade-offs & pitfalls (edge cases and complexity)
Complexity: with a single sequence bit and no pipelining, stop-and-wait can send only ONE unacknowledged message at a time, so throughput is bounded by one round trip per message (a real reliability layer would need a sliding window of sequence numbers, not just one bit, to use a high-bandwidth-delay-product link efficiently, exactly the same motivation as TCP's own window). Edge cases handled: a lost DATA packet (sender times out, retransmits), a lost ACK (receiver gets a duplicate DATA packet, re-ACKs it without re-delivering the payload to the application), and a delayed ACK arriving after the sender has already given up and retransmitted (the sender must ignore an ACK for the WRONG sequence number rather than treating it as confirmation, otherwise a stale ACK could be mistaken for acknowledging the NEXT message). What this exercise demonstrates: TCP is doing exactly this kind of bookkeeping (and much more, for a full sliding window, congestion control, and out-of-order buffering) on every connection, for free.

Situation & threat
CAM/MACTable exhaustion (or extreme MAC churn) occurs when an attacker or compromised VM farm floods a switch with many spoofed source MACs, forcing CAM eviction and causing traffic blackholing or flooding — a layer-2 DoS.

Hardware features

• Use switches with large TCAM/MACTable and hardware enforcement (ASIC-based policing).
• Enable MAC address aging tweak (increase aging only if safe) and hardware-based MAC move limits.

Switch configuration

• Port-security: set maximum MACs per access port (example Cisco):

interface Gi1/0/1
 switchport mode access
 switchport port-security
 switchport port-security maximum 2
 switchport port-security violation restrict
 switchport port-security mac-address sticky

• For trunks/host links to hypervisors, set higher sensible limits and use sticky or static MACs.
• Enable DHCP snooping, Dynamic ARP Inspection (DAI), and IP Source Guard to prevent spoofing.
• BPDU Guard, Root Guard and storm-control to limit broadcast/multicast.

Rate-limiting & policing

• Apply ingress MAC learning rate-limit or control plane policing (CoPP) to drop excessive new-mac learning attempts.
• Use storm-control or per-VLAN/port policers to cap unknown-unicast flooding.

Monitoring & alerting

• Monitor MAC count per VLAN/port via SNMP, telemetry, or NetFlow; set alerts when MACs/MAC-change-rate exceed thresholds.
• Log port-security violations and SNMP traps; use SIEM for correlation.
• Track MAC churn metrics (MACs learned/min) and MACTable utilization.

Operational response

• Automated: quarantine offending port (shutdown or move to quarantine VLAN), syslog/SNMP alert, disable port-security violation to restrict.
• Manual: isolate hypervisor, suspend VMs, collect PCAP, review hypervisor promiscuous mode and vSwitch configs, check for VM compromise.
• Long-term: enforce VM-level controls (security groups, microsegmentation), adjust MAC limits per host, inventory static MACs, run periodic audits.

Trade-offs

• Tight limits increase operational overhead with VM mobility; tune per host (higher on hypervisors) and use automation to adapt limits.

Summary & goals
Standardize telemetry across cloud, edge, and on‑prem to: unify metrics/logs/traces, reduce alert fatigue by 60%, and introduce SLOs for key network services (WAN links, VPN auth, SD-WAN pathing).

Technical choices

• Collectors: OpenTelemetry agents for traces/metrics, Fluent Bit at edge/on‑prem for logs; use Telegraf for SNMP/NetFlow integration.
• Protocols: OTLP for metrics/traces, gNMI/gRPC and SNMP for device telemetry, syslog(via TLS) for logs.
• Storage/processing: Prometheus (federated) + Thanos for long-term metrics; Elasticsearch for logs; Jaeger/Tempo for traces. Alerting via Alertmanager and a unified dedupe/route layer (Opsgenie/PagerDuty). Use Kafka as telemetry buffer in unreliable edge links.

SLOs & SLIs

• Examples: WAN latency SLI = p95 latency < 120ms; SLO = 99.9% monthly. VPN auth success rate SLO = 99.95%. Define error budgets and burn policies.

Rollout phases

1. Pilot: 2 sites — deploy OTEL + Fluent Bit, export to sandbox Prometheus/ES, define 3 SLOs.
2. Expand: federate collectors to majority of sites, enable buffering, standard alert rules.
3. Enforce: full rollout, deprecate old agents, automate onboarding.

Change management

• Cross‑functional working group (network, SRE, security, apps).
• Runbooks, training sessions, and phased cutovers.
• Gradual rule migration: shadow alerting for 2 weeks then enable.
• Feedback loop: weekly reviews, tweak thresholds.

Success metrics

• Alert volume reduction: target −60% noisy alerts (measured by alerts/week).
• MTTR decrease (target −30%).
• SLO compliance % and error budget burn rate.
• Precision of alerts (true positive rate) and mean false positive rate.
• Coverage: % of devices/sites sending OTLP/metrics.

This plan balances network protocols and device realities, provides buffering for edge instability, and uses shadowing and training to manage change while measuring success quantitatively.

How evaluation differs

• Ordered rule list: rules are checked top-to-bottom; first match wins. Order encodes precedence; a specific allow below a broad deny will never be reached if broad deny is earlier.
• Set-based (specificity-driven): all candidate rules that match a packet are considered; engine selects the most specific rule (or resolves by priority tie-breakers). Specificity is computed from rule attributes (prefix length, port ranges, tags).

Operational benefits of set-based evaluation

• Predictability: precedence is explicit (most specific) instead of implicit order — easier reasoning about intent.
• Reduced rule-count and drift: overlapping broad rules + exceptions collapse into fewer, clearer policies (intent-based).
• Parallel/eager evaluation: engines can index by attributes (prefix trees, port indexes) for faster lookup at scale.
• Safer changes: adding a specific exception doesn't require reordering; lower risk of accidental shadowing.
• Better analytics: clarity on why traffic matched (specificity traces) simplifies troubleshooting.

Migration challenges

• Semantic gaps: many access-controls depend on order; translating to specificity requires formalizing “intent” (what was the intended exception).
• Shadowed/implicit rules: buried rules that relied on order may disappear or change effect — discovery needed.
• Complexity of specificity metrics: defining what counts as “more specific” across dimensions (IP, ports, application, user) and tie-breakers.
• Testing & rollout: need exhaustive test traffic, staged deploys, and fallbacks; audit logs must be mapped.
• Tooling and staff retraining: existing tooling/scripts assume ordered semantics; operational runbooks must be updated.

Practical migration steps

1. Inventory and label rules (intent, owner, criticality).
2. Identify order-dependent patterns (catch-alls before specifics).
3. Define specificity model and tie-break rules.
4. Convert rules iteratively: simulate with replayed traffic, verify matches, run in shadow mode.
5. Gradual cutover with monitoring and quick rollback.

This approach minimizes risk while capturing the operational advantages of set-based policy engines.

Direct answer

I own it as my call, in exactly those words, to whoever was affected, before anyone else has to describe it for me. Then I separate what I got wrong from what simply happened, because in a real incident the honest answer is usually a reasonable decision made on incomplete information, not recklessness. Learning from it means changing the actual process or system that let a reasonable-sounding decision go wrong, then proving that change works rather than just promising it will.

Structured elaboration

• Owning the mistake: state plainly, to the people affected, that the decision was mine, what I based it on, and what it caused, without folding it into vague passive language like "issues arose." Owning it before someone else has to point it out is what keeps the conversation focused on fixing the problem rather than assigning blame.
• Communicating with stakeholders: the people whose data or work was affected get told directly, not just informed via a general incident summary, and told what is known, what is still uncertain, and what happens next. If the honest answer is that the full extent isn't known yet, saying that plainly is better than a confident guess that turns out wrong a second time.
• What I learned: separate the surface lesson (don't do that specific thing again) from the deeper one about the decision-making itself, usually something like: what information was I missing at decision time, and was there a way to get that information faster instead of deciding blind.
• Concrete changes: turn the lesson into something that doesn't depend on me remembering it under pressure next time. Fixes that only live in memory ("I'll be more careful") don't survive a middle-of-the-night page (an automated on-call alert that pulls you back into the same situation) with the same instinct pulling the same way; a check, a changed default, or a required confirmation step does.
• Follow-through and verification: implementing the change isn't the end. Confirm it actually closes the gap, ideally by finding a case, real or deliberately constructed, where the old version of me would have made the same wrong call, and checking that the new safeguard now stops it, rather than assuming a written policy will simply be remembered.

Worked example

During a partial database outage, our primary was returning errors for a subset of writes. Under pressure to restore full service, I decided to fail over to the secondary, believing replication was current because the lag metric I checked, showing how far behind the secondary copy of the data was from the primary, read under a second moments earlier. What I didn't check was that the specific table under load was on a separate, higher-lag replication path the dashboard I glanced at didn't surface, and it turned out to be several minutes behind. The failover restored service, but a window of writes that had succeeded on the primary was missing on the new primary, so a batch of legitimate customer actions from that window silently disappeared instead of erroring visibly.

I found this within the hour during a routine post-failover check, and told the affected team directly rather than waiting for someone to notice a discrepancy on their side: what I did, why I believed the data was current at the time, exactly which window and which table were affected, and that I didn't yet know if the missing writes were recoverable. I worked with them to identify every affected record from application-side logs and manually replayed what could be reconstructed, then explicitly told them which handful of records couldn't be recovered at all, rather than implying we'd fully fixed it once the replay was done.

What I learned wasn't "check replication lag more carefully," which is too vague to act on, but that I'd trusted a single aggregate lag metric without confirming it covered every table involved in the failover, when in fact different tables replicated on different paths with different lag. The concrete change I implemented was a pre-failover check script that queries per-table replication lag for every table in the affected schema and refuses to proceed with an automated or manual failover unless every one of them is under a defined threshold, rather than relying on a single dashboard number. To verify it actually closed the gap, I deliberately reproduced the original condition in a staging environment, artificially lagging one table's replication path while keeping the aggregate metric low, and confirmed the new check correctly blocked the failover on that exact scenario before I considered the fix done.

Trade-offs and pitfalls

The natural instinct after a mistake like this is to over-correct into a rule so broad it slows down every future incident, such as always waiting several minutes before any failover, which trades a rare severe cost for a routine one. A better fix targets the specific gap in the specific decision that went wrong, here per-table lag rather than a blanket delay. The other pitfall is stopping at "I told the team and we replayed the data" as if that's the complete story: without the deliberate reproduction step, there's no real evidence the new check works, only a belief that it should, which is the same category of overconfidence that caused the original miss.

What RPKI / Origin Validation is (high level)
RPKI is a PKI system that binds Internet number resources (ASNs, IPv4/IPv6 prefixes) to cryptographic certificates issued by Regional Internet Registries. Origin validation uses Route Origin Authorizations (ROAs) from RPKI to check whether a BGP route’s originating AS is authorized to announce a prefix.

How ROAs are created & published

• Resource holder creates a ROA via their RIR or delegated CA, specifying: prefix (or max-length) and authorized origin AS.
• ROA is signed and uploaded to RPKI repositories (RIR/hosted).
• RPKI validators fetch repositories, validate the certificate chain, and produce a locally accessible validated cache (RTR/JSON).
• Routers pull this validated cache (RTR protocol or RPKI-to-Router) to perform origin validation.

Router validation results & meaning

• Valid: an exact match exists — prefix and origin AS authorized. Safe to accept.
• Invalid: a ROA exists that covers the prefix but the origin AS (or length) doesn’t match — indicates potential hijack or misconfiguration.
• Not-found: no ROA covers the prefix — no authoritative assertion; treat neutrally.

Operational caveats (practical points)

• ROA coverage limited: many prefixes lack ROAs, so origin validation only protects covered space.
• Propagation delay: certificate/ROA publication and validator cache updates take time; newly created ROAs may not be immediate.
• Risk of rejecting invalid routes: misconfigured or overly-specific ROAs can mark legitimate routes invalid — be cautious before auto-rejecting invalids; prefer staged policies (monitor -> reject), use prefix/aggregate ROAs, and maintain good change/coordination practices.

Situation & goals (brief)
Design DNS for hybrid cloud so internal-only services remain private, public services resolvable externally, and service discovery supports dynamic services securely.

Key concepts

• Split-horizon DNS: Maintain separate internal and external zones with same names but different records; internal views include private IPs and SRV records, external view exposes public A/AAAA or proxied endpoints.
• Forwarding: Internal DNS servers forward unknown queries to upstream (e.g., cloud DNS or ISP). Use stub/forwarders to reduce recursion and improve performance.
• Conditional forwarding: Route queries for specific domains (on-premises, cloud VPC) to the authoritative DNS in that environment—useful for .internal.company or cloud.local zones.
• Service discovery: Use SRV records or a service registry (Consul/Eureka) for dynamic ports/IPs; registries integrate with DNS (DNS interface or DNS-SD) for health-aware responses.

Design (practical)

• On-prem DNS cluster runs split-horizon views: internal zone with private records + SRV for services; external authoritative DNS (managed DNS provider) holds public records.
• Conditional forwarders: on-prem DNS forwards cloud-specific zones to cloud VPC DNS (or Private Hosted Zone) and cloud DNS forwards on-prem zones over secure DNS peering.
• Service registry for dynamic services: deploy Consul with ACLs and mTLS; expose service entries into internal DNS via Consul DNS interface. Use SRV for protocols needing port discovery.
• Security: DNS over TLS/HTTPS between sites or DNS forwarding over IPsec/VPN; DNSSEC externally; RBAC and ACLs on registries; restrict zone transfers (AXFR) to authorized servers; firewall rules to permit only trusted resolvers.

Example flow
Client queries db.internal.company → internal view returns private A and SRV via local DNS/Consul. External client queries api.company.com → public DNS returns load balancer IPs. Cross-site lookup for svc.cloud.local → conditional forward to cloud VPC DNS over secure peering.

Trade-offs

• Split-horizon adds operational complexity and risk of drift—automate with IaC.
• Registries add reliability and security overhead but enable health-aware discovery.

This design keeps internal services private, exposes only intended services publicly, and provides scalable, secure service discovery across hybrid environments.