Amazon Staff-Level Network Engineer Interview Preparation Guide

Situation & goal
As a network engineer I must ensure automation systems (Ansible, Terraform, CI pipelines) access devices/cloud APIs without exposing long-lived secrets.

Secure methods

• Use centralized vaults: Store secrets in HashiCorp Vault or cloud secrets managers (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager). Enforce TLS, network policies, and IP allowlists.
• Ephemeral credentials: Issue short-lived API keys or certificates (Vault dynamic secrets, STS tokens) for device or cloud access to limit blast radius.
• RBAC & least privilege: Define roles for automation users (playbook runner vs. human admin). Map network actions to fine-grained policies (e.g., read-only show vs. config write).
• Secret rotation: Automate rotation (daily/weekly) for keys and device creds; use Vault leases and renewal workflows.
• Auditing & logging: Enable audit logs in vaults and cloud managers; forward to SIEM for alerting on abnormal access (time, source, frequency).
• Integration points:
  • Ansible: use HashiCorp Vault lookup plugins or cloud secret lookup; configure Vault auth backends (AppRole, OIDC) for playbook runners; avoid embedding creds in vars.
  • Terraform: use provider integrations with cloud secret stores or Vault data sources; use workspace-specific secrets and avoid state containing plaintext secrets.
  • CI/CD: Acquire ephemeral tokens at runtime via service principal or AppRole, inject into runners as environment variables, and ensure runner isolation.
• Operational practices: Rotate device admin passwords, use certificate-based SSH/TACACS+, document approvals, and test secret rotation on staging before production.

These measures reduce credential exposure, improve traceability, and align automation with security best practices for networking environments.

Overview & goals
I would create clear security zones (PCI, Production, Non‑Prod, Public DMZ, Management/Bastion, Shared Services) across on‑prem and AWS, enforce least privilege, and provide auditable controls for PCI scope and admin access.

Zones

• Public DMZ: Internet LB/edge firewall, public web servers (stateless).
• Production: App & DB subnets for prod VMs (private subnets).
• PCI zone: Dedicated subnets and VPCs/VRFs for payment systems; strict ACLs and limited ingress/egress.
• Non‑Prod: Separate VPCs/subnets for dev/test; no access to prod/PCI by default.
• Management: Admin workstations, bastion hosts (MFA), jumpbox subnet, monitoring, patching servers.
• Shared Services: DNS, AD, NTP, logging, SIEM (centralized).

Routing & Isolation

• Use VRFs / AWS Transit Gateway + route table segregation to enforce zone separation.
• Explicit deny default route between VRFs; only allow specific routes for necessary services (e.g., logging, backups).
• Use private VIFs or AWS Direct Connect for on‑prem ↔ cloud traffic with per‑VLAN/VRF segmentation.

Firewall placement & policy

• Edge firewalls at internet egress (NGFW) in front of LB in DMZ.
• East‑west NGFWs between zones: virtual FW instances in AWS (or AWS Network Firewall) and physical/virtual FWs on‑prem.
• Microsegmentation via host/endpoint firewalls (e.g., iptables, Windows FW) and security groups for VM-level controls.
• PCI zone sits behind an additional firewall tier with strict stateful inspection and limited NAT.

NAT & Load Balancers

• Public LBs in DMZ handle HTTP/S; terminate TLS there and forward to internal ALBs or private LBs in Prod.
• NAT Gateways for outbound internet access from private subnets; for PCI outbound, use controlled proxy NAT with egress filtering and allowlist.
• No inbound NAT to PCI systems; access only via bastion or highly restricted APIs.

Bastion & Admin Access

• Centralized bastion hosts in Management zone; enforce MFA (IdP integration), time‑boxed sessions, and just‑in‑time provisioning.
• No direct SSH/RDP from internet to Prod/PCI. Admin traffic flows: Admin workstation → Bastion → Target. All sessions proxied/recorded.

Enforce & Audit Controls

• Centralized log aggregation (SIEM) collecting FW logs, VPC flow logs, host logs, bastion session recordings, and IDS events.
• Implement automated policy enforcement with IaC (Terraform/Ansible) and NACM for network device configs.
• Continuous compliance scans (vuln scanner, PCI SAQ/ASV scans), periodic segmentation tests (internal pentests, firewall rule review).
• Use network telemetry and alerting (flow anomalies, unexpected inter‑zone flows). Maintain RBAC, change control, and immutable audit trails for firewall and route changes.

Tradeoffs

• Strong isolation increases operational overhead (extra VPCs/VRFs, firewall licenses). Use automation and centralized services to minimize burden while meeting PCI and separation requirements.

Approach (brief)
I’ll list descriptive NSG rules (human-readable), priorities, directions, protocols/ports, and target source/destination. Then explain priority numbering and implicit deny behavior.

NSG rules for web-app subnet

1. Allow outbound HTTPS

   • Priority: 300
   • Direction: Outbound
   • Source: Web-App-Subnet (CIDR or NSG tag)
   • Destination: Internet (or 0.0.0.0/0 or specific service FQDNs)
   • Protocol: TCP
   • Destination port: 443
   • Action: Allow
   • Description: Permit web app to call external HTTPS APIs

2. Deny outbound SSH from subnet

   • Priority: 310
   • Direction: Outbound
   • Source: Web-App-Subnet
   • Destination: Internet (0.0.0.0/0)
   • Protocol: TCP
   • Destination port: 22
   • Action: Deny
   • Description: Block direct SSH egress to reduce lateral/exfil risk

3. Allow inbound health-checks from load balancer range

   • Priority: 200
   • Direction: Inbound
   • Source: Public-LB-IP-Range (CIDR or AzureLoadBalancer tag if internal)
   • Destination: Web-App-Subnet (specific VM NICs or backend pool)
   • Protocol: TCP (or as required)
   • Destination port: <health-port> (e.g., 80 or custom probe port)
   • Action: Allow
   • Description: Permit LB probes for health check

4. (Optional) Allow established/return traffic

   • Priority: 320
   • Direction: Inbound
   • Source: Internet
   • Destination: Web-App-Subnet
   • Protocol: TCP
   • Source port: *
   • Destination port: ephemeral ports (1024-65535)
   • Action: Allow
   • Description: Allow response traffic for outbound HTTPS if needed (stateful in Azure often handles this)

Priority numbers & ordering

• Lower numeric priority runs first (100 is higher precedence than 200). Azure evaluates rules from lowest to highest number and stops at the first match. Choose ranges (e.g., 100–199 for inbound infra, 200–399 for allowed app traffic, 400+ for denies) to keep room for later rules.

Implicit deny

• NSGs have an implicit deny all rule at the end. If no rule matches a packet, it is dropped. Therefore explicitly allow required traffic and explicitly deny sensitive flows (like SSH) where necessary. Relying solely on implicit deny is fine, but explicit denies make intent and logging clearer.

Notes / Best practices

• Use service tags or FQDN tags where possible to limit broad 0.0.0.0/0 rules.
• Log NSG flow logs to monitor denied/allowed traffic.
• Test rules in a staging subnet before applying in production.

Direct answer
A decision genuinely reflects a stated value, rather than merely being compatible with it, when the value actually changed what you chose to do, not just how you described it afterward. The strongest answers make that causal link explicit: what you would have done differently if the value hadn't been a factor.

Structured elaboration

• Situation and task: the decision point, described briefly.
• The counterfactual test: name what the default, easier choice would have been, and what specifically made you choose differently.
• Action: what you actually did, including who you had to convince or coordinate with.
• Result: the outcome, and ideally a signal that the choice was validated rather than merely feeling principled at the time.

Worked example
Faced with a choice between shipping a quick, directionally useful analysis in time for a decision meeting, or spending an additional two weeks on a more rigorous version, the default and professionally "safer" choice would have been to wait for rigor. Choosing to ship the quicker, clearly caveated version instead, because the business decision had a hard deadline and a rigorous-but-late analysis would have been useless, shows a genuine trade-off rather than a reflexive one. The decision was validated when the more rigorous follow-up analysis, completed afterward, confirmed the same direction, meaning the faster call hadn't cost the business a wrong decision.

Trade-offs and pitfalls
A story where the value and the easy choice happen to be the same thing doesn't actually demonstrate anything, since no real trade-off was made; choose a story with genuine tension in it. Naming the value first and building a story to fit it, rather than the reverse, tends to produce something that sounds rationalized rather than genuine; a genuinely reflective answer usually names the counterfactual without being asked. A result stated only as "and it felt right" is weaker than any concrete validation signal, even an imperfect one.

Clarify goals & constraints

• Scale: thousands of hosts, realistic L2–L7 traffic
• Test types: throughput/latency, control-plane (BGP/OSPF/ECMP/SDN), convergence, failures
• CI friendly: repeatable, automatable, observable, resource-bounded

High-level architecture

• Lab/CI pipeline orchestrator (GitLab/GitHub Actions + Terraform/Ansible)
• Emulation layer: mix of lightweight network namespaces/containers (Linux netns, Docker) + accelerated dataplane (DPDK/VPP) for scale
• Traffic generators: TRex (stateless + stateful), Scapy/pktgen for microflows, Gatling/Locust for L7
• Control-plane simulators: FRR instances, Quagga, or virtual routers (VyOS) per namespace
• Observability/validation: Prometheus+Grafana, ELK, packet captures (tcpdump), flow collectors (sFlow/IPFIX)

Core components & responsibilities

• Provisioning: Terraform to spin VMs/containers and network topology templates (spine-leaf, BGP fabrics)
• Host scaling: instantiate thousands of netns with veth pairs or simulate many hosts via TRex/generative profiles if CPU-limited
• Traffic patterns:
  • Mice: Poisson arrivals, short-lived TCP/UDP, randomized endpoints
  • Elephant: Long-lived bulk flows, sustained high-throughput streams between selected endpoints
  • Use heavy-tailed distributions (Pareto) to mix sizes and inter-arrival times
• Control-plane testing:
  • Emulate route churn: scripted BGP session flaps, route advertisements/withdrawals
  • Validate convergence times, RIB/FIB consistency, route dampening effects
• Failure injection:
  • Link/node failures via ip link/netem, simulate device reboot, CPU exhaustion, partitioning
  • Chaos scheduler integrated into CI (Chaos Toolkit/Ansible)

Data flow & scenarios

• CI job deploys topology → baseline tests (connectivity, capacity) → ramp traffic (mice → mixed → elephant) → inject failures/churn → record metrics
• Post-run validators: check reachability matrix, packet-loss thresholds, CPU/memory, convergence SLA, flows rebalanced under ECMP

Scalability & optimizations

• Offload heavy packet generation to DPDK-enabled hosts or hardware (Ixia) to avoid host-count explosion
• Use host multiplexing (one container simulates many hosts via flow ID namespaces)
• Parallelize CI stages and use cached images

Metrics & success criteria

• Control-plane convergence time < target (e.g., BGP convergence < 5s)
• Packet loss < X% under normal load; latency P99 within SLA
• No routing inconsistencies (RIB vs FIB), no blackholes after failures

Trade-offs

• Full-fidelity hardware tests costlier but more realistic; software emulation is cheaper and CI-friendly
• Multiplexing reduces resource needs but can miss per-host state issues

Example tools: TRex, FRR, VPP/DPDK, Mininet/EVE-NG for topology modeling, Prometheus/Grafana, Chaos Toolkit.

Core elements of an SLA and matching SLOs for a cloud-managed VPN (network engineer view)

1. Availability / Uptime SLO

• Example: 99.99% gateway availability per month.
• Why: Enterprises rely on continuous connectivity; quantifies expected downtime and drives redundancy/HA design (active-active gateways, health checks).

2. Latency & Performance SLOs

• Example: 95th-percentile round-trip latency < 50 ms for regional peers; packet loss < 0.1%.
• Why: VPN impacts application performance; customers need guarantees for VoIP, real-time apps and capacity planning.

3. Throughput / Bandwidth & Scaling

• Example: Per-connection throughput up to X Gbps; auto-scale time < 5 min.
• Why: Ensures the service handles peak loads, supports capacity planning and outlines scaling behavior for burst traffic.

4. Security & Incident Response

• Example: 24/7 SOC monitoring, time-to-detect SLA < 15 min, time-to-mitigate < 2 hours for critical incidents; cryptographic standards (AES-256, IKEv2).
• Why: VPN is critical security boundary — customers require response SLAs and explicit encryption/authentication assurances.

5. Change & Maintenance Windows / Notification

• Example: Scheduled maintenance limited to defined windows with 72-hr advance notice.
• Why: Minimizes surprise disruptions and lets customers plan change controls.

6. Support & Remedies

• Example: Tiered response times (P1: 15 min), credits or termination rights for SLA breaches.
• Why: Defines escalation, accountability and commercial remedies when the service fails.

Each element ties to network design choices (redundancy, monitoring, DDoS protection, autoscaling) and gives customers measurable expectations.

Direct answer

Inheriting a messy, poorly understood system as its new owner calls for a 30/60/90-day plan with a deliberate shape: the first 30 days are almost entirely learning and risk assessment with no risky changes, the next 30 add a small number of low-risk quick wins that build trust while you're still learning, and the final 30 start the real medium-term fixes, with progress measured the whole time by a small set of health signals you define early, not by how much code has changed.

Structured elaboration

Days 1 to 30, learn and assess risk: map what actually exists (services, dependencies, who else touches this system), identify where undocumented behavior or missing tests create the biggest blind spots, and deliberately make no risky changes yet, the goal is an accurate risk map, not visible progress. Talk to whoever used to own it or adjacent teams that depend on it, since institutional knowledge not written down anywhere is exactly the risk this phase exists to surface.

Days 30 to 60, quick wins: pick 2 to 3 small, genuinely low-risk fixes discovered during the learning phase, ideally things that are visibly annoying to the team or reduce a clear, contained risk, such as a missing alert on an already-known failure mode or a manual step that's easy and safe to automate. These build trust and credibility precisely because they're small enough to be confident about, not because they're impressive.

Days 60 to 90, medium-term fixes: start the larger, riskier work the learning phase identified as actually mattering, such as adding real test coverage to the highest-risk undocumented path, or setting up CI (continuous integration, automated build-and-test on every change) if none exists. Sequence it so the riskiest changes happen with the safety net of whatever quick wins already shipped (monitoring, a rollback path), rather than being the very first thing touched.

Measuring real progress without destabilizing production: track a small number of concrete health signals defined in the first 30 days, an error or incident rate baseline, and a rough measure of how much of the system now has any test coverage versus none, and use those same signals throughout, not a shifting definition of progress. Real progress means those numbers move in the right direction while the incident rate does not get worse, not just that visible work is happening.

Worked example

In week 1, discover the system has no CI and three services with no documented owner besides "the previous person," and that on-call has been fielding an average of 4 pages a week related to one specific, undocumented failure mode. That failure mode and its frequency becomes the baseline health signal to track. Days 30 to 60, the first quick win is adding an alert with a documented runbook for that exact failure mode, not fixing its root cause yet, just making it visible and handled faster when it fires, which is low-risk (adding observability, not changing behavior) and immediately useful to whoever's on-call. Days 60 to 90, the medium-term fix addresses the failure mode's actual root cause, now done with the new alert already in place as a safety net if the fix doesn't fully resolve it. Progress at day 90 is measured against the same baseline: page frequency for that failure mode should be visibly down from the original 4-a-week baseline, not just "we shipped a fix."

Trade-offs and pitfalls

Trying to fix real problems in the first 30 days, before you actually understand the system, is the most common way a new owner destabilizes production early and burns the trust they were trying to build. The opposite mistake, spending 60 or 90 days purely learning with nothing shipped, reads as inaction to a team that needs to see the new owner is capable, the quick-wins phase exists specifically to avoid that trap. Picking a "quick win" that turns out to not be low-risk, because the learning phase missed a dependency, is the sharpest version of this failure, which is why the quick wins should specifically come from what the first 30 days actually surfaced, not from a generic list of common fixes applied without local context. And measuring progress by activity (commits, pull requests merged) instead of the health signals defined up front lets real problems keep degrading even while it looks like a lot is happening.

Approach (brief)
Use VRF-Lite to create isolated route/forwarding tables per tenant. Assign physical interfaces or subinterfaces into each VRF with ip vrf forwarding. Use vrf-aware static routes (ip route vrf ...) or a centralized "shared-services" VRF and controlled route-leaking (prefer MP-BGP for scalable, auditable leaking). Prevent leakage by never sharing interfaces between VRFs and only installing explicit route-leaks.

Configuration example (IOS)

Create VRFs:

ip vrf TENANT_A
 rd 1:10
!
ip vrf TENANT_B
 rd 1:20
!
ip vrf SHARED_SVC
 rd 1:30

Assign interfaces / subinterfaces:

interface GigabitEthernet0/0.10
 encapsulation dot1Q 10
 ip vrf forwarding TENANT_A
 ip address 10.0.0.1 255.255.255.0
!
interface GigabitEthernet0/0.20
 encapsulation dot1Q 20
 ip vrf forwarding TENANT_B
 ip address 10.0.0.1 255.255.255.0
!
interface GigabitEthernet0/1
 ip vrf forwarding SHARED_SVC
 ip address 192.168.100.1 255.255.255.0

Static routes in each tenant VRF to reach shared service (vrf-aware next-hop is the router SVC IP reachable via physical routing in that VRF):

ip route vrf TENANT_A 0.0.0.0 0.0.0.0 10.0.0.254      ! tenant default via local gateway
ip route vrf TENANT_A 192.168.100.0 255.255.255.0 10.0.0.2
ip route vrf TENANT_B 192.168.100.0 255.255.255.0 10.0.0.2

(If you need controlled inter-VRF routing, implement a route-leak via a router-on-a-stick or use MP-BGP import/export route-targets between VRFs on the same device — preferred for production.)

Prevent route leakage:

• Do not configure "ip vrf forwarding" on same interface for multiple VRFs.
• Avoid global routing-table redistribution into VRFs.
• Use route-targets + MP-BGP if selective import/export required (instead of ad-hoc static next-hops).

Verification commands:

show ip vrf
show ip vrf interfaces
show ip route vrf TENANT_A
show ip route vrf TENANT_B
show ip cef vrf TENANT_A
show ip cef vrf SHARED_SVC

Explaination & best practices

• Each VRF has its own RIB/FIB: overlapping 10.0.0.0/24 is fine because they live in separate tables.
• Use ip route vrf <name> for vrf-scoped static routes.
• For shared services, prefer a dedicated VRF and controlled leaking via MP-BGP (route-targets) to avoid manual static route sprawl and accidental leakage.
• Always verify per-VRF routes and CEF entries to confirm forwarding separation.

Direct answer
Selective Acknowledgment (SACK) lets a receiver tell the sender exactly which non-contiguous blocks of data it has ALREADY received, so after a loss the sender only has to retransmit the specific missing segment(s), not everything that came after it.

Structured elaboration
Without SACK, TCP uses cumulative acknowledgment: an ACK only confirms "I have received everything up through this byte, contiguously." If segment 3 of a 10-segment flight is lost but segments 4 through 10 all arrive fine, the receiver can only ACK up through the end of segment 2, it has no way to tell the sender "I actually already have 4 through 10, I'm just missing 3." A sender using only cumulative ACKs, upon detecting the loss, may end up retransmitting segments 3 through 10 (everything the receiver hasn't cumulatively acknowledged), even though 4 through 10 were never actually lost.

With SACK enabled (negotiated via a permitted option in the handshake, then carried on subsequent ACKs), the receiver's ACK can include SACK blocks, explicit ranges of sequence numbers it holds that are NOT contiguous with the main acknowledged run, in the example above, a SACK block spanning segments 4 through 10. Now the sender knows precisely that only segment 3 needs retransmitting.

Worked example
Say a sender has segments with sequence ranges [1000-1500), [1500-2000), [2000-2500), ... up to [4500-5000), and segment [2000-2500) is lost in transit while everything else arrives. Without SACK: the receiver's ACKs stay pinned at ack=2000 (the last contiguous byte received) even as segments up through 5000 keep arriving; the sender, upon detecting the loss (via duplicate ACKs all saying ack=2000), knows only that SOMETHING after 2000 needs resending and, in older/naive implementations, could resend everything from 2000 onward. With SACK: the same duplicate ACKs at ack=2000 now also carry a SACK block like sack=2500-5000, telling the sender explicitly that only the single segment [2000-2500) is actually missing, so it retransmits exactly that one segment and nothing else.

Trade-offs & pitfalls
SACK is most valuable on connections with a large amount of data in flight (a large window relative to segment size) and where losses are isolated rather than in a solid burst, since that's exactly the scenario where "retransmit everything after the gap" wastes the most bandwidth compared to "retransmit only the gap." On a connection with a tiny window, or where an entire flight is lost at once (nothing left to selectively acknowledge), SACK provides little advantage.

Direct answer

Flapping like this, tripping on a brief 1-minute blip and closing again a few minutes later, is almost always caused by an evaluation window that's too short and a threshold that's checked as a raw instantaneous rate instead of a smoothed one, with no hysteresis between the open and closed conditions. The fix is to require the elevated error rate to persist for a sustained period before tripping (smoothing or a minimum-duration requirement), and to require a different, lower threshold sustained for a while before closing again (hysteresis), so the breaker doesn't oscillate around a single threshold value that noisy traffic keeps crossing in both directions.

Why this specific symptom happens

A breaker that evaluates a short window (say a single 10 to 30 second bucket) against a fixed absolute threshold (say 3 to 5 percent) will trip the instant any one bucket crosses that number, regardless of whether the elevated rate is a real sustained problem or a one-minute noise blip from a handful of slow requests. Once it trips and the cooldown expires, it closes again because the very next window looks normal, and then trips again a few minutes later the next time normal traffic variance happens to produce another short blip above the same threshold. The breaker isn't wrong that error rate crossed 5 percent, it's wrong that a single brief crossing is sufficient evidence of a real outage worth failing traffic away from a service that's actually healthy most of the time.

Fixes, in order of how much they change behavior

1. Smoothing the signal. Replace the raw per-window rate with an exponential moving average (EMA), so a short spike gets damped rather than immediately crossing the threshold:

With $\alpha = 0.2$, a baseline error rate of 1 percent, and a 1-minute burst of 5 percent sampled every 10 seconds (6 samples):

Against a 3.5 percent trip threshold, the raw signal crosses it on sample 1 (5 percent, instantly), but the EMA doesn't cross 3.5 percent until sample 5, roughly 50 seconds into the burst. That 50-second delay is the point: it means a true 1-minute-and-done blip barely trips the breaker at all (it only just crosses right as the burst is ending), while a genuinely sustained failure keeps climbing well past threshold and trips decisively.

2. Hysteresis between open and close conditions. A breaker actually cycles through three states: closed (normal, calls flow through), open (tripped, calls are rejected outright without even trying the dependency), and half-open (a brief trial period after opening where a small number of requests are deliberately let through to test whether the dependency has actually recovered) before it's allowed back to closed. Use a different, lower threshold to close than to open, and require it sustained for a minimum duration, not a single good sample: for example, trip open at EMA > 3.5 percent sustained for 2 evaluation periods, but only close from half-open back to closed once EMA stays below 2 percent (not 3.5 percent) for 2 consecutive periods. This asymmetry is what actually stops the flap-then-immediately-reopen cycle, because closing requires meaningfully cleaner traffic than the level that caused the trip, not just traffic that's dipped fractionally under the same number.

3. Minimum sample size. Require a floor on request volume before evaluating the rate at all (for example at least 200 requests in the window); at low traffic volumes a handful of failed requests can swing the percentage wildly even though the absolute failure count is tiny, and no smoothing scheme fixes a rate computed from too small a denominator.

4. Gradual re-entry. When the breaker closes, ramp traffic back in (5 percent, then 20 percent, then full) rather than snapping straight back to 100 percent, so a dependency that's only marginally recovered doesn't get immediately re-tripped by the full traffic load the instant it reopens.

Trade-offs & pitfalls

Every one of these fixes trades detection speed for stability: a smoothed, hysteresis-gated breaker takes longer to trip on a real outage than a naive instant-threshold one, which is the correct trade for a dependency where false trips are expensive (unnecessary failover, alert fatigue), but the wrong trade for something where even a few seconds of cascading failure is unacceptable, so the constants here (alpha, thresholds, minimum duration) should be tuned against the actual cost asymmetry for that specific dependency, not copied from another service. A subtler pitfall is that smoothing and retries interact: if callers retry failed requests, each retry counts as an additional data point in the error-rate window, so a retry storm during a real degraded period can itself inflate the EMA further and trip the breaker faster than the underlying failure rate alone would justify, which is usually the desired outcome (retries are evidence something is actually wrong) but is worth knowing explicitly rather than discovering by surprise. Validating any change to these constants should happen by replaying real historical traffic traces (including past incidents and known-noisy periods) through the new policy offline, comparing false-trip rate and time-to-detect-real-outage against the old policy, before rolling the new thresholds out as a canary.