Google Site Reliability Engineer (Mid-Level) Interview Preparation Guide

Direct answer
Running a blameless postmortem meeting well is mostly about preparation and framing, not clever facilitation tricks in the room. Before the meeting: assemble a factual, timestamped timeline from logs, dashboards, and deploy history, invite the people who were actually involved plus anyone who owns a system in the causal chain, and share a draft timeline in advance so the meeting starts from shared facts instead of competing memories. In the meeting: state the ground rules explicitly (we are here to understand the system, not to find who to blame), walk the timeline together, surface root cause and contributing factors as a group, and end with specific, owned, dated action items written down before people leave.

Structured elaboration

• Before: Pull raw evidence (metrics, logs, traces, deploy and change history) into a draft timeline. Doing this before the meeting, rather than reconstructing it live, keeps the discussion from turning into a memory-recall exercise, which is exactly where blame tends to creep in.
• Framing at the start: Explicitly name the ground rule. A single leading question like 'someone must have known this was a problem, why didn't anyone raise it?' is enough to make people defensive within seconds and shut down honest disclosure for the rest of the meeting, so the facilitator has to actively watch for and redirect that kind of framing, not just hope it doesn't come up.
• During: Walk the timeline chronologically, ask 'what made this possible' rather than 'who did this,' and treat 'human error' as the start of an investigation rather than its conclusion, since a person's reasonable action being unsafe is itself evidence of a system gap.
• Assigning action items: Every action item gets a single named owner and a date before the meeting ends. 'The team will look into X' produces nothing; 'Priya will add schema validation to the deploy pipeline by the 15th' produces something trackable.
• After: Circulate the finished writeup, and treat the meeting output as a living document only until the action items are confirmed done, not indefinitely.

This same structure holds even when the failure being reviewed is not a software outage. A postmortem for a failed partnership launch or a research study that led to a wrong product decision follows the identical discipline: timeline, impact, root cause versus contributing factors, and owned action items, adapted to a business rather than a technical vocabulary.

Worked example
A production incident: a deploy caused a spike in checkout failures. A poorly-run version of this meeting opens with 'who approved this deploy?' and spends 20 minutes on defensive explanations. A well-run version opens with a shared timeline already on screen, the facilitator asks 'what in our deploy process let a change with this blast radius reach 100% of traffic without a canary stage,' the group identifies that canary deployment was skipped because the on-call playbook doesn't clearly require it for config-only changes, and the meeting ends with two action items: update the playbook to require canary for all changes touching this service, owner and date named, and add an automated gate that blocks a full rollout if canary metrics haven't been checked, owner and date named.

Trade-offs and pitfalls
The most common failure mode is drifting from 'what happened' into 'who is responsible' the moment the timeline reaches a specific person's action. The facilitator's job is to notice that drift in real time and redirect toward the system gap that let the action cause harm. A second failure is ending the meeting with vague, unowned action items that read like good intentions rather than commitments; if nobody can point to a name and a date, the item will not get done.

Situation: 0.5% of users experience p99 spikes caused by queuing and head-of-line blocking in a downstream service. Below are system-level mitigations, their expected impact on latency, throughput, cost, and implementation notes.

1. Hedged requests (duplicate-request policy)

• Latency: Reduces p99 significantly by racing a second request after a short delay (e.g., 10-50ms) — often cuts long tails.
• Throughput: Increases downstream request rate (extra load proportional to hedging frequency); negligible for rare tails, larger if aggressive.
• Cost: Higher CPU/network and request charges; potential extra cost for downstream services.
• Notes: Use adaptive hedging (only when latency > threshold), idempotency or de-duplication at receiver, measure wasted requests and back off under sustained overload.

2. Priority queues (request classification + differentiated scheduling)

• Latency: Lowers p99 for high-priority traffic by avoiding head-of-line blocking from expensive lower-priority jobs.
• Throughput: Overall throughput unchanged but lower-priority requests may see higher latency; avoids systematic tail amplification for critical flows.
• Cost: Small infra/software cost to implement; possible need for more instances if lower-priority backlog grows.
• Notes: Define clear priority rules, fairness controls, starvation prevention (weighted fair queuing).

3. Admission control (reject/ shed low-value work when overloaded)

• Latency: Improves p99 for accepted requests by preventing queues from growing; rejects increase error rate but protects latency SLO.
• Throughput: Reduces accepted throughput under overload; keeps system operating within capacity.
• Cost: Saves backend resources; may require capacity planning to align SLOs with business priorities.
• Notes: Implement graded degradation, circuit breakers, and clear client signals (429/503 + Retry-After).

4. Timeouts and retries (client-side timeouts + backoff)

• Latency: Prevents clients from waiting indefinitely; reduces cascading queues if combined with proper retry/backoff.
• Throughput: Avoids request pile-up from long waits; aggressive retries can worsen load if not exponential and capped.
• Cost: Minimal; careful tuning avoids wasted work.
• Notes: Use conservative timeouts for p95 and shorter for p99-sensitive flows; implement jittered exponential backoff and retry budgets.

5. Client-side fallbacks (cached responses, degraded features, stale read)

• Latency: Eliminates many downstream calls, dramatically improving p99 for affected users.
• Throughput: Lowers downstream load, improving overall system headroom.
• Cost: Development/operational cost to build and maintain fallbacks and cache invalidation; potential consistency trade-offs.
• Notes: Use TTLs, stale-while-revalidate, and clear UX signaling when degraded.

Recommendation: Combine measures — prioritize critical traffic, apply conservative timeouts + retries, add hedging selectively for rare long tails, and implement admission control + fallbacks for overload protection. Instrument: measure p99, hedged hit rate, wasted work, rejection rate, and business impact (errors vs latency) to tune policies.

p50 (median) shows the latency experienced by the “typical” request, but it ignores the tail — the small fraction of requests that are much slower and often drive user dissatisfaction and SLO breaches. For user-facing services the tail matters because a few slow requests can ruin perceived performance (page jank, timeouts, retries).

Example:

• p50 = 100 ms (half of requests <=100 ms)
• p90 = 300 ms
• p99 = 2,000 ms
  These numbers tell two stories: p50 suggests excellent performance, but p99 reveals that 1% of users wait 2 seconds — long enough to trigger timeouts or abandonment.

Practical consequence of relying only on p50:

• You may miss degrading tail behavior and fail to alert until customer complaints or SLO breaches occur. That leads to surprise incidents, missed SLAs, and higher user churn.

Best practice: track multiple percentiles (p50, p90, p95, p99), distributions/histograms, and define SLIs/SLOs focused on relevant tails.

Cloud and network-backed storage for Kubernetes splits into two families: block storage (AWS EBS, GCE PD, Azure Disk), which is fast and durable but normally attachable to only one node at a time, and network filesystems (AWS EFS, Azure Files, self-managed NFS), which are shareable across many nodes at once but pay a latency and throughput cost for that flexibility. Picking between them is really picking whether the workload needs raw single-writer performance or multi-node shared access.

Comparison

How to choose

• Single-writer, latency-sensitive, durable (a relational database's primary, a message queue's log): block storage (EBS, GCE PD, Azure Disk). Access mode ReadWriteOnce, sized and provisioned for the IOPS the workload actually needs.
• Shared, multi-reader-or-writer, latency-tolerant (shared configuration, static assets, a CI build cache used by many concurrent jobs): a managed network filesystem (EFS, Azure Files) if available on your cloud, or self-managed NFS if not, understanding that NFS's durability and availability are now your responsibility to engineer.
• Regional or multi-zone resilience for a block-storage workload: look at the provider's own cross-zone replication option (GCE's regional Persistent Disk is the clearest example) rather than assuming ordinary zonal block storage survives a zone failure; ordinary zonal EBS/PD/Azure Disk does not.

Worked example

A team needs (a) a primary Postgres (Postgres) volume and (b) a shared directory of report templates read by 20 replica Pods across multiple nodes.

• (a) is single-writer and latency-sensitive: provision an EBS/GCE PD/Azure Disk volume through a StorageClass with ReadWriteOnce, sized for the database's IOPS profile.
• (b) needs concurrent multi-node reads: provision an EFS/Azure Files/NFS volume through a StorageClass supporting ReadWriteMany, since a block-storage volume cannot satisfy that access pattern at all, regardless of performance tier.

Trade-offs and pitfalls

• Don't reach for a network filesystem by default "to be safe" for multi-node access; if the workload is genuinely single-writer, block storage's lower latency is the better fit and the shared-filesystem's variability is pure downside.
• Zonal block storage (the common case for EBS/GCE PD/Azure Disk) does not survive the loss of its Availability Zone; if that's a real requirement, either use the provider's cross-zone replicated variant where one exists, or handle replication at the application layer (e.g., a database's own streaming replication to a replica in another zone) rather than assuming the storage layer covers it.
• A "multi-writer" flag on a block-storage product is not the same guarantee as a real shared filesystem: it typically still requires a cluster-aware filesystem and application-level write coordination, so verify exactly what's supported for your disk type before relying on it, rather than assuming ReadWriteMany-equivalent behavior.

Direct answer

The strongest version of this story shows a blocker noticed before it became someone else's emergency, an action matched to how much authority you actually had over what you changed (fixing it directly if it was fully yours, coordinating if it touched shared infrastructure, escalating only if it was genuinely outside your reach), and a real, specific before-and-after change in the delivery timeline, not a general sense that things felt smoother.

Structured elaboration

1. Spotting it: point to a concrete detection signal, not a vague feeling, for example a recurring delay pattern that showed up in the same way across the last several delivery cycles, or a piece of infrastructure that everyone quietly routes around because nobody actually owns fixing it.
2. Matching the action to your authority: a fix fully within your own control (your own code, your own team's process) can be built and shipped directly. Something that touches shared infrastructure or another team's system calls for coordinating with the right people first, rather than unilaterally changing something you do not own.
3. When to loop others in: loop in the owner of anything outside your own immediate scope before changing it, not after, and loop in your manager or affected stakeholders whenever the fix will visibly shift timeline expectations, even a positive shift, so nobody is caught by surprise either way.
4. Measuring the result: close with an honest, specific before-and-after comparison of the exact thing that changed, a step in a process, a recurring wait time, a manual task that used to eat calendar time, described plainly rather than with inflated precision.

Worked example

Across the last four release cycles, a manual, ticket-based approval step to provision a test environment consistently cost the team roughly half a day of waiting each time, even though the criteria for approval had become entirely mechanical, the same three checks every time, with no real judgment call left in it. The fix: a small self-service script that ran those same three checks automatically and auto-approved when they passed, falling back to the manual ticket only when a check actually failed. Because the change touched a provisioning system owned by another team, its lead was looped in before anything shipped, rather than deploying a workaround directly into a system this person did not own, and sign-off came quickly since the change only automated a decision that was already mechanical, not a judgment call being taken away from anyone. Result: the environment-provisioning wait dropped from roughly half a day to about 15 minutes for the common case, recovering close to half a day of calendar time per release cycle, and across the next four releases where the team used it, this removed what had been the single largest scheduled delay in the release checklist.

Trade-offs and pitfalls

Unilaterally changing shared infrastructure without looping in its owner, even with good intentions, erodes trust and risks breaking something not fully understood. Choosing a blocker that is satisfying to fix but not actually on the delivery-critical path does not move the timeline at all, no matter how good the fix feels. And closing with a vague claim that things felt smoother afterward, instead of naming the one specific thing that got measurably faster, is the difference between a story that demonstrates ownership and one that only asserts it.

With 60% of the monthly error budget already spent by day ten and a feature rollout scheduled for day eleven, the decision should not be an automatic freeze or an automatic green light; it should weigh how much budget is genuinely left, whether the rollout itself is a meaningful new source of risk, and who needs to be in the room before deciding. A service-level objective (SLO) of 99.95% over 30 days leaves an error budget of 0.05% of requests for the month; having burned 60% of it in the first third of the month is a real signal, but the right response depends on what caused the burn and what the rollout would add on top of it.

Working through the decision

1. Understand the cause of the burn before deciding anything else. If the 60% came from a single, already-resolved incident (a bad deploy that's been rolled back), the ongoing risk profile for the rest of the month is different than if it's an unresolved, intermittent issue that's still actively burning budget; the second case argues much more strongly against adding a new source of risk on top of it.
2. Estimate the rollout's own risk contribution, not just its existence. A well-tested, low-blast-radius change behind a feature flag is a different risk than a large, untested change touching the same code path as the current instability.
3. Decide among proceed, delay, or modify, with the default leaning toward caution given 60% is already spent with two-thirds of the month remaining: proceed only with extra safeguards, delay until the underlying issue is understood or resolved, or modify the rollout to reduce its own risk contribution (a smaller canary percentage, an easier automated rollback path, tighter feature-flag gating so it can be turned off instantly without a redeploy).
4. Involve the right stakeholders before deciding, not after: the engineering lead who understands the ongoing burn, the product owner who owns the rollout's business urgency, and, if the budget is this depleted, whoever owns the team's error-budget policy (often an SRE lead), since a policy freeze may already be triggered independent of this specific rollout.

Worked example of "proceed, with controls"

The team determines the 60% burn came from a resolved, one-time incident rather than an ongoing issue, and the planned rollout is a well-tested change behind a feature flag. Decision: proceed, but with a small initial canary (1% of traffic instead of the originally planned 10%), automated rollback wired to the same error-rate alert that would page on-call, and an explicit go/no-go check with the product owner and on-call lead 24 hours after the canary starts before expanding further. If the remaining 40% of budget were instead being burned by an ongoing, unresolved issue, the more defensible call would be to delay the rollout until that issue is fixed, since proceeding would be adding new risk on top of an already-active problem rather than a resolved one.

Trade-offs and pitfalls

Treating the error-budget percentage as a mechanical stop light (over 50% burned means automatic freeze, regardless of cause) is simpler to apply consistently but ignores real information about whether the remaining risk is active or already resolved, and can block genuinely low-risk, valuable work unnecessarily. The opposite failure, treating the number as advisory and rolling out anyway without adjusting the rollout's own risk profile, undermines the entire point of having an error budget as a real decision input rather than a dashboard nobody acts on. The healthiest version of this decision is fast (it should not take days to resolve) and transparent (the reasoning and the people consulted are visible after the fact, especially if the call turns out to be wrong).

Direct answer
When a client keeps retransmitting its SYN and never sends the final ACK (while the server's SYN-ACK is only retransmitted once before the connection times out), the most likely causes are that the client's connect timeout hasn't fired yet, that something between the two hosts is dropping the ACK specifically (an asymmetric path or a stateful device confused about direction), or that the client-side application itself never actually attempted the ACK due to a bug. A different failure shape, the server sends SYN-ACK and the client immediately sends RST, points to a different family of causes entirely: the client rejecting the connection outright.

Structured elaboration
For the "client retransmits SYN, no final ACK" pattern, work through causes in order of likelihood:

1. Asymmetric routing dropping only the return-to-forward-direction ACK path. If the SYN-ACK reaches the client (we know it does, since the client keeps retransmitting new SYNs rather than giving up, meaning it IS getting a response of some kind) but the client's ACK can't get back to the server along some other path, a stateful firewall or NAT device on that asymmetric path may be dropping the ACK because it doesn't recognize the connection's state in that direction.
2. A middlebox is rewriting or dropping specific TCP options in the SYN-ACK that the client's stack doesn't handle gracefully, causing it to silently discard the SYN-ACK and retry instead of ACKing it.
3. Client-side firewall or security policy is specifically blocking outbound ACKs to that destination while permitting the outbound SYNs, an unusual but real misconfiguration.
4. MTU (Maximum Transmission Unit)-related silent packet loss on the SYN-ACK's return path if it happens to be an unusually large segment (rare for a SYN-ACK specifically, since it typically carries little payload, but worth ruling out if other symptoms point that way).

For the different shape (SYN-ACK followed by an IMMEDIATE client RST): this usually means the client-side application decided, upon establishing the connection, that it doesn't actually want it, for example an application-level timeout that already expired while the handshake was in flight, a client-side connection pool that raced two connection attempts and is aborting the loser, or a security tool on the client actively resetting connections that don't match an expected certificate or policy.

Worked example
To distinguish these hypotheses in practice, compare timestamps and evidence on BOTH ends: if the server's capture shows the SYN-ACK leaving on time but the client's capture never shows it arriving, the problem is in the path (asymmetric routing, a device eating it). If the client's capture shows the SYN-ACK arriving cleanly but no ACK is ever generated by the client's own stack, the bug is on the client host itself (application logic, local firewall) rather than the network path.

Trade-offs & pitfalls
A common mistake is assuming a stuck handshake is always a network problem; a client-side timeout race (the application gives up right as the handshake completes) produces an outwardly identical-looking symptom to a network drop and is only distinguishable by comparing what each side's own capture actually shows, not by reasoning about the network path alone.

Direct answer

Don't force one team's cadence onto the other. Design a policy that separates what must be shared (the guardrails that protect everyone) from what can stay team-specific (how fast a given team is allowed to move within those guardrails), then negotiate the guardrails, not the cadence itself. That reframing turns "fast team vs. cautious team" into a joint design problem both sides can own.

Structured elaboration

1. Split invariant from flexible. List what truly must be uniform across teams (a working rollback path, a minimum test bar, an incident-response process) versus what can legitimately vary (deploy frequency, staging gate count, review depth). Most conflicts collapse once you see that only a small slice actually needs to be shared.
2. Reframe cadence as risk exposure. Ask each side what they're protecting (customer trust, an SLA, a compliance obligation) versus what they want (velocity). Convert both into measurable guardrails: blast radius limits (how much of the system or traffic a change could affect if it goes wrong), an automated rollback trigger (a rule that reverts the change automatically once a threshold is crossed, without waiting for a human to notice), a minimum observation window before a change is considered "safe."
3. Build a tiered policy, not a single rule. Changes that touch a small blast radius and have a fast, automatic rollback can move on the fast-moving team's cadence. Changes that touch shared, hard-to-reverse surfaces get the slower team's gates, regardless of which team wrote the change. The tiering criteria, not the team identity, decides the process.
4. Add an explicit exception path. Either side can request a deviation (ship something in a higher tier faster, or hold something in a lower tier longer) with a documented reason and a named approver, so departures from the policy are visible instead of quiet workarounds.
5. Time-box a trial and revisit with real data. Don't debate the policy hypothetically forever. Run it for a fixed period, then bring incident counts and delivery-time data back to the table instead of re-litigating the original positions.

The same negotiation pattern applies beyond deploy-frequency disputes: whenever two functions have structurally different operating rhythms, the fix is a shared cadence at the boundary, not a winner. As a concrete cross-team cadence clash from the machine-learning world: a feature store (the shared system that stores and serves the data used to train and run machine-learning models) team can only refresh labels every two weeks, while the product team needs weekly model retraining (rerunning the training process on newer data so the model's predictions stay current). That isn't a risk-tolerance disagreement at all. It's a hard technical constraint on one side meeting a business cadence need on the other, and it gets negotiated the same way: agree what must move on the constrained cadence (the underlying label refresh) versus what can be decoupled (the product team retrains weekly on the two most recent completed label batches, accepting known staleness, rather than blocking on a refresh that can't happen faster).

Worked example

Team A ships to production many times a day behind feature flags. Team B owns a regulated, customer-facing billing surface and wants a weekly release train. Instead of debating "how often should we deploy," the negotiated policy ties process to blast radius: any change gated behind a flag to less than 1% of traffic can auto-promote if the error rate stays under 2x the pre-change baseline for a 30-minute observation window (a policy parameter both sides agreed to, not a claimed result). Changes that touch the billing ledger directly, regardless of author, require the slower manual review and a scheduled release window. Team A keeps most of its velocity because most of its changes are low blast radius; Team B keeps its protection because the surface it cares about is gated the same way no matter who wrote the change.

For the cadence-mismatch variant: the feature store team commits to publishing a refreshed label snapshot every two weeks, on a fixed schedule the product team can plan around. The product team's weekly retraining job consumes the most recent snapshot plus a lightweight, clearly-labeled interim signal for the intervening week, rather than either side pretending the refresh can happen weekly or the product team silently retraining on stale labels without acknowledging it.

Trade-offs & pitfalls

• Pitfall: writing a single global policy. It's either too loose for the regulated team or too strict for the fast-moving one, and both sides end up circumventing it.
• Pitfall: treating this as a one-time meeting. Without a scheduled revisit, the policy calcifies around the political balance of the original conversation instead of actual incident/velocity data.
• Pitfall: hiding exceptions. If deviations aren't logged and visible, the "shared" part of the policy erodes silently and trust breaks down the next time there's an incident.
• Senior differentiator: designing the guardrail so it's parameterized by risk (or, in the cadence case, by the actual constraint) rather than by team identity. That's what lets both sides keep their operating model instead of one side losing the negotiation.

Race conditions and concurrency bugs leave a distinct fingerprint that a deterministic logic bug doesn't: they are inconsistent under identical-looking conditions.

Signs and the diagnostic for each

• Fails intermittently on the same input/code path. A deterministic bug fails every time given the same input; a race fails only sometimes. Diagnostic: rerun the exact same case N times and check the failure rate is neither 0% nor 100%.
• Fails more often under load or parallelism, passes when run alone. Diagnostic: run the same test serially vs. in parallel and compare failure rates.
• Disappears or changes when you add logging or attach a debugger (a heisenbug symptom). Diagnostic: this itself is evidence of timing sensitivity, since instrumentation changes the exact interleaving.
• Values that look "impossible" given the code's logic (a counter lower than the number of increments performed, a list missing an entry that was definitely inserted). Diagnostic: instrument the shared state with fine-grained before/after logging and check whether two threads' logged operations interleave in an order the code never anticipated.
• Different results across repeated runs of the identical binary/input. Diagnostic: run twice, diff the outputs; a deterministic bug produces identical (wrong) output both times.

Trade-offs and pitfalls

The trap is treating a low reproduction rate as "rare enough to ignore"; a race with a 1-in-10,000 trigger rate in a test can be a 1-in-a-minute event at production scale, so the diagnostic above should be run at a scale proportional to production traffic, not just a handful of manual retries.

Start with a clear hypothesis-driven plan: isolate whether loss is (A) application/socket, (B) kernel/host, (C) NIC, or (D) network path. Work top-down and iterate measurements + fixes.

1. Reproduce & baseline

• Reproduce loss under controlled load. Record timestamps, metrics, SLO impact.
• Note traffic pattern (pps vs throughput, packet size).

2. Application / socket layer checks

• Verify receive buffer: check SO_RCVBUF and kernel rmem_default/rmem_max. Temporarily increase SO_RCVBUF on receiver and measure.
• Use socket options: set SO_TIMESTAMP/SO_TIMESTAMPING to correlate capture timestamps with app logs.
• Ensure app reads fast enough (non-blocking, epoll) and isn’t dropping due to queuing backpressure.
• Add drop counters/logs in app if possible.

3. Kernel network stack & counters

• Check UDP error counters: netstat -su (or ss -u -s) for receive errors/drops.
• Inspect /proc/net/dev for interface-level rx/tx drop counters.
• dmesg/syslog for kernel NIC driver messages.
• Check skbuff/alloc failures: /proc/net/softnet_stat (columns 4–6 are drops/overruns). Use sar -n DEV, ifstat.
• Tune net.core.rmem_max, net.core.netdev_max_backlog, net.core.rps_sock_flow_entries, net.ipv4.udp_mem if needed.

4. NIC tuning and ring sizes

• Query and set ring sizes with ethtool -g / -G (rx/tx rings). Increase RX ring and netdev_max_backlog, test effect.
• Examine driver-specific ring and queue stats (ethtool -S).
• Ensure NIC firmware/driver up-to-date.

5. Interrupt handling and CPU affinity

• Check /proc/interrupts to see IRQ distribution and hot CPUs.
• If high pps, enable/configure RSS/RFS: ensure flows spread across CPUs. Configure irqbalance or set rps/rx-queue affinity manually.
• Monitor softirqs (cat /proc/net/softnet_stat, vmstat/ sar -q). Consider increasing RPS/ XPS for performance.
• If CPU saturation on processing cores, move interrupts or processing to spare CPUs.

6. Offloads (checksum, TSO/GSO, GRO/LRO)

• Capture behavior with offloads on vs off: ethtool -K eth0 gro off gso off tso off rxvlan off txvlan off rx-checksumming off.
• Offloads can hide packet reception reality in kernel; disabling helps get canonical packets for tcpdump and debugging.
• If disabling fixes loss, tune NIC or driver firmware or adjust MTU/GSO settings.

7. Captures & measurements to isolate location of loss

• Use packet capture tools:
  • tcpdump -i eth0 -w /tmp/a.pcap (with snapshot length large enough) on receiver host.
  • Simultaneously capture on NIC (if supported via port mirroring or on sender host) so you can compare sequence numbers/timestamps.
• For high-performance capture, use AF_PACKET TPACKET V3 / packet_mmap or hardware timestamping to avoid capture-induced loss.
• Run iperf3 (UDP) or flent/packetgen to send known sequence/pps and measure loss. Vary packet size and rate.
• If sender and receiver captures show packets arrive at NIC but not to kernel/app, problem is NIC/driver or kernel. If packets disappear on path, both captures differ — problem on network path.

8. Path testing

• Test server-to-server on same top-of-rack switch to rule out network path. If loss disappears, likely network.
• Use traceroute/mtr and per-hop statistics (where supported).
• Use controlled tests with tc/netem to simulate drops/latency and compare.

9. Additional tools & metrics

• ethtool -S, ethtool -i for driver stats; ifconfig/ip -s for rx/tx errors.
• perf / bpftrace to see time spent in softirq/netdev_receive.
• pktgen (kernel module) to generate high pps locally to stress the stack.

10. Remediation steps (iterative)

• If app-limited: increase SO_RCVBUF, improve reading concurrency.
• If kernel-limited: bump net.core.netdev_max_backlog, rmem_max, enable RPS/NRPS, apply CPU affinity.
• If NIC/driver: increase ring sizes, update driver/firmware, disable problematic offloads, tune NAPI weight.
• If network path: involve network team with pcap evidence; check switch port drops, buffer congestion, QoS shaping.

11. Validation

• After each change, re-run controlled iperf/tcpdump tests and compare sender vs receiver counts.
• Keep artifacts: pcaps, ethtool outputs, /proc counters, timestamps to support root cause and postmortem.

Key principles: make small, reversible changes; collect measurable before/after data; isolate by comparing captures on both ends; correlate kernel counters with pcap evidence to pinpoint host vs NIC vs network.