Entry-Level Systems Administrator Interview Preparation Guide (FAANG Standards)

Direct answer
I treat a company's published culture and values as a claim to be tested, not a fact to accept, and I look for evidence in three places: how people describe real, specific incidents (not slogans) when I ask about them, whether the org's actual structures and incentives would make the stated behavior easy or hard to practice, and whether the story is consistent across different people I talk to in the process.

Structured elaboration

1. Ask for a specific recent incident, not a description of the value. A question like "tell me about a time the team had to choose between shipping fast and following the documented review process" forces a real story; a question like "how would you describe the engineering culture here" invites a rehearsed, values-page-adjacent answer that tells you little.
2. Check whether the org's structure actually supports the stated value, independent of what anyone says. If a company claims to value psychological safety but every interviewer you meet is visibly guarded about naming any team problem, or if a company claims strong autonomy but every technical decision in the loop turns out to require a director's sign-off, the structural evidence contradicts the claim regardless of the wording used to describe it.
3. Triangulate across multiple people, ideally at different levels and tenures. A single enthusiastic interviewer proves little; a hiring manager, a peer-level engineer, and someone from a different function independently describing the same specific behavior (not the same slogan) is much stronger evidence.
4. Ask what the company would do differently if it stopped believing the value, and watch for a concrete, structural answer versus a vague one. People who work inside a genuinely lived value can usually name a real trade-off it costs them; people describing marketing usually cannot.
5. Treat your own discomfort as data. If a described norm (pace, feedback directness, decision-making style) makes you visibly uneasy during the process itself, that is a more reliable signal about fit than anything printed on the careers page, because it is your own live reaction rather than a claim you are being asked to evaluate secondhand.

Worked example
Suppose a company's careers page says it "empowers engineers with high autonomy." During the loop, ask the hiring manager for a specific recent example: "Tell me about the last time an engineer on this team made a production architecture decision without it going through a review committee first." A genuine, lived-autonomy answer sounds like: "Last quarter one of our engineers decided independently to switch a service from synchronous to async processing after noticing latency complaints; she looped in two people for a sanity check, shipped it, and reported the outcome in the next team sync." A marketing-only answer sounds like: "We really believe in empowering our engineers," repeated with no specific incident when pressed twice. If a peer engineer you speak to separately can also describe a comparable specific incident in their own words, that consistency is strong corroborating evidence; if the hiring manager's story turns out to be the ONLY example anyone can produce company-wide, that is itself informative about how common the behavior actually is.

Trade-offs & pitfalls
The main failure mode is accepting an interviewer's fluent, confident description of the culture as sufficient evidence on its own; confidence and specificity are not the same thing, and a well-rehearsed answer to a values-page question is exactly what a company under-delivering on its stated culture is most likely to have prepared. A second pitfall is over-weighting a single glowing anecdote from one enthusiastic interviewer without checking whether it generalizes; one great story is an anecdote, not a pattern. A third is treating any inconsistency you find as automatically disqualifying: it is normal for a large or growing organization to have real variance across teams, so the useful conclusion is usually about the SPECIFIC team and manager you'd actually join, not the company as a monolithic whole.

A bug that only appears under heavy load needs a test harness that can manufacture load deterministically, since waiting for production traffic to happen to trigger it again is not a repeatable investigation.

Building the harness

1. Synthetic traffic generation at a controlled, repeatable concurrency and request mix (a load-testing tool driving realistic request shapes, not just raw throughput).
2. Seed shared state deterministically (fixed starting data, fixed random seeds where the system uses randomness) so the only varying factor between runs is scheduling/timing, not also data variance.
3. Concurrency controls and deterministic schedulers where available, to bias the interleaving toward the suspected contention point rather than relying purely on load volume to eventually hit it.
4. Time manipulation (accelerating or controlling clock-dependent logic) when the bug involves timers, TTLs, or scheduled work, so you don't need to wait real-world hours to observe a time-triggered condition.
5. Minimize noise, prove causality: run the same load profile with and without the suspected contributing factor (a specific code path enabled/disabled, a specific config toggled) and compare failure rates statistically rather than trusting a single run either way.

A concrete run

Suppose the suspected bug is a cache-eviction race that only shows up under heavy concurrent writes. Running the harness at a fixed seed (seed=42) and concurrency=20 reproduces the race 0 times in 50 runs; the same harness at concurrency=200 reproduces it in 12 of 50 runs, a clear, statistically meaningful signal that concurrency level, not chance, drives the failure, and a concrete number, 12 of 50 at 200 versus 0 of 50 at 20, a reviewer can rerun and check rather than take on faith.

Applying this without dedicated load-test infrastructure

A single server that fails intermittently under heavy load, or that can't reproduce in staging, benefits from the cheaper version of the same idea: increase load safely on that one host (careful, bounded synthetic load) while capturing fine-grained logs/traces, rather than waiting for the next natural production spike.

Trade-offs and pitfalls

Synthetic load rarely matches production traffic shape exactly (arrival patterns, payload variety, cache warmth); the harness's value is in reliably reproducing the class of failure for iteration, not necessarily reproducing the exact production timeline, so a fix validated only against synthetic load still needs confirmation against real traffic before being called durable.

Direct answer

A migration of this scale, 5 million users, succeeds or fails on one design choice made early: which credential-handling model connects the existing Active Directory, the on-premises directory holding every user's password and Kerberos identity, to the cloud identity provider (IdP), since that choice determines how much on-premises infrastructure has to stay up, how fast a password change actually propagates, and what happens during a cloud outage. The three options, synchronizing password hashes, pass-through authentication that checks passwords against on-premises Active Directory in real time, and full federation to a separate identity system, trade off differently on latency, on-premises dependency, and blast radius, and the safest migration keeps Kerberos-dependent legacy services working throughout by exporting or bridging Kerberos trust rather than trying to eliminate it before every legacy service has actually been moved off it, cutting over the user population in small, reversible waves rather than one global event.

Structured elaboration

• Credential handling options.
  • Password hash synchronization: a lightweight sync agent periodically synchronizes a secure hash of each user's on-premises password to the cloud identity provider, which then authenticates independently, without a live call back to on-premises infrastructure at sign-in time. This is the simplest option operationally, no dependency on on-premises availability at login time, but means a password change takes as long as the sync interval to reach the cloud side, and it does not, by itself, preserve Kerberos-based single sign-on (SSO) to on-premises legacy services, since Kerberos still needs the on-premises domain controller.
  • Pass-through authentication: sign-in requests are validated by the cloud provider calling back to lightweight on-premises agents, which validate the password directly against Active Directory in real time. A password change is effective the instant it's made on-premises, but sign-in now depends on the on-premises agents being reachable, so an on-premises outage becomes a cloud sign-in outage too.
  • Federation: the cloud identity provider hands off authentication entirely to a separate, trusted identity system, traditionally Active Directory Federation Services or a third-party federation provider, which has full control over authentication logic, custom multi-factor authentication (MFA) requirements, smart-card login, custom conditional logic, at the cost of the most on-premises infrastructure to build, secure, and keep highly available, since a federation-service outage is a full sign-in outage.
  • Choosing for this migration: for a population this large, password hash synchronization is the lowest-risk starting point for the majority of users, since it removes on-premises availability from the sign-in critical path; pass-through authentication or federation are reserved for populations with a specific requirement hash sync can't satisfy, a compliance rule against passwords ever leaving the premises even as a hash, or a custom multi-factor requirement federation already implements. Multiple methods can coexist for different user populations during migration; they don't have to be an all-or-nothing organization-wide choice.
• Kerberos interoperability and export strategies. This migration is explicit that Kerberos-based services must keep working, which means the plan cannot simply retire Active Directory the moment users are synced to the cloud. Two complementary approaches: first, keep a right-sized on-premises Active Directory presence alive specifically to keep issuing Kerberos tickets for legacy services, even after most users authenticate to cloud-facing applications through the cloud identity provider, treating Kerberos-dependent legacy services as their own migration track with their own timeline, separate from the user-authentication migration. Second, for legacy services that can be migrated off Kerberos, front them the same way a legacy web application without modern federation support would be handled, a gateway or agent translating a modern token into whatever the legacy service expects, which is the path that eventually lets the on-premises Kerberos infrastructure be retired rather than kept alive indefinitely as a permanent parallel system.
• Staged cutover with rollback. Migrate in cohorts, by business unit, office, or a similarly natural population boundary, never the whole 5 million at once. For each cohort: provision the cohort in the cloud identity provider and enable the chosen credential-handling method for their accounts while their existing on-premises sign-in path remains fully functional in parallel; run a defined validation window, see testing below, with real users in that cohort using the new path for non-critical access first; cut the cohort's primary sign-in over to the new path, keeping the old path disabled but not deleted for a defined rollback window, so a serious problem discovered just after cutover can be reversed by re-enabling the old path rather than needing to be fixed forward under pressure; only after a cohort has been stable on the new path past its rollback window does its old path get permanently retired.
• Syncing group memberships and policies. Group memberships, which drive both application access and Kerberos-based authorization to some legacy services, need to replicate to the cloud identity provider on a cadence tight enough that a change made on-premises is reflected in the cloud side before it would matter operationally, not just eventually. Validate this specifically as part of cutover testing rather than assuming the sync mechanism's default interval is good enough for every group, since a slow-to-sync removal from a privileged group is a more serious problem than a slow-to-sync addition.
• Testing and validation strategies. Before any cohort's real cutover, validate, for a representative sample of that cohort's actual accounts: sign-in succeeds through the new credential-handling path; group-membership-driven access matches what it was on the old path, so no user gains or loses access as a side effect of the migration itself; every Kerberos-dependent legacy service that cohort actually uses still authenticates correctly, since this is the requirement most likely to be silently broken by an incomplete migration; and the rollback path itself has actually been exercised at least once before relying on it, not just designed on paper, since a rollback plan that's never been tested is a plan that fails at the worst possible moment.
• Minimizing user disruption and authentication issues. Communicate to each cohort specifically, ahead of their cutover, what will change and what to do if something doesn't work, a specific support contact, not a generic help-desk queue that doesn't know this migration is happening. Schedule each cohort's cutover outside that population's peak usage window where the business allows it. Keep the previous path's credentials and configuration valid but inactive, not deleted, through the rollback window specifically so disruption during a real problem is measured in minutes, re-enabling the old path, rather than however long it takes to diagnose and fix the new path under live pressure.

Worked example

flowchart LR
    P0[Phase 0: inventory forest, SPNs, group memberships] --> P1[Phase 1: deploy sync, pilot group]
    P1 --> P2[Phase 2: staged cutover by business unit]
    P2 --> P3[Phase 3: Kerberos interop for legacy services]
    P3 --> P4[Phase 4: full cutover, legacy AD read-only]
    P4 -.->|rollback path| P1

1. Phase 0: inventory the on-premises Active Directory forest, every service that depends on Kerberos tickets, and current group structure, to scope both the user-authentication migration and the separate Kerberos-legacy-service track.
2. Phase 1: deploy the sync infrastructure and pilot password hash synchronization with a small, low-risk group, commonly IT or security staff, validating sign-in, group-membership fidelity, and Kerberos access to the legacy services that pilot group actually uses.
3. Phase 2: staged cutover cohort by cohort, by business unit, each cohort running in parallel with its old path active, validated, cut over, and held in a rollback window before the old path is retired for that cohort.
4. Phase 3: for legacy services still depending on Kerberos after most user cohorts have migrated, address them on their own track, either keeping a minimal on-premises Active Directory presence alive specifically for Kerberos ticket issuance, or fronting them with a gateway that translates a modern credential into what the legacy service expects.
5. Phase 4: once every cohort is stable past its rollback window and the Kerberos-legacy track has either retired or explicitly kept its minimal on-premises dependency, the on-premises directory's role shrinks to exactly that remaining Kerberos-legacy purpose, if any, rather than being retired in name only while still secretly load-bearing.

A rollback from Phase 2 for any single cohort re-enables that cohort's original on-premises sign-in path, which remained configured, not deleted, specifically to make this possible without a rebuild.

Trade-offs and pitfalls

Choosing federation for the whole population by default, because it offers the most control, when hash synchronization would satisfy the actual requirement for most users adds unnecessary on-premises infrastructure and a new single point of failure, the federation service, for no real benefit; the credential-handling choice should be made per requirement, not as a single organization-wide default chosen for the hardest case. Treating Kerberos interoperability as solved just because pass-through authentication or federation technically keeps passwords validated on-premises is a common confusion: none of the three credential-handling options, by themselves, preserve Kerberos ticket issuance for legacy services, since that's a separate mechanism, the domain controller acting as a Kerberos key distribution center, that has to be explicitly kept alive or replaced, not an automatic side effect of the authentication method chosen. A rollback plan that exists only as a document, never actually exercised, is the single most common way a staged-cutover-with-rollback migration turns into an unplanned, high-pressure fix-forward event the first time a cohort's cutover has a real problem; the testing strategy in this plan deliberately requires exercising the rollback itself, not just designing it. Syncing group additions promptly while treating removals as lower priority, because the sync tooling's default behavior sometimes does exactly this, quietly recreates a privilege-creep problem during the migration itself: someone removed from a sensitive group on-premises should lose the corresponding cloud access on a timeline that matches, not lag behind it.

Direct answer

Give every data centre its own local-time window, never patch both sides of a redundant service in the same window, make the change records standard and pre-approved so every wave is auditable without a fresh meeting, and sequence reboots by dependency with a go or no-go check between steps. Time zones, not tooling, are where this usually breaks: windows must be defined in each site's local time, converted to UTC for each specific date, and published with both times side by side.

1. The window schedule (computed)

Example: 10 data centres, each with a Saturday 01:00 to 05:00 local window. The table converts that window for 2026-11-14, the week-1 date; none of these regions changes its clock between 2026-11-14 and 2026-11-21, so the same UTC times apply to the week-2 date, and the rule below decides which sites use which week. The Coordinated Universal Time (UTC) start and end below were computed with the time-zone database (the IANA tz database, the shared list of every region's offset and daylight saving rules that operating systems and programming languages use), not estimated.

How to redo a row: UTC = local time minus the offset. The offset says how far the local clock is ahead of (+) or behind (-) UTC. Sydney is +11:00, so 01:00 local Saturday minus 11 hours is 14:00 UTC on Friday, and the window ends 4 hours later at 18:00. Mumbai is +05:30, so 01:00 minus 5 hours 30 minutes is 19:30 Friday. Dallas is -06:00, so 01:00 minus (-6) hours is 07:00 Saturday. The offsets in the table are the ones in force on 2026-11-14: Sydney's +11:00 is its daylight saving offset (in July the same computation gives +10:00), London and Frankfurt are on standard time, and Ashburn (-05:00) and Dallas (-06:00) are on US standard time after the November clock change.

Read the table for the conflicts that would arise if every site patched on the same Saturday (these are examples, not the full list): London (01:00 to 05:00 UTC) overlaps Frankfurt (00:00 to 04:00 UTC) for three hours; Ashburn (06:00 to 10:00) overlaps Dallas (07:00 to 11:00) for three hours; Sao Paulo (04:00 to 08:00) overlaps Ashburn for two; Mumbai and Dubai overlap for two and a half. If a service runs in both members of any overlapping pair, the overlap is an outage waiting to happen.

Rule: the two sites of any redundant service are in different weeks. Example pairing (replace with your real replica map): Ashburn-Dallas, London-Frankfurt, Singapore-Sydney, Mumbai-Dubai, Sao Paulo-Johannesburg. Set A (Ashburn, London, Singapore, Mumbai, Sao Paulo) patches in week 1; Set B (Dallas, Frankfurt, Sydney, Dubai, Johannesburg, the partner of each Set A site) patches in week 2. At least one side of every service stays in service at all times, and the one-week gap between the two Saturdays (2026-11-14 for Set A, 2026-11-21 for Set B) doubles as the observation period for Set A.

2. Time-zone traps

• Daylight saving time shifts UTC. London 01:00 local is 00:00 UTC on 2026-07-11 (British Summer Time) and 01:00 UTC on 2026-11-14 (Greenwich Mean Time). If you convert to UTC once and reuse that number for every month, the windows drift an hour twice a year, so the UTC table is regenerated for each date.
• Tool behaviour: in Configuration Manager the maintenance window has a UTC option so all devices in one collection start together; Microsoft documents that the client uses the offset in force on the window's effective date, so after a daylight saving change the window opens an hour early or an hour late. Prefer local-time windows per data centre unless you truly need simultaneity. In practice the two artefacts coexist: the patch tool's window is set in each data centre's local time, and the schedule people read is a UTC and local table regenerated for each patch date.

3. Coordinated reboots

Per application, in dependency order and one node of a redundant pair at a time:

1. Confirm last night's backup and that the partner node is healthy.
2. Drain the node from the load balancer or move the clustered role off it.
3. Patch and reboot.
4. Run the health check (service answers, replication in sync, cluster node up).
5. Return it to service, wait the soak period (a fixed observation time, for example 30 minutes, with the node serving real traffic and being watched), then move to the next node.

Order across tiers: databases replica first, then application tier, then web tier. A failed health check stops the sequence for that application; it does not stop other applications.

4. Change approvals that stay auditable

Use a standard change (a pre-approved, low-risk, repeatable change type) so the approval is given once to the procedure, and each wave is a record under that template. A record has fixed fields:

Application owners can decline a window only before a published cutoff, and a declination creates an exception with a new date, so the record shows who delayed what and why. Out-of-band fixes use a separate emergency change type with after-the-fact review.

5. Contingencies across 10 data centres

Per data centre: a named local on-call, a go or no-go checkpoint at the first third of the window, a hard stop 30 minutes before the window ends (no new reboots after that point), and a communication to application owners at start and at close. If the window is cut short, remaining hosts roll to the next scheduled window for that data centre; they do not run into business hours.

Trade-offs

Fully parallel patching across data centres finishes fastest but concentrates risk; the two-set rotation takes two weeks for the whole estate and is the right trade for services with service-level agreement (SLA) commitments. Fully standard changes reduce meeting load but only work if the template is truly low risk, so database cluster upgrades stay as normal changes.

Direct answer. Build a directed dependency graph from the map, topologically order it so nothing starts before what it depends on is verified healthy, and treat truly independent branches as parallel recovery tracks. Validate each step before advancing, and when a step fails, hold everything downstream in a known state and escalate through a defined decision point rather than retrying indefinitely.

1. Build and order the graph

• List every service and every "depends on" edge from the map. Anything with no unresolved incoming edges (here, the shared database) starts first.
• Order the rest by dependency depth: services that depend only on the database, then services that depend on those, and so on. A service that depends on the external payment API doesn't block anything upstream, since nothing needs it in order to start; it just needs its own dependency confirmed before it accepts payment-related traffic.
• Two services that depend on each other (a cycle) can't be strictly ordered against one another. Treat them as one recovery unit: bring both up together, then verify the pair as a whole before opening it to traffic.

2. Find the parallel opportunity

Any two services with no dependency edge between them, direct or transitive, can recover on separate tracks. The common case here is two services that both depend only on the shared database but not on each other: once the database is verified healthy, both can start at the same time instead of one waiting on the other. Parallelizing shortens wall-clock recovery time without changing the correctness of the order, since neither branch depends on the other's state.

3. Checkpoint and verify data integrity at each step, not just liveness

"Started" and "verified healthy" are not the same thing. At each step, confirm the service isn't just accepting connections but is serving data consistent with the last known-good state (for example: row counts within an expected range, no partial-write artifacts, a plausible latest-committed-transaction time). Skipping this and checking only process liveness is how a corrupted or half-restored dependency gets silently built on top of.

Log a checkpoint (what was verified, when, by what evidence) at each step. That makes the sequence auditable afterward and lets a later step's failure be traced back to a specific, verified starting point instead of an assumption.

4. Which pieces can stay in a degraded state while others recover

Not everything needs to be fully healthy before it's usable. A service that only reads reference data from the shared database can often be brought up in a degraded, read-only mode as soon as the database is verified, well before every other service has finished its own recovery. The service that depends on the external payment API is a good candidate to deliberately hold in a degraded state (accept and queue requests, decline new payment attempts) until its own dependency check passes, rather than blocking the rest of the sequence on an external system nobody on the recovery team controls.

What counts as an acceptable degraded state is a business call as much as a technical one: it depends on what each function actually needs to deliver during the outage, not on what's technically possible to bring online.

5. Handling a failed step

Define a bounded number of retries with a real timeout, not an indefinite retry loop; retrying against a dependency that isn't going to recover in that window just burns time. When the bound is hit, hold everything downstream of the failed step in its current state and escalate to a named decision point (the recovery coordinator, or the owner of the affected business function) who chooses between: wait longer, fail over to an alternate path if one exists, or proceed with dependents running in a degraded mode that doesn't require the failed step. Whatever is decided gets logged with the reasoning, same as the checkpoints, so a later review can tell what was planned and what was improvised under pressure.

Worked example

Map: shared-database (no dependencies); service-A and service-B (both depend only on shared-database, not on each other); service-C (depends on service-A); service-D (depends only on the external payment API).

Order: shared-database first. Then service-A and service-B in parallel, since neither depends on the other and both only need the database. service-C waits specifically for service-A's checkpoint to pass, and can start while service-B is still on its own track. service-D can be started any time after the database step but is deliberately held in a degraded (read-only or queued) mode until the external payment API's own health check clears, since that dependency is outside the team's control.

If service-A's checkpoint fails (health check passes but the row-count integrity check does not), service-C does not start. service-C's traffic is held, service-A is retried up to the defined bound, and if it doesn't clear, the recovery coordinator escalates: either fail service-A over to a secondary if one exists, or accept that service-C runs in a reduced-functionality mode that doesn't need service-A's data, whichever gets the organization to an acceptable state fastest.

Trade-offs & pitfalls. The most common wrong turn is treating "process is up" as "safe to build on top of": a liveness check is not a correctness check, and skipping the integrity checkpoint is how a downstream service inherits a silently corrupted starting point. Being too conservative (fully sequential, no parallel tracks) wastes recovery time when the graph has real independence in it; being too aggressive (parallelizing across an actual dependency edge) risks starting a service against data that isn't there yet. The graph should decide what's parallelizable, not intuition. And a dependency map that looks authoritative but isn't kept current is worse than no map at all, because it gives false confidence about what's actually safe to run in parallel.

Direct answer

I treat a paper as a claim to be tested against my own situation, not a text to summarize. I triage fast to see whether it's even worth deeper investment, then build the smallest thing that could prove or disprove the specific claim against my own data or context, and I judge the result against my own baseline rather than the paper's reported numbers.

Structured elaboration

• Triage before investing real time. I read the summary, the method, and the results first, and ask directly whether this actually applies to my problem, my scale, and my constraints, before going any deeper. Most things that look relevant from the headline don't survive this first pass.
• Decide the reproduction scope on purpose. I'm not obligated to rebuild the whole thing; I pick the smallest slice that actually tests the specific claim I care about, and I'm explicit with myself about what fidelity I'm giving up to get there, such as simplified data or a toy version of the setup, so I don't end up trusting a shortcut more than it deserves.
• Build something that runs, not just a mental summary. A claim only becomes genuinely checkable once it's instantiated against real inputs I control, not just reasoned about on paper.
• Compare against my own baseline, not the source's. The source's own reported baseline was almost certainly measured under different conditions than mine, so the only comparison that actually tells me something is against what I'm currently doing, or would do without this.
• Decide adopt, adapt, or discard from that comparison, and write the verdict down so the next person doesn't have to redo the same triage from zero.

Worked example

I came across a paper proposing a locality-sensitive hashing (LSH) scheme for near-duplicate detection in a large text corpus, claiming it could find duplicates within a fixed similarity threshold at a fraction of the compute cost of the pairwise cosine-similarity comparison our own pipeline already used. The triage pass took maybe twenty minutes: our corpus was a similar order of magnitude to theirs, but their reported numbers came from a dataset of well-formed articles, while a meaningful share of what we processed was short, noisy user-generated text, so I knew going in that a direct comparison to their published numbers wouldn't mean much. I decided the smallest slice worth reproducing was just the hashing-and-banding step the approach relied on, not their full indexing and clustering pipeline, and built a small runnable version of just that against a sample of our own real documents, explicitly accepting that I was skipping their canonicalization preprocessing to keep it fast. I then ran it head to head against our existing pairwise comparison on the same sample, measuring both duplicate pairs found and wall-clock time, rather than comparing to their published numbers, and it matched our existing method's results about ten times faster, but only once I'd widened their suggested hash-band parameters, since their published default missed several near-duplicates that were common in our noisier text. I wrote a short note with the parameter change and the before-and-after timing, and we adopted it as the pipeline's first-pass filter, keeping the slower pairwise comparison as a confirming check on anything it flagged as a near-miss.

Trade-offs and pitfalls

The clearest trap is trusting a paper's reported numbers as if they'd transfer directly to your own situation, when they were almost always measured under different conditions. The same is true of a method's tuned parameters, not just its headline numbers: the published defaults are calibrated for the paper's own data and may need to be re-derived for yours before the comparison is fair. The opposite trap is full-fidelity reproduction of something a day-long scoped test would have been enough to evaluate, which burns real time on a claim that didn't need that much rigor to check. A published venue or well-known authors can also create false authority that skips the validation step entirely, which is exactly the habit this whole approach is meant to guard against.

Direct answer
When a client keeps retransmitting its SYN and never sends the final ACK (while the server's SYN-ACK is only retransmitted once before the connection times out), the most likely causes are that the client's connect timeout hasn't fired yet, that something between the two hosts is dropping the ACK specifically (an asymmetric path or a stateful device confused about direction), or that the client-side application itself never actually attempted the ACK due to a bug. A different failure shape, the server sends SYN-ACK and the client immediately sends RST, points to a different family of causes entirely: the client rejecting the connection outright.

Structured elaboration
For the "client retransmits SYN, no final ACK" pattern, work through causes in order of likelihood:

1. Asymmetric routing dropping only the return-to-forward-direction ACK path. If the SYN-ACK reaches the client (we know it does, since the client keeps retransmitting new SYNs rather than giving up, meaning it IS getting a response of some kind) but the client's ACK can't get back to the server along some other path, a stateful firewall or NAT device on that asymmetric path may be dropping the ACK because it doesn't recognize the connection's state in that direction.
2. A middlebox is rewriting or dropping specific TCP options in the SYN-ACK that the client's stack doesn't handle gracefully, causing it to silently discard the SYN-ACK and retry instead of ACKing it.
3. Client-side firewall or security policy is specifically blocking outbound ACKs to that destination while permitting the outbound SYNs, an unusual but real misconfiguration.
4. MTU (Maximum Transmission Unit)-related silent packet loss on the SYN-ACK's return path if it happens to be an unusually large segment (rare for a SYN-ACK specifically, since it typically carries little payload, but worth ruling out if other symptoms point that way).

For the different shape (SYN-ACK followed by an IMMEDIATE client RST): this usually means the client-side application decided, upon establishing the connection, that it doesn't actually want it, for example an application-level timeout that already expired while the handshake was in flight, a client-side connection pool that raced two connection attempts and is aborting the loser, or a security tool on the client actively resetting connections that don't match an expected certificate or policy.

Worked example
To distinguish these hypotheses in practice, compare timestamps and evidence on BOTH ends: if the server's capture shows the SYN-ACK leaving on time but the client's capture never shows it arriving, the problem is in the path (asymmetric routing, a device eating it). If the client's capture shows the SYN-ACK arriving cleanly but no ACK is ever generated by the client's own stack, the bug is on the client host itself (application logic, local firewall) rather than the network path.

Trade-offs & pitfalls
A common mistake is assuming a stuck handshake is always a network problem; a client-side timeout race (the application gives up right as the handshake completes) produces an outwardly identical-looking symptom to a network drop and is only distinguishable by comparing what each side's own capture actually shows, not by reasoning about the network path alone.

A grub rescue> prompt means GRUB (GRand Unified Bootloader) could not find its own modules or configuration, not that the kernel or initramfs is broken. On a UEFI (Unified Extensible Firmware Interface) machine that almost always means the EFI System Partition, or ESP (the small FAT32 partition the firmware boots from directly), is missing, unmounted, reformatted, or no longer matches what the firmware's NVRAM (the firmware's persistent boot-entry memory) boot record points at. The fix is to boot a live USB, mount the installed root and the ESP under a working directory, bind the live kernel's /dev, /proc, and /sys into that tree, chroot (change root, so every command you run afterward executes against the installed system's binaries and libraries instead of the live USB's) into it, reinstall GRUB against the correct ESP, regenerate the GRUB configuration, and rebuild the initramfs (the small temporary root filesystem the kernel loads first, before it can find and mount the real root) before rebooting.

Repair sequence

flowchart TD
    A["Boot live USB in UEFI mode"] --> B["lsblk -f: identify ESP and root"]
    B --> C["cryptsetup luksOpen and vgchange -ay if encrypted"]
    C --> D["mount root at /mnt/root"]
    D --> E["mount ESP at /mnt/root/boot/efi"]
    E --> F["bind /dev /proc /sys /run"]
    F --> G["chroot into /mnt/root"]
    G --> H["grub-install target x86_64-efi"]
    H --> I["update-grub then update-initramfs -u -k all"]
    I --> J["efibootmgr -v, unmount, reboot"]

1. Boot the live USB in the same firmware mode

Most firmware shows two entries for the same USB stick: one prefixed "UEFI:" and a plain legacy one. Pick the UEFI entry. Booting the live session in legacy (BIOS) mode and repairing from there produces a BIOS-style GRUB install that will not satisfy a UEFI firmware, so nothing changes on reboot.

2. Identify the real partitions, do not guess

In the live session:

lsblk -f

You are looking for two things: the ESP, a small (commonly 100 to 512 MiB) partition with filesystem type vfat (the FAT32 format UEFI firmware requires for the ESP), and the OS root, which shows as one of the common Linux filesystem formats (ext4, xfs, or btrfs, each just a different on-disk structure for storing files), an LVM (Logical Volume Manager) member, or, if disk encryption is in use, a partition whose type is crypto_LUKS rather than a normal filesystem. Do not assume the first vfat partition you see is the right one: a disk can carry a vendor recovery partition or a second OS's ESP that is the same size and type. Mount the candidate root read-only (mount -o ro) first and read its own /etc/fstab (the config file listing every filesystem the OS expects to mount at boot, and where) to confirm which device it expects at /boot/efi before you write anything to a partition.

3. Open the encrypted root, if it uses LUKS

If root sits on LUKS (Linux Unified Key Setup, the standard Linux full-disk-encryption format), it will not mount until you decrypt it:

cryptsetup luksOpen /dev/sda3 cryptroot
vgchange -ay          # only if LVM sits on top of the LUKS container

This exposes the plaintext block device at /dev/mapper/cryptroot (or activates the logical volumes on top of it).

4. Mount root and ESP, then bind the live kernel's virtual filesystems

mkdir -p /mnt/root
mount /dev/mapper/cryptroot /mnt/root      # or /dev/sda3 directly if root is not encrypted
mount /dev/sda1 /mnt/root/boot/efi         # the ESP, at the SAME path the installed OS expects

for d in dev proc sys run; do
  mount --bind /$d /mnt/root/$d
done

/dev, /proc, and /sys are what the question asks for; binding /run too is not strictly required by GRUB itself but avoids errors from udev (the kernel's device-node manager, which creates and updates entries under /dev), D-Bus (the inter-process messaging system many system services use to talk to each other), and os-prober (a boot-time script that scans for other installed operating systems to add to the GRUB menu), plus the LUKS initramfs hook while you are inside the chroot.

5. Chroot in and repair GRUB

chroot /mnt/root /bin/bash
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck
update-grub
update-initramfs -u -k all
exit

--target=x86_64-efi tells grub-install to write EFI application binaries instead of a BIOS boot sector. --efi-directory must point at the ESP you just mounted. --bootloader-id names both the directory under EFI/ on the ESP and the NVRAM label (here it lands at /boot/efi/EFI/ubuntu/grubx64.efi). update-grub is Debian and Ubuntu's wrapper around grub-mkconfig -o /boot/grub/grub.cfg; it regenerates the boot menu from whatever kernels are actually installed. update-initramfs -u -k all rebuilds the initramfs for every installed kernel, which matters because the update that broke GRUB may also have shipped a new kernel or a driver whose initramfs hook never ran.

6. Verify the firmware boot entry, then unmount and reboot

efibootmgr -v

Confirm an entry points at \EFI\ubuntu\grubx64.efi on the right disk. grub-install does not always create an NVRAM entry when it runs inside a chroot (this depends on whether efivarfs (the virtual filesystem that exposes the UEFI firmware's own variables, including boot entries, to the running OS) was writable from the live session), so add one explicitly if it is missing:

efibootmgr -c -d /dev/sda -p 1 -L "ubuntu" -l '\EFI\ubuntu\grubx64.efi'

Then leave and unwind everything in reverse order:

exit
umount -R /mnt/root
reboot

Worked example

Take a concrete layout: /dev/sda1 is a 512 MiB ESP (vfat, meant to mount at /boot/efi), /dev/sda2 is a 1 GiB /boot (ext4), and /dev/sda3 is an LVM physical volume holding a volume group ubuntu-vg with a root logical volume, unencrypted for this pass. After booting the live USB, lsblk -f shows exactly that layout, so the repair is:

vgchange -ay ubuntu-vg
mkdir -p /mnt/root
mount /dev/ubuntu-vg/root /mnt/root
mount /dev/sda2 /mnt/root/boot
mount /dev/sda1 /mnt/root/boot/efi
for d in dev proc sys run; do mount --bind /$d /mnt/root/$d; done
chroot /mnt/root /bin/bash
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck
update-grub
update-initramfs -u -k all
exit
umount -R /mnt/root

Every flag here (grub-install's --target, --efi-directory, --bootloader-id; update-grub as a one-line wrapper around grub-mkconfig -o /boot/grub/grub.cfg; update-initramfs -u -k all) matches GRUB 2.12 and initramfs-tools as they ship on current Ubuntu LTS. What changes between servers is only the device and volume names, never the shape of the sequence.

Trade-offs and pitfalls

The judgment call that separates a clean repair from a worse mess is resisting the urge to reformat or recreate the ESP. A genuinely corrupted ESP is rare; a correctly-formed one that was simply mounted at the wrong path, or targeted with the wrong --efi-directory, is common, and recreating it throws away the partition GUID (Globally Unique Identifier, the unique ID stamped onto the partition itself when it's created) the firmware's existing NVRAM entries may still reference.

The circuit breaker pattern stops calling a failing dependency once it's clearly unhealthy, so callers fail fast instead of piling up waiting on a dependency that isn't going to answer, and the dependency gets breathing room to recover instead of being hit with an ever-growing retry storm on top of whatever's already wrong with it.

The three states

stateDiagram-v2
  [*] --> Closed
  Closed --> Open: failure threshold crossed
  Open --> HalfOpen: cooldown elapses
  HalfOpen --> Closed: probe requests succeed
  HalfOpen --> Open: probe requests fail

Choosing the threshold and window for a real dependency

Base the threshold on the dependency's own historical baseline, not a round number picked by feel: if a dependency's normal error rate is 1-2%, a threshold like "error rate exceeds 50% over a 1-minute window" is a real signal of degradation, not noise. Combine multiple signals rather than trusting one: an error-rate threshold alone can be fooled by a burst of retriable timeouts, so pairing it with a consecutive-failure count and a latency percentile (for example, p99 exceeding a set ceiling) catches degradation that shows up as slowness before it shows up as outright errors.

Worked example: why the half-open probe count matters

Say the breaker opens, waits out its cooldown, and moves to half-open, sending 5 probe requests before deciding whether to close. If the dependency is still genuinely degraded, with a true underlying failure rate of $p = 0.3$ (30% of calls failing), the probability that all 5 probes happen to succeed by chance despite that is:

That's not a rare fluke, it's roughly a 1-in-6 chance of prematurely closing the breaker on a dependency that's still 30% broken, which then immediately re-floods it with full traffic and likely reopens the breaker on the very next window. This is the concrete argument for either using more probes (the same calculation with 10 probes drops the false-close probability to $0.7^{10} \approx 0.028$, about 2.8%) or ramping traffic gradually after a half-open success instead of jumping straight from 5 probes to 100% traffic.

Trade-offs and pitfalls

Setting the threshold too sensitive (a low error-rate bar or a short window) causes flapping: the breaker opens on transient noise, degrades the user experience with unnecessary fallbacks, and can itself become a source of alerts nobody trusts. Setting it too lax delays protection long enough for the caller's own retries and connection-pool exhaustion to cascade into a second incident on top of the first. The half-open probe-count math above is the same trade-off in miniature: too few probes risk a premature, false-positive close; too many probes delay recovery and keep failing extra requests during the test window. In practice this is tuned with production data and game-day testing rather than picked once and left alone, and the same three-state logic applies regardless of what's on the other side of the call, an AI inference endpoint that starts throwing GPU-OOM errors under load trips the same breaker, on the same threshold logic, as a slow downstream REST dependency; only the specific error signal being watched changes.

Direct answer

When packet loss is reported but interface counters on the involved devices show no errors, look above and below where standard counters measure: transient microbursts and per-queue tail drops that come and go faster than a counter's polling interval can capture, QoS shaping or policing discarding traffic by policy rather than by fault, and hardware-level buffering behavior (bufferbloat, offload features) that hides the real picture from a simple errors/drops counter.

Structured elaboration

1. Understand what standard interface counters actually measure, and their blind spot: most polled counters (SNMP or show interface) sample at intervals of seconds; a microburst that fills a queue and causes a tail drop for a few milliseconds, then clears, can produce zero visible increment in a counter polled every 30 or 60 seconds, even though real packets were genuinely dropped.
2. Look for per-queue, high-resolution telemetry instead: many modern switch ASICs expose per-queue drop counters (as opposed to aggregate interface-level counters) at a finer resolution; if available, these can reveal drops on a specific priority queue that never surface in the aggregate interface statistics.
3. Check whether QoS shaping or policing is discarding traffic by design, not by fault: a policer enforces a committed rate and deliberately discards or remarks traffic above that rate at ingress, usually incrementing a policy-specific counter (a conform/exceed/violate counter on the policy-map) rather than the generic interface error/drop counter an engineer checks first; a shaper, by contrast, delays and queues excess traffic rather than dropping it outright, so it manifests as added latency and jitter rather than loss, unless its own buffer also overflows. A recent QoS policy change (a lowered committed rate, or a class reclassified into a stricter policer) is a common, entirely policy-driven cause of loss that will never show up as an interface error.
4. Consider bufferbloat as a related but distinct pattern: an oversized buffer does not drop packets outright, but holds them long enough to inflate latency dramatically under load; this can look like loss to an application with a tight timeout (the packet was never actually dropped, but arrived too late to be useful), so distinguish true loss from excessive queuing delay using timestamps, not just counters.
5. Check for hardware offload masking the real picture: some NICs and switch ASICs handle certain processing (checksums, some queueing decisions) in hardware in ways that are not reflected in the counters the OS or standard management interface exposes; a discrepancy between what the application experiences and what standard counters report can be a sign that the relevant activity is happening below where those counters look.
6. Correlate timing precisely: gather the highest-resolution telemetry available (ASIC-level counters, per-queue stats, or policy-map conform/exceed/violate counters if accessible) and correlate the exact timestamps of reported application-level loss against any spike in queue depth, utilization, or policing activity at that same moment, even a spike too brief for a standard 30-second poll to register.

Worked example

An application reports occasional lost requests. Standard show interface counters on every device in the path show zero errors or drops over the reporting period. The policy-map attached to that egress interface, however, shows a nonzero and growing exceed counter under a QoS policer applied to this traffic class; a recent change lowered the committed rate for that class as part of a broader capacity reallocation. Enabling per-queue statistics on the relevant egress interface (polled every 1 second instead of every 60) corroborates this, showing brief spikes where the policed class's queue hits its maximum and experiences tail drops lasting under two seconds, precisely correlated with the timestamps of the application's reported failures; neither the standard 60-second interface counters nor a naive check of the interface's own drop counter would have surfaced this, since the drop is a deliberate policy action recorded in a QoS-specific counter.

Trade-offs & pitfalls

'The counters are clean' is often treated as proof there is no network-side loss, but standard interface counters have a real, specific blind spot for both short-duration events and policy-driven drops recorded elsewhere (in QoS policy-map counters, not the interface's own error/drop counters); before concluding the network is innocent, confirm you have looked at the highest time-resolution telemetry actually available on that hardware, and at any QoS policy applied to the affected traffic class, not just the generic interface counters. Distinguishing true drops (tail drop, policing) from bufferbloat-induced delay matters because the fixes are different: one calls for capacity, queue-management, or policy-rate changes, the other for buffer-sizing and queue-discipline tuning.