Apple Site Reliability Engineer (Mid-Level) Interview Preparation Guide 2026

SSH public-key auth: the client holds a private key; the server stores the matching public key. During auth the server sends a challenge the client signs with its private key; the server verifies using the public key in ~/.ssh/authorized_keys.

Steps:

• Generate: ssh-keygen -t ed25519 (or rsa 4096); keep private (/.ssh/id_ed25519) and public (/.ssh/id_ed25519.pub).
• Install public key: append the .pub contents to remote ~/.ssh/authorized_keys; ensure permissions (700 ~/.ssh, 600 authorized_keys).
• Secure private key: chmod 600, use a strong passphrase, store on encrypted disk/credential manager.
• ssh-agent: holds unlocked private keys in memory so you don’t retype passphrases; add keys with ssh-add.
• Agent forwarding: forwards your local agent socket to a remote host so you can SSH onward without private key on the remote. Use only to trusted hosts (ForwardAgent yes) — it exposes your agent to that host's root-level processes.

Common pitfalls:

• Incorrect file/dir permissions (most common), wrong key format (copying newline/extra chars), SELinux/umask issues, forgetting to restart sshd after config changes, enabling agent forwarding to untrusted machines, or using weak/unencrypted private keys.

Direct answer
In the first 15 to 30 minutes: acknowledge the page immediately so others know someone is on it, check your two or three primary signal sources (error-rate dashboard, recent deploys, recent config changes) to get a rough sense of scope and cause, apply an obviously safe temporary mitigation if one exists, post a first status update even if it just says 'investigating,' and decide whether you need more people. What you don't do: silently investigate alone for the whole window, or start deep root-causing before you know how big the problem is.

Structured elaboration

• Acknowledge and orient (minute 0-2). Confirm the page is real, not a flapping alert, and check the primary dashboard to see current error rate, latency, and traffic. This tells you if you're dealing with a full outage or a partial degradation.
• Check for an obvious cause (minute 2-8). Look at what changed recently: deploys, config pushes, feature flag flips, infrastructure changes. Most production incidents correlate with a recent change, and this is the highest-value early check.
• Decide on scope and escalation (minute 5-10). Based on what you've seen, decide if this affects one service or many, one region or all, and whether you need to page anyone else. Getting help early is cheap; struggling alone for 25 minutes before asking is expensive.
• Apply a safe temporary action if available (minute 10-20). If a recent deploy correlates with the problem, rolling it back is usually the safest first move, since it's typically reversible. If there's no obvious cause, avoid guessing with an irreversible action.
• Post a first status update (by minute 15-ish). Even 'we're aware, investigating, will update in 15 minutes' is far better than silence, both for stakeholders and for your own discipline of staying on a cadence.
• What not to do: don't start a deep root-cause investigation before you understand the blast radius; don't make an irreversible change (a manual database edit, a permanent config change) under time pressure without a second person's input; don't go quiet.

Worked example
Page fires at 14:02 for elevated error rate on the checkout service. 14:02-14:03: engineer acknowledges, opens the dashboard, sees error rate at 8% (up from a normal 0.1%), affecting checkout only, not the whole site. 14:03-14:06: checks recent deploys, finds one shipped at 13:55, seven minutes before the alert. 14:06-14:08: decides this is likely deploy-related, pages a second engineer to help verify while they prepare a rollback, and opens an incident ticket marked SEV2 (partial, revenue-affecting feature). 14:08-14:11: rolls back the deploy. 14:11-14:14: watches error rate drop back to 0.1% and holds there. 14:15: posts 'checkout errors have returned to normal following a rollback of a recent deploy; monitoring to confirm and will follow up with a summary.'

Trade-offs and pitfalls
Spending the first 15 minutes purely gathering information without acting risks letting an easily-mitigated problem run longer than necessary; acting immediately without any scoping risks mitigating the wrong thing entirely, or worse, taking an action whose blast radius you don't understand. The 'lone hero' pattern, where a responder tries to fully resolve the incident single-handedly before looping anyone in, is a recurring failure mode: it delays getting a second set of eyes on a decision made under pressure, and it delays stakeholder communication that people are actively waiting on.

Situation: Multiple services are being killed by the OOM killer across nodes at peak load. Immediate goals: stop user-impacting kills, gather evidence, restore stable service, and produce a remediation plan.

1. Triage & contain (0–15 min)

• Silence noisy alerts, page on-call war room, assign roles (commander, scribe, infra, app owner).
• Short-term mitigation: reduce load — enable circuit-breakers, scale up replicas if autoscaling exists, disable non-essential batch jobs, divert traffic via load balancer.
• If immediate memory spike persists, cordon/evacuate worst nodes: kubectl cordon <node>; drain if safe: kubectl drain <node> --ignore-daemonsets --delete-local-data.

2. Evidence collection (0–30 min)

• On affected node(s): check kernel OOM logs: sudo dmesg --ctime | grep -i oom -A5 -B5
• Inspect syslog/journal: sudo journalctl -k --since "15 minutes ago" | grep -i oom; sudo journalctl -u kubelet --since "30m"
• For containers/pods: kubectl describe pod <pod> -n <ns>; kubectl logs --previous <pod> -n <ns>
• Check memory and cgroups: cat /sys/fs/cgroup/memory/kubepods.slice/.../memory.usage_in_bytes; docker stats / crictl stats
• Node memory state: free -h; vmstat 1 5; top -b -n1 | head -n20
• Check kube events: kubectl get events -A --sort-by='.lastTimestamp'
• Collect heap/stack traces from apps if possible; capture pmap /proc/<pid>/smaps for high-PID processes.

3. Root-cause analysis (30–90 min)

• Correlate OOM timestamps with deployments, cron jobs, traffic spikes, GC pauses, memory leaks.
• Determine scope: single service vs. system-wide vs. kernel/pagecache pressure.
• Identify culprit: consistent process names/PIDs from dmesg and pod restarts; check memory limits vs. requests in pod specs.
• Verify if OOM is triggered inside container cgroup (container killed) or by host (oom_reaper killing system processes).

4. Short-term fixes (minutes–hours)

• If container memory limits too low: increase limits temporarily (kubectl patch deployment ... --type='json' ...)
• If node underprovisioned: add nodes or increase instance size; scale nodepool.
• If memory leak in app: rollout previous stable version, revert recent change.
• Tune OOM behavior: set oom_score_adj for critical processes; avoid killing kubelet/evicted system daemons.
• Enable swap only if validated (short-term stopgap), or tune vm.overcommit_memory and vm.panic_on_oom.

5. Long-term remediation (days–weeks)

• Root cause fix: memory leak patch, GC tuning, or adjust workload memory usage.
• Capacity planning: increase headroom, autoscaling thresholds, add node pools for memory-heavy services.
• Kubernetes best-practices: set accurate requests/limits, use Vertical Pod Autoscaler, pod disruption budgets.
• Observability: add fine-grained memory metrics (process RSS, container OOM events), alerts on memory headroom, and end-to-end tracing to find slow requests causing memory growth.
• CI: add stress tests/soak tests and memory regression tests; add automated heap-dump collection on OOM.
• Run post-incident review: timeline, root cause, action items, owners, deadlines; track in incident tracker.

6. Commands & logs checklist (copyable)

• sudo dmesg --ctime | grep -i oom -A5 -B5
• sudo journalctl -k --since "1 hour ago" | grep -i oom
• kubectl describe pod <pod> -n <ns>
• kubectl logs --previous <pod> -n <ns>
• kubectl get events -A --sort-by='.lastTimestamp'
• free -h; vmstat 1 5; top; ps aux --sort=-rss | head
• cat /proc/<pid>/smaps or pmap -x <pid>
• crictl stats or docker stats

Result: immediate user impact minimized by reducing load and/or scaling; evidence gathered to identify whether root cause is application memory leak, misconfigured limits, or insufficient capacity; remediation plan created with short-term fixes and long-term reliability improvements assigned to owners.

Direct answer

Tier storage by how recently and how urgently data is queried, not by a single uniform policy: keep a short hot window fully indexed for fast incident response, a longer warm window with reduced indexing and cheaper storage, and a cold archive that's raw and compressed with only a thin metadata index for lookup. Index only the fields you actually search or alert on, and cut ingested volume before it ever reaches storage (dropping known-noisy events, sampling verbose debug logs, deduplicating repeats) rather than trying to compress your way out of ingesting everything at full fidelity.

Structured elaboration

Tiering by access pattern

• Hot (recent, e.g. the last several days): fully indexed and parsed, on fast storage, because this is the window most incident investigations actually query.
• Warm (weeks to a couple of months): reduced indexing, kept for less-frequent-but-still-plausible investigations, on cheaper storage.
• Cold (long tail out to the retention limit): raw, compressed, minimally indexed (just enough metadata, e.g. timestamp/service/id, to locate and rehydrate a specific slice), on the cheapest available storage.

What to index versus what to keep raw
Index only the fields actually used in searches and alerts (service, severity, request id, error code, user id) rather than full free-text message bodies by default. Indexing costs compute and storage roughly proportional to what you index, so indexing everything "just in case" is usually what makes a fully-indexed retention policy expensive in the first place. Keep the raw, unindexed payload in cheap storage so a rare deep investigation can still re-parse it on demand, even though day-to-day queries never touch it.

Reducing ingested volume before it lands anywhere

• Drop known-noisy, low-value events at the source: health-check pings, synthetic monitoring traffic that's already captured elsewhere.
• Sample verbose debug-level logs outside of active incidents, while retaining 100% of error/warn-level events; a debug log's value degrades fast once nothing is actively wrong, but an error's value doesn't.
• Deduplicate: collapse many identical repeated events within a short window into one record with a count and a first/last timestamp, rather than storing each occurrence individually.

A budget-driven decision framework
Rather than picking retention, sampling, and rollup settings independently and hoping the bill comes out reasonable, work backward from a fixed observability budget: classify data by purpose (alerting needs short-window full fidelity; long-term trend analysis can tolerate rollups; compliance may fix a minimum retention regardless of cost), then allocate the budget so the fidelity you actually rely on (recent, alert-relevant data) is protected first, and lower-value long-tail data absorbs the compression.

Worked example

Compare two policies for a service ingesting 500 GB/day of raw logs, over a 90-day retention requirement: "fully indexed for all 90 days" versus a tiered plan.

Fully indexed, 90 days, no tiering (assume indexing overhead adds 30% on top of the compressed size, and compression brings raw data to 25% of its original size, both stated assumptions used consistently below):

Tiered plan (hot: 7 days, fully indexed, same 0.25 compression and 1.30 index overhead; warm: days 8-30, still compressed 0.25 but only 30% of it indexed; cold: days 31-90, raw compressed only, no index, using a better 0.10 compression ratio typical of infrequently-touched archival data):

Under these stated assumptions, the tiered plan stores about half the volume of the fully-indexed policy (7,271 GB vs 14,625 GB) for the same 90-day retention window, purely from indexing less of the older data and compressing the coldest tier harder; it doesn't require dropping a single day of retention. The actual ratio in a real system depends entirely on the true compression and index-overhead numbers for the specific backend in use, which is why this example states its assumptions rather than presenting the savings as a general industry figure.

Trade-offs & pitfalls

• Rehydrating cold data for an investigation has a retrieval delay; make sure whoever owns compliance sign-off on the retention policy accepts that trade explicitly, rather than discovering it during an actual audit.
• Sampling debug logs outside incidents means you can't always sample-up retroactively for an incident that's already happened; if incident detection is delayed, the fine-grained debug data for the affected window may already be gone. Pair aggressive debug sampling with a short full-fidelity buffer (e.g., last 24-48 hours at 100%) to cover that gap.
• Deduplication that collapses "identical" events too aggressively can hide a subtle change (an error message that's 99% identical but has a materially different stack trace) if the dedup key isn't specific enough.
• Under-indexing to save cost can make routine investigations slower even when they're within the retention window; the goal is indexing the fields actually queried, not indexing as little as possible.

Direct answer

At 500 hosts, the real question isn't 'which tool is technically capable' (both are), it's which one you want to own the operational complexity of: inventory management, secrets distribution, and failure isolation, versus building and maintaining that yourself.

Scalability and inventory management

Ansible's inventory system (static files, dynamic inventory scripts/plugins for cloud providers) is a mature, well-tested solution to 'which hosts, grouped how, with what per-group variables' -- a custom concurrent.futures + paramiko runner has to build this from scratch, and inventory management is a deceptively large surface (host groups, per-host overrides, dynamic discovery) to get right. At 500 hosts specifically, both approaches can technically execute concurrently at that scale; the difference is how much of the surrounding machinery (retry-per-host, connection pooling, structured per-host result reporting) you get for free versus have to build.

Secrets distribution

Ansible Vault (or integration with an external secrets manager via lookup plugins) has existing, audited patterns for distributing credentials to a fleet without them landing in plaintext on every target host. A custom runner needs to solve this itself -- likely reusing the secrets-handling patterns discussed elsewhere in this topic (short-lived credentials fetched per-connection rather than distributed and stored), which is very doable but is additional design and implementation surface Ansible gives you largely for free.

Retry behavior and idempotency

Ansible's module system is BUILT around idempotency as a first-class concept -- most built-in modules check current state before acting and report 'changed' vs 'ok' accordingly, which is exactly the idempotency discipline this whole topic argues for, already baked into the tool. A custom runner running an arbitrary script over SSH has NO such guarantee -- idempotency (or the lack of it) is entirely a property of whatever script you're running, and the runner itself only adds retry-on-connection-failure, not retry-safety for the underlying operation.

When the custom runner is still the right call

Despite Ansible's advantages above, a custom concurrent.futures + paramiko runner is justified when: the operation is a one-off, tightly-scoped action that doesn't fit Ansible's declarative module model well; you need custom batching/failure-isolation logic Ansible's default execution strategy doesn't offer out of the box (e.g., 'stop launching new batches immediately if failure rate exceeds 10%, but let in-flight batches finish'); or the team is already deep in a Python-based automation framework (per the language-choice discussion elsewhere in this topic) and adding Ansible as a second tool has its own onboarding/maintenance cost that isn't worth it for this one task.

Batching and failure isolation with a custom runner

If building custom: batch hosts into fixed-size groups (e.g. 50 at a time) submitted to a bounded ThreadPoolExecutor, track per-host success/failure independently (one host's connection failure must never abort or block the others), and implement a circuit-breaker-style check between batches -- if failure rate in the last batch crosses a threshold, pause and surface to an operator rather than blindly continuing through all 500 hosts and only discovering a systemic problem (bad SSH key rotation, a network partition) after every host has already failed.

Trade-offs and pitfalls

The most common mistake building a custom runner is treating the FIRST version as sufficient to reach production without the retry, secrets, and failure-isolation depth Ansible already has -- what starts as 'this is simpler than learning Ansible' quietly grows into reimplementing large parts of it, worse and less tested, exactly the scope-creep risk the custom-vs-mature-tools trade-off discussed elsewhere in this topic warns about.

AXFR vs IXFR (brief):

• AXFR = full zone transfer: slave requests entire zone from master. Good for first sync or when many changes happened.
• IXFR = incremental zone transfer: slave requests only deltas based on serial numbers (efficient when few changes).

Typical slave pull flow:

1. Slave queries SOA on master; compares serial.
2. If serial higher, it requests IXFR; if master can’t provide IXFR or slave asks, AXFR is used.

Common failure causes:

• Network blocks: firewall/ACL blocking TCP/53 (zone transfers use TCP). Also NAT issues or asymmetric routing.
• Serial mismatches: master SOA serial not incremented correctly or bad zone file edits; slave still at higher serial (rare) causing rejects.
• Permissions / configuration: master configured to deny transfers to slave IPs.
• TSIG key mismatch: wrong key name/algorithm/time skew.
• Resource/timeouts: large zones causing timeouts.

How to secure transfers:

• Use TSIG: sign transfers with a shared key (hmac-sha256). Prevents unauthorized pulls.
• IP allowlists: master’s allow-transfer limited to explicit slave IPs.
• Limit AXFR: prefer IXFR; configure policies to only allow AXFR for initial transfers; rate-limit/monitor AXFR requests.
• Monitor and alert on unexpected AXFR attempts.

Debugging steps (practical):

1. Check connectivity: tcp to master on 53
   • curl/nc: nc -vz master.example.net 53
2. Use dig to simulate:
   • Check SOA serial:

dig @master.example.net example.com SOA +short
# returns: ns1.example.net. hostmaster.example.com. 2025120101 7200 3600 1209600 3600

• Request AXFR (will fail if denied):

dig @master.example.net example.com AXFR

• Request IXFR from known serial (e.g., 2025120100):

dig @master.example.net example.com IXFR=2025120100

• Test with TSIG:

dig @master.example.net example.com AXFR -y keyname:base64key

3. Inspect server logs: on BIND check /var/log/messages or query log (named.log) for transfer errors: look for "transfer denied", "tsig verify failure", "network unreachable", "timed out".
4. Reproduce with increased verbosity on slave/master (rndc trace or logging channels).
5. Verify TSIG keys: matching names, algorithms, and that keys are on both sides and not expired.
6. Confirm firewall rules allow TCP 53 and that master’s allow-transfer matches slave IPs.

Result/Best practices:

• Use TSIG + IP allowlist as defense-in-depth, prefer IXFR, restrict AXFR, log/alert on transfers, and include transfer checks in monitoring. Regularly test fresh slave bootstraps to ensure policy and keys work.

Direct answer
TCP Fast Open (TFO) lets a client include application data directly in its SYN packet on a REPEAT connection to a server it has connected to before, skipping the usual "wait for the handshake to finish before sending anything" round trip; the risk is that some middleboxes, built assuming a SYN never carries a meaningful payload, may drop, strip, or misforward that data, or even the whole packet, since it doesn't look like a standard handshake to them.

Structured elaboration
TFO works by having the server issue a cryptographic cookie to the client the FIRST time they connect (during a normal handshake, no fast-open data yet). On any SUBSEQUENT connection attempt, the client includes that cookie plus its actual application data directly in the SYN packet. If the server validates the cookie, it can begin processing that data immediately, effectively saving a full round trip compared to the standard "handshake completes, THEN data starts flowing" sequence.

The middlebox risk comes from an assumption baked into a lot of older network equipment: that a SYN packet is control-plane-only and never carries a meaningful payload. Some middleboxes (older NAT devices, certain firewalls, some load balancers) built on that assumption may strip TCP options they don't recognize (breaking the cookie mechanism itself), drop SYN packets that carry payload as anomalous or suspicious, or in some documented cases, mishandle the connection entirely.

Worked example
The required fallback behavior: if a TFO attempt's SYN (carrying the cookie and data) gets no response, or gets a response that doesn't correctly acknowledge the fast-open data, a correct client implementation must FALL BACK to a standard handshake, retransmitting the SYN WITHOUT the fast-open data and cookie, and only sending the actual application data after the connection is fully, conventionally established. This fallback is what keeps TFO safe to enable broadly: in the worst case (a hostile middlebox on the path), the connection degrades to ordinary TCP behavior rather than failing outright, the client just loses the round-trip savings TFO was trying to provide, it never loses connectivity because of it.

Trade-offs & pitfalls
The whole design of TFO is built around the assumption that middlebox interference IS common enough to plan for, not a rare edge case, that's precisely why the fallback-to-standard-handshake behavior is a REQUIRED part of any compliant implementation, not an optional nicety. An implementation that assumes TFO will always succeed and doesn't implement the fallback path correctly risks connections silently failing specifically on paths that include one of these older devices, exactly the class of path where the round-trip savings would have mattered most (since those paths tend to be higher-latency, longer routes).

Direct answer
Detect and fix deadlocks caused by lock-order inversion in async/coroutine code by enforcing a consistent global lock-acquisition order everywhere multiple locks are held simultaneously, using timeout-aware locks as a safety net that converts a silent hang into a loud, diagnosable failure, and replacing heavy locking with concurrency-safe data structures where possible to eliminate the need for multiple locks in the first place.

Structured elaboration

• The mechanism: a deadlock from lock-order inversion happens when coroutine A holds lock 1 and waits for lock 2, while coroutine B holds lock 2 and waits for lock 1; neither can proceed, and (unlike a thread blocked on a normal mutex, which an OS scheduler or a monitoring tool can often detect via stack inspection) an async deadlock can be harder to spot because both coroutines appear 'suspended', not obviously 'stuck', until you specifically look for the cyclic wait.
• Lock-ordering policy: assign every lock a global, fixed order (by a stable id or a declared hierarchy) and require every code path that needs multiple locks to acquire them in that SAME order; this makes the cyclic-wait condition structurally impossible, since no two code paths can ever be waiting on each other's held lock.
• Timeout-aware locks: acquiring a lock with a timeout, and treating a timeout as a recoverable error (log, back off, retry, or fail the specific operation) rather than blocking forever, converts an invisible hang into a visible, logged, diagnosable event, even if it doesn't fix the underlying ordering bug by itself.
• Concurrency-safe data structures: replacing 'acquire lock A, then lock B, mutate both' with a single operation on a data structure designed for safe concurrent access (an atomic compound operation, a lock-free structure, or a single higher-level lock covering both pieces of state together) removes the multi-lock-acquisition pattern that creates the ordering hazard in the first place.
• Reliably reproducing the deadlock in a test: construct a test that deliberately drives two coroutines to acquire the two locks in OPPOSITE order under controlled scheduling (using explicit yield points or a scheduling hook to force the interleaving), rather than hoping a race happens to occur naturally, since the whole point of a deadlock bug is that it depends on a specific, often-rare interleaving.

Worked example
Two coroutines both need to transfer between two accounts: coroutine A does transfer(account_1_lock, account_2_lock) (lock 1 then lock 2) while coroutine B, transferring in the opposite direction, does transfer(account_2_lock, account_1_lock) (lock 2 then lock 1); under the right interleaving, A holds lock 1 and waits for lock 2 while B holds lock 2 and waits for lock 1, deadlocking both. The fix: always acquire locks in a FIXED order regardless of transfer direction (say, always lock the account with the lower id first), which both A and B now do identically, making the cyclic wait impossible by construction.

Trade-offs and pitfalls
A fixed lock-ordering policy is only as good as its ENFORCEMENT; a policy documented but not checked will eventually be violated by a new code path someone adds without knowing the rule, so pairing it with either a lint/static-analysis check or a runtime assertion (verify at acquisition time that no already-held lock has a higher order than the one being acquired) is what actually makes it durable over time, not the policy alone.

Direct answer

Map weight to virtual-node count linearly (replicas proportional to weight), because that is what preserves the property that a node's expected key share equals its share of total weight, and cap the replica count per node so one extreme outlier weight cannot blow up ring size or per-lookup cost. Adding or removing a node then only remaps the keys that specifically belonged to that node's virtual replicas, everyone else's keys stay put, which is the whole point of consistent hashing over a plain hash-mod-N scheme.

Approach

The core mechanism is the same ring as unweighted consistent hashing (sorted virtual-node positions, binary search for ownership), with two additions: add_node takes a weight, and the number of virtual replicas it creates is proportional to that weight rather than a fixed constant. This keeps a node's fraction of total ring positions equal to its fraction of total weight, so its expected fraction of keys matches too (this is verified numerically below).

Without a cap, a single node with weight 100000 relative to peers at weight 1 would ask for 100000 times the base replica count, which is both a memory problem (ring size) and a lookup problem (O(log n) grows with total replicas). The cap trades strict weight-proportionality at the extreme end for a bounded ring: past the cap, an outlier-weighted node still gets more replicas than its peers, just not linearly more, which is an acceptable trade since a weight ratio that extreme usually means the two backends are different enough in kind that the whole model, not just the replica formula, should be reconsidered.

Code (Python)

import bisect
import hashlib


class ConsistentHashRing:
    def __init__(self, base_replicas=100, max_replicas=4000):
        self.ring = []            # sorted hash positions
        self.nodes = {}           # position -> node_id
        self.node_positions = {}  # node_id -> list of positions (for O(replicas) removal)
        self.base_replicas = base_replicas
        self.max_replicas = max_replicas   # caps ring growth for very large weights

    def _hash(self, key: str) -> int:
        return int(hashlib.sha256(key.encode("utf-8")).hexdigest(), 16)

    def _replica_count(self, weight: float) -> int:
        # Linear in weight: a node's replica share stays proportional to its
        # weight share, which is what makes expected key share == weight share.
        # max_replicas bounds ring memory (O(replicas)) and lookup cost
        # (O(log total_replicas)) against one extreme outlier weight.
        raw = int(round(self.base_replicas * weight))
        return max(1, min(raw, self.max_replicas))

    def add_node(self, node_id: str, weight: float = 1.0):
        replicas = self._replica_count(weight)
        positions = []
        for i in range(replicas):
            h = self._hash(f"{node_id}#{i}")
            if h in self.nodes:
                continue
            bisect.insort(self.ring, h)
            self.nodes[h] = node_id
            positions.append(h)
        self.node_positions[node_id] = positions

    def remove_node(self, node_id: str):
        for h in self.node_positions.pop(node_id, []):
            idx = bisect.bisect_left(self.ring, h)
            if idx < len(self.ring) and self.ring[idx] == h:
                self.ring.pop(idx)
                self.nodes.pop(h, None)

    def get_node(self, key: str):
        if not self.ring:
            return None
        h = self._hash(key)
        idx = bisect.bisect_right(self.ring, h)
        if idx == len(self.ring):
            idx = 0
        return self.nodes[self.ring[idx]]

Key points

• Weight-to-replica mapping is linear, then capped: replicas = clamp(round(base_replicas * weight), 1, max_replicas). Linear preserves proportional key share; the cap bounds ring size.
• node_positions tracks each node's own replica hashes so remove_node only has to touch that node's O(replicas) entries, not scan the whole ring.
• Minimal movement is structural, not incidental: because virtual nodes for different physical nodes are interleaved around the ring, adding one new node only steals the ring segments immediately preceding its own virtual points, wherever those land, from whichever nodes currently own them; it cannot affect a segment it doesn't touch.

Complexity

• add_node: O(replicas * log n) where replicas is capped at max_replicas and n is total virtual nodes on the ring.
• remove_node: O(replicas * log n) for that node's own replicas only.
• get_node: O(log n) for the binary search.
• Space: O(n) total virtual nodes across all live physical nodes, bounded by (number of nodes) times max_replicas.
• Caveat: the O(log n) parts above cover only the position search. bisect.insort and list.pop(idx) mutate a plain Python list, so the actual insertion/removal is O(n) per call (array shift), not O(log n); the true cost of add_node/remove_node is O(replicas * n) in the worst case. A balanced tree or skip list would be needed to make the mutation itself sub-linear at large node counts.

Worked example (weight-to-replica mapping and minimal movement)

Weight-to-replica mapping with base_replicas=100, max_replicas=4000, showing the cap engage at large weight:

weight=      1 -> replicas=100
weight=      4 -> replicas=400
weight=     16 -> replicas=1600
weight=    100 -> replicas=4000
weight= 100000 -> replicas=4000

Weight 100 already saturates the cap at this configuration (100 x 100 = 10000, clamped to 4000); weight 100000 hits the same cap, confirming the ring size stays bounded regardless of how extreme a weight is declared.

Distribution check with three nodes of weight 1, 4, and 1 (total weight 6), over 20000 fixed keys (key-0 through key-19999):

weighted distribution (A:1, B:4, C:1), 20000 keys: {'A': 3483, 'B': 13097, 'C': 3420}
expected B share ~= 4/6 = 0.6667, measured = 0.6549

B's weight share is 4/6 = 0.6667; its measured key share is 0.6549, within normal sampling variance for a ring at this replica density.

Minimal-movement check: adding a fourth node D with weight 2 (new total weight 8):

keys moved after adding D(weight=2): 4583/20000 = 0.2291
of those, moved specifically to D: 4583/4583 = 1.0000 (should be ~1.0)

22.91% of keys moved, close to the 25% expectation, and (critically) every single key that moved, moved specifically to D, none of A/B/C's other keys were disturbed by each other. Removing B afterward confirms the same locality in the other direction:

after removing B: 9957 keys needed reassignment, 0 non-B keys moved

Every key that needed reassignment had previously belonged to B; zero keys that belonged to A, C, or D were touched by B's removal, this is the "minimal key movement" property the question asks to demonstrate, shown numerically rather than asserted.

Edge cases

• Weight rounds down to 0 replicas: guarded with max(1, ...) so every added node gets at least one ring position, even a very small declared weight.
• Extremely large weight: bounded by max_replicas, verified above.
• Removing a node not present: no-op, since node_positions.pop(node_id, []) returns an empty list.
• Empty ring: get_node returns None.

Trade-offs and pitfalls

• The cap breaks strict proportionality at the high end. Two nodes both past the cap (say weight 500 and weight 100000) get the same replica count and therefore roughly the same key share, even though their declared weights differ by 200x; if that distinction matters operationally, the cap needs to be raised or the weight scale needs to be redefined so realistic weights don't approach it.
• base_replicas and max_replicas are a joint tuning knob, not independent ones. Raising base_replicas to smooth distribution variance also lowers the effective weight ceiling before the cap engages (since max_replicas / base_replicas is the highest weight that still gets proportional treatment); changing one without reconsidering the other can silently shrink your usable weight range.
• Weight changes at runtime are a remove-then-add, not an update-in-place, in this design; recomputing a node's replica count means discarding its old positions and generating new ones, which moves keys even for a node that didn't fail, this is worth calling out explicitly if the interviewer asks about live capacity rebalancing.
• Ring divergence across processes is still the same risk as unweighted consistent hashing: every client needs the same node set, weights, base_replicas, max_replicas, and hash function to agree on ownership.

RSA static key transport vs ephemeral Diffie-Hellman (DHE/ECDHE) in TLS

RSA static key transport:

• How it works: Client encrypts a pre-master secret with the server’s long-term RSA private-key certificate (public key). Server decrypts and both derive session keys.
• Weaknesses: If the server’s private key is ever compromised (the cert key or its backups), an attacker can decrypt recorded TLS handshakes and recover past session keys — no forward secrecy. Also vulnerable to certain implementation attacks and less agility for modern cipher suites.

Why ECDHE/DHE is preferred:

• Ephemeral DH: server uses a short-lived (ephemeral) DH key pair for each handshake. Client and server compute a shared secret via Diffie-Hellman; that secret is never transmitted.
• Forward secrecy (PFS): Because ephemeral private keys are not stored long-term, compromise of the server’s certificate private key does not allow decryption of past sessions — past session keys require the ephemeral private values which were discarded.
• ECDHE is faster and uses smaller keys than classic DHE for comparable security (better perf and lower CPU/handshake cost).

Operational considerations for SREs:

• Performance: ECDHE uses less CPU and bandwidth than DHE; choose widely supported curves (e.g., X25519, secp256r1) for low latency and high throughput.
• Compatibility: Maintain a reasonable TLS policy — enable ECDHE by default but retain RSA key-exchange only if you must support very old clients. Use TLS 1.2+ and prefer TLS 1.3 where ECDHE is mandatory.
• Key management: Protect long-term certificate private keys (HSMs, limited access), rotate certs regularly, and ensure ephemeral keys are not logged/stored.
• Monitoring and capacity planning: ECDHE increases handshake CPU; measure and autoscale TLS terminators/load balancers if handshake-heavy (e.g., many short-lived connections).
• TLS termination location: If terminating at load balancers or proxies, ensure they support required curves and are patched for mitigations.
• Compliance and risk: For high-security environments, enforce PFS-only cipher suites and disable RSA key transport; document compatibility impacts and rollback plan.

Summary: Prefer ECDHE (or DHE when ECDHE unsupported) for PFS and performance; reserve RSA static only for legacy compatibility and manage keys and capacity accordingly.