HardTechnical

49 practiced

You operate microservices on Kubernetes and observe suboptimal network throughput and high p99 request latency. Propose TCP and networking tunings (sysctl and socket options like rmem/wmem, tcp_tw_reuse, SO_REUSEPORT), Kubernetes-level changes (CNI MTU, proxy or service mesh tuning, daemonset params), and validation steps for each change (benchmarks and metrics to capture). Discuss safety and rollback strategies.

Sample Answer

Situation: You're seeing poor network throughput and high p99 latency in Kubernetes microservices. Below is a structured SRE plan covering TCP/sysctl/socket tunings, Kubernetes/CNI/service-mesh changes, validation steps and safe rollback.

1) Kernel / TCP tunings (node-level, via sysctl or DaemonSet applying to /proc/sys):- rmem_max / wmem_max (e.g., net.core.rmem_max=16777216, net.core.wmem_max=16777216) and net.ipv4.tcp_rmem / tcp_wmem (min, default, max tuned to allow large window scaling).- net.ipv4.tcp_window_scaling=1 (enable), tcp_timestamps=1 (default), tcp_sack=1.- net.ipv4.tcp_tw_reuse=1 and tcp_tw_recycle=0 (reuse TIME_WAIT sockets for new connections; do NOT enable tcp_tw_recycle on NATted clusters).- net.core.netdev_max_backlog increase (e.g., 5000→10000) and net.core.somaxconn (e.g., 1024→4096).- net.ipv4.tcp_max_syn_backlog increase.- net.ipv4.tcp_congestion_control: consider bbr for high-bandwidth/long-RTT links (test carefully).

Why: Larger socket buffers and window scaling let TCP use available bandwidth; backlogs reduce drops during bursts; safe TIME_WAIT reuse reduces ephemeral port exhaustion.

Safety: Apply gradually, per-node via DaemonSet with readiness checks. Avoid tcp_tw_recycle on NAT. Record baseline before changes.

2) Socket options / app-level:- SO_REUSEPORT to allow multiple process threads/pods to bind same port (improves load distribution at kernel level).- Adjust application socket rcvbuf/wnd if possible.- Keep-alives (TCP_KEEP*), and tune timeouts for long-lived connections.

Why: SO_REUSEPORT reduces kernel lock contention; per-socket buffers align with kernel settings.

Safety: Feature toggle in app or config flag; can roll back by restarting pods with old flags.

3) Kubernetes / CNI:- CNI MTU: match overlay MTU to underlying network (e.g., if underlying MTU is 1500 and VXLAN adds 50 bytes, set CNI MTU to 1450). Mismatched MTU causes fragmentation and throughput loss.- Service mesh / proxy tuning: increase proxy worker threads, connection pool sizes, upstream/outbound connection limits, disable/optimize mTLS if it’s CPU-bound, tune keepalive and idle timeouts.- kube-proxy mode: consider switching iptables→IPVS for better performance at scale.- DaemonSet parameters: increase hostNetwork usage for critical high-throughput services; tune host NIC offloads (GRO/LRO, GSO) and IRQ affinity.- Node sizing: ensure CPU pinned for proxies; ensure no CPU throttling (cpuRequests/limits).

Why: MTU fragmentation kills throughput; proxies/mesh can be bottlenecks; IPVS scales better.

Safety: Change CNI MTU in a controlled upgrade window; test on small node pool first. For mesh/proxy tuning, use canary deployments.

4) Validation & Benchmarks (for each change):- Baseline capture: p50/p90/p99 latency, throughput (req/s, MB/s), retransmits, cwnd distributions, socket queue lengths, system CPU, NIC counters, and packet drops (ifconfig/ethtool).- Tools: wrk/ghz/hey for HTTP; iperf3 for raw TCP; tcpreplay; perf and ss/netstat, netstat -s, ip -s link, ethtool -S, BPF tools (bcc / bpftrace) for packet drops/latency; Prometheus metrics, Grafana dashboards.- Metrics: TCP retransmits, RTOs, time_wait count, active connections, connections/sec, packet drops, CPU on proxy and app, socket queue sizes, P99 latency.- Method: run synthetic load at several levels (baseline, 50%, 100%, 150% expected peak). Change one variable at a time (A/B or canary). Collect traces and compare tail latency and retransmits.

5) Safety and rollback:- Change one knob at a time; use small canary node pool or namespace.- Automate rollback: IaC/Helm charts with previous values ready; DaemonSet that can revert sysctls; job to revert sysctl on failure.- Define abort criteria: e.g., p99 increases >10% or retransmits increase >20% or errors increase.- Gradual rollout: 5→25→50→100% nodes/pods with monitoring gating.- Post-mortem after each change and keep an audit trail.

6) Additional considerations:- If using NAT/gateway load balancers, ensure ephemeral port range (net.ipv4.ip_local_port_range) large enough.- Use kernel BPF/XDP or CNI optimizations (e.g., eBPF-based CNI) for high performance.- For encrypted mesh traffic, test CPU impact; consider offloading TLS or using mTLS only where needed.

Example quick experiment:- Baseline iperf3 between two pods (hostNetwork and non-hostNetwork) measuring throughput and retransmits.- Set CNI MTU to underlying-50 on a subset, run iperf3 and application load test, compare p99 and retransmits.- If improved, roll out per node.

This plan gives targeted tunings, validation methods and safe rollout/rollback steps so you can iteratively improve throughput and reduce tail latency without risking cluster stability.

HardTechnical

32 practiced

Design a streaming analytics pipeline to process 200k events/sec with end-to-end processing latency under 200ms. Choose between Kafka, Pulsar, Kinesis, or alternatives and justify your choice. Describe partitioning strategies, consumer scaling, stateful processing options (local state stores vs remote), processing guarantees (exactly-once vs at-least-once), checkpointing, and backpressure handling.

Sample Answer

Requirements & constraints:- Ingest 200k events/sec with end-to-end latency < 200 ms, high reliability, observable & operationally simple for SRE.

Choice of messaging layer- I’d choose Apache Kafka + a stream processor (Apache Flink) as the primary design. Reason: Kafka’s proven high-throughput, mature ecosystem, strong client libraries, configurable durability, and support for Kafka Transactions + Flink’s robust low-latency stateful processing, checkpointing and backpressure integration. Pulsar is a good alternative (better multi-tenancy and lower tail-latency in some setups); Kinesis can work for managed environments but has shard and latency limitations at this scale and less operational visibility.

High-level architecture- Producers -> Kafka topic(s) -> Flink cluster (consumers/operators) -> sinks (Kafka / DB / metrics)- Observability: Prometheus/Grafana, Kafka JMX, Flink metrics, distributed tracing.

Partitioning strategy- Partition by the unit of parallelism that keeps hot keys balanced. If events include a natural key (customer, device), use key-based partitioning to preserve ordering for that key. To avoid hotspots, use hash(key) % N, or use a dual-key (key + coarse shard) approach for heavy keys and re-aggregate in-process.- Sizing: target per-partition throughput < ~5–10k events/sec to keep consumer CPU, IO, and GC in budget for sub-200ms latency. For 200k eps that implies 20–40 partitions minimum; use ~100 partitions for headroom and parallelism for mixed workloads.

Consumer scaling- Use Kafka consumer groups; each Flink parallel subtask reads from one or more partitions. Set Flink parallelism = number of Kafka partitions (or a divisor) to maximize locality.- Scale consumers horizontally: add Flink task slots/instances; rebalance on scale events. Use rolling restarts and warm-up to avoid long rebalances.

Stateful processing & state backend- Use Flink with RocksDB local state backend (embedded local RocksDB per task) for large state and fast local access. RocksDB + local SSDs gives low read/write latency.- Persist checkpoints to durable remote storage (S3/GCS) so tasks can restore after failure.- Avoid purely remote state (e.g., remote key-value store on every access) — remote state increases per-event tail latency. If cross-process sharing is required, use change-log topics (Kafka) and local state for reads.

Processing guarantees & checkpointing- Target exactly-once semantics end-to-end where necessary: Flink’s two-phase commit sinks + Kafka transactions provide end-to-end exactly-once between Kafka ↔ Flink ↔ sink (e.g., Kafka, transactional DB).- Checkpointing: configure frequent lightweight checkpoints (e.g., 500ms–2s) balancing throughput vs stable restore points. For sub-200ms processing, make sure state access is local and checkpointing is asynchronous to avoid pausing processing.- Configure checkpoint timeout and incremental checkpoints (RocksDB incremental snapshots) to reduce IO.

Backpressure & overload handling- Leverage Flink’s built-in backpressure propagation: when downstream slows, Flink applies backpressure upstream. Ensure Kafka producers support rate limiting and retries with backoff.- Implement ingress shaping: token-bucket at API / producer layer, circuit breakers for spikes, and a fallback path (accept to a lower-priority topic or batch write) for burst smoothing.- Autoscaling: monitor key metrics (lag, processing latency, CPU, GC, task backpressure) and scale Flink task managers and Kafka consumer groups automatically. For Kafka, add partitions proactively; use partition reassign with care (rolling reassignment).

Operational tuning for <200ms latency- Keep producer batch sizes small (low linger.ms) for low ingest latency but balance with throughput—experiment (e.g., linger.ms 1–5ms, batch.size tuned).- Use compression (lz4/zstd) for network efficiency but ensure CPU cost doesn’t push latency—lz4 is a good middle ground.- Ensure SSDs, tuned disk scheduler, sufficient network bandwidth (10Gb+), and GC tuning (avoid full GCs).- Keep per-partition message size small; avoid synchronous calls to external systems inside hot loops.

Trade-offs & alternatives- Pulsar: better isolation, built-in tiered storage and lower tail-latency for some workloads — consider if multi-tenant or geo replication is critical.- Kinesis: simpler managed option but shard limits and higher per-unit cost; harder to guarantee <200ms consistently.

Failure & recovery- Test restores using production-representative state. Use quorum and monitor Kafka ISR, topic under-replicated partitions, Flink checkpoint success rate.- Maintain SLOs, alerts on consumer lag, checkpoints failing, GC pauses, and disk saturation. Use canary traffic and gradual rollouts.

Summary- Kafka + Flink with RocksDB local state, Kafka transactions + Flink 2PC for exactly-once, ~100 partitions for headroom, frequent asynchronous checkpoints to S3, ingress rate limiting and autoscaling, and careful latency tuning will meet 200k eps with <200ms E2E if hardware/networking and partitioning are sized correctly.

EasyTechnical

33 practiced

Describe what p50, p95 and p99 percentiles mean for request latency and why p99 is often more important for user experience. Given latency samples (ms): [10, 20, 30, 1000, 1100], show how you'd compute p50, p95 and p99 approximately from the ordered list and interpret each result in terms of user impact.

EasyTechnical

29 practiced

Explain why averages can be misleading for latency-sensitive services and why tail percentiles (p95/p99/p999) matter. Provide concrete examples where optimizing average latency could degrade user experience and outline approaches focused specifically on reducing tail latency.

EasyTechnical

44 practiced

Describe how you would determine whether a production service is CPU-bound or I/O-bound. List the metrics, tools, and signals you would check (e.g., CPU utilization, iowait, disk/network metrics, syscall tracing, flamegraphs) and describe the actions you'd take for each diagnosis (e.g., optimize code path vs increase concurrency vs tune disk/network).

Sample Answer

First, clarify symptom and scope: which service, timeframe, and user-visible impact (latency, error rate, throughput). Then gather metrics and signals.

Metrics & tools to check- CPU: %util, steal, per-core load, run queue (top, mpstat, dstat, vmstat, Grafana). Long runnable queue → CPU-bound.- I/O: iowait, disk throughput/latency, ops/sec (iostat, sar, blktrace, Prometheus node_exporter). High latency and low iops → disk bottleneck.- Network: NIC utilization, retransmits, RTT, socket queues (ifstat, ss, tcptrace, iftop). Saturation or high retransmits → network-bound.- Application-level: request latency/throughput, p99/p95, error rates (APM: Jaeger, Datadog, NewRelic).- Profiling: flamegraphs, perf, eBPF (bcc/pyflame, BPFTrace) to see hot CPU stacks or syscall waits.- Syscall tracing: strace, perf trace, or bpf to spot blocking syscalls (fsync, read).- Containers/VMs: cgroup limits, CPU shares, throttling metrics (docker stats, cAdvisor).

Interpretation & actions- CPU-bound (high CPU%, long runqueue, flamegraph shows CPU hotspots): - Optimize hot code paths, algorithmic improvements, use more efficient libraries, enable JIT tuning. - Increase parallelism: scale out replicas or increase CPU resources. - Offload work: move heavy tasks to background jobs, use native extensions or vectorized ops.- I/O-bound (high iowait, high disk latency, blocking syscalls): - Reduce disk syncs, batch writes, tune fsync/flush, enable async I/O. - Use faster storage (NVMe, local SSD), increase IOPS, RAID, or caching (Redis, memcached). - Introduce queuing, backpressure, or rate-limit producers.- Network-bound (high NIC usage, packet loss, long RTTs): - Compress payloads, reduce chattiness, enable keepalives or HTTP/2, batch requests. - Increase bandwidth, tune TCP (window sizes), offload via CDN or regional replicas.- Hybrid or throttled (container/VM limits, noisy neighbor): - Adjust cgroup/VM resources, move to dedicated hosts, use QoS and resource requests/limits in k8s.- Verification: after changes, run load tests, compare latencies, resource metrics, and profile again. Document root cause and remediation in postmortem; add alerts for the key metrics discovered.

Latency Analysis & Optimization Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank