Systems Engineering Coding and Problem Solving Questions

Practical coding and algorithmic skills applied to systems and infrastructure tasks. Candidates should demonstrate the ability to write syntactically correct and maintainable code or scripts to automate operations, parse logs, collect metrics, implement health checks, and perform diagnostics; choose appropriate simple data structures and algorithms; reason about time and space complexity at a practical level; apply defensive programming and error handling; debug effectively and write tests to validate behavior; and solve timed technical problems by prioritizing core functionality, correctness, and maintainability. Interviewers commonly use small coding exercises, scripting tasks, or live problems that emphasize operational automation and system oriented problem solving.

HardTechnical

65 practiced

Given logs from multiple machines with timestamps and a known maximum clock skew S seconds, describe an algorithm to reconstruct a partial ordering of events that is consistent with causal ordering as much as possible. Explain uncertainty windows, how to present ambiguous order to users, and complexity of merging streams in real-time.

Sample Answer

**Clarify problem & assumptions**- Each machine produces monotonic timestamps with unknown offset ≤ S seconds (clock skew). Network delays ignored in logs; goal: produce a partial order consistent with possible real-time causality given skew bound.

**Algorithm (high-level)**1. Treat each log entry as interval [t - S, t + S] representing possible true time.2. Merge streams by sorting events by interval start (t - S). When intervals are disjoint (end < next start) their order is certain.3. For overlapping intervals, preserve partial order: create a DAG where an edge A→B exists only if A.end < B.start (A must be before B). If intervals overlap, no ordering edge is added (ambiguous).4. Optionally, refine using intra-host ordering (local logs are totally ordered) and known happens-before relations (e.g., RPC IDs, causal markers) to add edges.

**Uncertainty windows**- Define uncertainty window = [t - S, t + S]. Use these to compute overlap; overlap implies temporal ambiguity.- Visualize windows on timeline; compute connected components of overlap graph to identify ambiguous groups.

**Presenting ambiguity to users**- Show deterministic edges in one color; ambiguous groups as collapsible bands with per-event earliest/latest timestamps.- Provide confidence score: fraction of possible total orders consistent with a hypothesis (approx via sampling).- Allow filters: show only deterministic ordering or expand ambiguous windows.

**Real-time merging complexity**- Use streaming merge with a min-heap keyed by interval start. Maintain a sliding frontier = min of (processed intervals' max end). Emit events whose interval end < frontier to guarantee order.- Per event: O(log k) enqueue/dequeue where k = number of streams. Memory bounded by events with overlapping windows (expected O(rate * S)).- Worst-case memory/time depends on burstiness and S; with large S or correlated clocks many events buffer—mitigate via heuristics (truncate uncertainty, user alerts).

**Trade-offs**- Larger S increases ambiguity and buffering. Incorporating application-level causality reduces ambiguity but requires instrumentation.

MediumTechnical

60 practiced

A deployment intermittently fails because two services race to claim the same resource during startup. Describe a methodical debugging and mitigation plan: how to reproduce, what instrumentation to add, short-term mitigations for production, and longer-term fixes to prevent the race, including protocol or locking changes.

Sample Answer

**Situation & goal**We have intermittent startup failures because Service A and Service B race to claim Resource X. Goal: reliably reproduce, collect evidence, mitigate in prod quickly, then eliminate race.

**How to reproduce**- Create an automated start-orchestration that launches A and B simultaneously (kubernetes Job with parallelism=2 or scripted systemd restarts).- Add randomized delays and high concurrency (repeat 1000x) to surface timing windows.- Run in staging with identical config and load.

**Instrumentation to add**- Correlate events with request IDs and precise timestamps (ns): log attempt/claim/release with PID, host, container id.- Emit metrics (Prometheus) for claim_attempts, claim_success, claim_conflict.- Trace via distributed tracing (OpenTelemetry) for startup path.- Add audit entries to the resource manager (who, when, token).

**Short-term mitigations (production)**- Introduce exponential backoff + jitter on claim retries.- Failover to a safe-read-only mode until claim succeeds.- Use a conservative startup ordering via init containers or systemd dependencies to ensure one service boots before the other.- Add a temporary leader-election using existing datastore (e.g., create-if-not-exists key in Redis with TTL).

**Longer-term fixes**- Implement atomic claim via a single source of truth: use a transactional datastore (etcd/consul/DB) with compare-and-swap (CAS) or distributed lock primitives.- Design idempotent claim protocol: include lease with TTL and clear renewal semantics.- Add explicit leader-election (Raft/lease) or a coordinator service to serialize claims.- Define clear failure/recovery protocols and document startup ordering.- Add integration tests that run concurrent starts as part of CI.

**Trade-offs**- Short-term ordering reduces randomness but hides concurrency problems; prefer durable locking for correctness.- Prefer simple CAS with TTL for low-latency control; use full consensus only if resource ownership must survive partitioning.

EasyTechnical

60 practiced

Explain idempotency in the context of systems engineering and operations. Provide at least three concrete examples where idempotency is critical (for example: configuration management, deployment scripts, API calls). Describe strategies to design idempotent operations and pitfalls to avoid.

HardSystem Design

70 practiced

Design a scalable concurrent health-checker service that runs health probes across thousands of hosts, respects a global rate limit of R checks/sec, supports exponential backoff for flapping hosts, and can be horizontally sharded across workers. Sketch the architecture, data flow, failure modes, and a worker-level implementation strategy (language of choice).

Sample Answer

**Overview & goals**Design a horizontally-shardable health-checker that schedules probes across thousands of hosts, enforces global R checks/sec, supports exponential backoff for flappers, and tolerates failures.

**Requirements / constraints**- Global rate limit R (soft/hard)- Per-host exponential backoff on failures- Sharding across N workers (workers may change)- Idempotent probes, observable results

**High-level architecture**- Coordinator/Control Plane: stores host list, health policy, global token-bucket in Redis (or etcd). Exposes assignment map.- Workers: pull shard assignment, acquire rate tokens from global rate limiter, run probes, update state.- Storage: Redis for fast state (last result, backoff exponent, next allowed time); long-term metrics to TSDB (Prometheus/Influx).- Message bus (Kafka) optional for events.

**Data flow**1. Control plane assigns host ranges (consistent hashing) to workers.2. Worker iterates eligible hosts (next_allowed <= now), requests tokens from global limiter (Redis Lua script implementing token bucket), runs probe, writes result and updates backoff in Redis, emits metrics.

**Backoff logic**- On failure: exponent = min(exponent+1, max); next_allowed = now + base * 2^exponent + jitter.- On success: reset exponent=0.

**Rate limiting**- Global token bucket in Redis with remaining tokens = R per second; workers atomically decrement via Lua script returning success/fail.

**Shard rebalancing**- Use consistent hashing + ephemeral worker IDs (leases). When worker dies, its slots reassign quickly.

**Failure modes & mitigations**- Worker crash: lease expiry triggers reassignment.- Redis outage: degrade to local token estimation and emergency low-rate mode; buffer events to disk.- Slow probes: per-worker concurrency cap and probe timeouts to avoid blocking tokens.- Clock skew: use monotonic timers and server time from Redis for next_allowed comparisons.

**Worker-level implementation (Go sketch)**- Use goroutines + bounded worker pool, context for cancelation, net/http with timeouts.- Use Redis Lua scripts for atomic token acquisition and state update.

Example token-acquire Lua (conceptual):

text

-- KEYS[1]=bucket_key, ARGV[1]=now, ARGV[2]=rate, ARGV[3]=burst, ARGV[4]=tokens_needed
-- returns 1 if allowed else 0

Go pseudocode:

// worker loop
for host := range eligibleHosts {
  if !AcquireToken(hostTokensNeeded) { sleep(short) ; continue }
  go func(h Host){ ctx, cancel := context.WithTimeout(...); res := Probe(ctx,h); UpdateState(h,res) }(host)
}

**Trade-offs**- Redis token bucket centralizes rate control (strong consistency) but is a single dependency; Kafka+distributed token allocation reduces coupling.- Per-host backoff in Redis ensures global consistency; local caching improves latency.

This design provides scalable sharding, strict global rate control, robust backoff, and clear failure handling appropriate for production cloud infrastructure.

EasyTechnical

66 practiced

Write a Python script that reads a potentially large log file line-by-line (streaming, do not load entire file into memory) and counts occurrences of distinct error codes that match the pattern ERR followed by digits (example: ERR1234). The script should handle missing or unreadable files gracefully, accept the file path as an argument, and print the top 5 error codes and their counts in descending order.

Sample Answer

**Approach**- Stream the file line-by-line to avoid high memory use, use regex to extract codes, count with collections.Counter, handle file errors, and print top 5.

**Code**

python

#!/usr/bin/env python3
import sys, re
from collections import Counter

ERR_RE = re.compile(r'\b(ERR\d+)\b')

def top_errors(path, top_n=5):
    counts = Counter()
    try:
        with open(path, 'r', encoding='utf-8', errors='replace') as f:
            for line in f:
                for m in ERR_RE.findall(line):
                    counts[m] += 1
    except FileNotFoundError:
        print(f"Error: file not found: {path}", file=sys.stderr)
        return
    except PermissionError:
        print(f"Error: permission denied: {path}", file=sys.stderr)
        return
    except OSError as e:
        print(f"Error reading file {path}: {e}", file=sys.stderr)
        return

    for code, cnt in counts.most_common(top_n):
        print(f"{code}: {cnt}")

if __name__ == '__main__':
    if len(sys.argv) != 2:
        print("Usage: script.py /path/to/logfile", file=sys.stderr); sys.exit(1)
    top_errors(sys.argv[1])

**Explanation & Complexity**- Streams file (O(1) memory besides counts). Time O(N) where N = lines processed. Counter size equals distinct error codes.- Edge cases: binary/unicode lines handled via errors='replace'; missing/unreadable file messages; codes spanning tokens are avoided by word boundaries.

**Systems-engineer notes**- For very high cardinality logs, consider sharding counts (e.g., by time window) or using external store (Redis, BigQuery).

Unlock Full Question Bank

Get access to hundreds of Systems Engineering Coding and Problem Solving interview questions and detailed answers.

Join thousands of developers preparing for their dream job.