Hands On Projects and Problem Solving Questions

Discussion of practical projects and side work you have built or contributed to across domains. Candidates should be prepared to explain their role, architecture and design decisions, services and libraries chosen, alternatives considered, trade offs made, challenges encountered, debugging and troubleshooting approaches, performance optimization, testing strategies, and lessons learned. This includes independent side projects, security labs and capture the flag practice, bug bounty work, coursework projects, and other hands on exercises. Interviewers may probe for how you identified requirements, prioritized tasks, collaborated with others, measured impact, and what you would do differently in hindsight.

MediumSystem Design

63 practiced

Design logging and distributed tracing for a microservices architecture to enable fast root-cause analysis. Include how to propagate correlation IDs, use structured logs, set trace sampling and retention policies, and how you would integrate logs, traces, and metrics for investigation dashboards.

Sample Answer

Requirements:- Fast root-cause: trace a user request end-to-end, jump from trace span to relevant logs/metrics.- Low overhead: minimal latency/size impact.- Security/compliance: retention and PII controls.

High-level architecture:API Gateway → Service A → Service B → DBInstrumentation: OpenTelemetry for tracing + logs + metrics; Jaeger/Tempo for traces; Elasticsearch (or Loki) + Kibana for logs; Prometheus + Grafana for metrics. Centralized correlator links them.

Key components & practices:1. Correlation IDs- Generate a single trace_id at ingress (API Gateway) and propagate via standardized headers (W3C Traceparent or x-request-id).- Each service extracts/injects trace_id into outgoing requests and into local logs/metrics. Use middleware to enforce propagation.- Also include span_id for fine-grained correlation where possible.

2. Structured logs- JSON logs with fields: timestamp, service, env, level, trace_id, span_id, request_id, user_id (if allowed), error_code, message, context.- Log library (e.g., logback, structlog) enriches logs automatically from context.- Avoid logging PII; use tokenization or hashing for sensitive fields.

3. Tracing: sampling & retention- Default adaptive sampling: high sample rate (100%) at ingress for short windows on errors; production baseline 1-5% for traces, 100% for errors/alerts via tail-based sampling.- Keep full traces for errors for longer retention (e.g., 30 days); sampled traces for 7 days.- Store traces in Jaeger/Tempo with searchable index for retained traces.

4. Metrics- Emit high-cardinality tags sparingly. Metrics for latency, error rates, saturation per service and endpoint.- Use Prometheus for short-term high-resolution, long-term aggregated metrics in remote storage.

5. Integration & dashboards- Link logs/traces/metrics by trace_id: Kibana dashboard fields allow clicking trace_id to open Jaeger trace; Jaeger UI links back to logs via service+timestamp+span_id query to Elasticsearch.- Alerting: Grafana alerts on metrics trigger runbooks that include links to pre-filtered logs and trace queries.- Investigation dashboard pattern: top panel shows service-level latency/error trends, middle panel lists recent error traces (clickable), bottom panel shows filtered logs for selected trace/span and relevant metrics for that time window.

Scalability & trade-offs:- Tail-based sampling reduces storage but adds latency at collector; balance using local probabilistic sampling + centralized tail-sampling for errors.- Indexing logs heavily increases cost; use hot/cold indices and rollups; keep structured fields minimal.- Ensure collectors are horizontally scalable; use batching and backpressure.

Operational notes:- Instrument critical paths first; add observability in CI gating.- Run blameless postmortems to refine sampling/retention.- Periodically review retention for cost vs. forensic value and rotate keys for PII controls.

This design enables fast root-cause analysis by guaranteeing an unbroken trace_id chain, structured searchable logs, targeted trace retention for errors, and dashboards that let engineers pivot between metrics, traces, and logs quickly.

EasyTechnical

111 practiced

Explain step-by-step how you set up a reproducible local development environment for a full-stack project. Include specific tools (IDEs, package managers, containers, dev databases), setup scripts or Docker Compose files, environment variable and secret handling, and how you ensured new contributors could get started in one command.

Sample Answer

I set up a reproducible local dev environment so any new contributor can run the app with one command. Steps I took:

1) Choose tools and conventions- IDE: VS Code with recommended extensions (settings in .vscode/)- Package managers: npm/yarn for frontend, pip/poetry for Python backend or Maven/Gradle for Java- Containers: Docker + Docker Compose to ensure consistent runtimes- Dev DB: Postgres container, Redis container for caching- Local dev-only services: mailcatcher, MinIO for S3 emulation

2) Repository layout & docs- Add README with “Quick start”- Add .env.example, .devcontainer/ for VS Code Codespaces/Dev Containers- Add Makefile and scripts/bin/start-dev for one-command start

3) Docker Compose (example)

yaml

version: '3.8'
services:
  db:
    image: postgres:15
    environment:
      POSTGRES_USER: dev
      POSTGRES_PASSWORD: dev
      POSTGRES_DB: app_dev
    volumes:
      - db_data:/var/lib/postgresql/data
  backend:
    build: ./backend
    command: ./scripts/run_dev.sh
    env_file: .env
    ports: ["8000:8000"]
    depends_on: ["db"]
  frontend:
    build: ./frontend
    ports: ["3000:3000"]
    env_file: .env
    depends_on: ["backend"]
volumes:
  db_data:

4) Environment variables & secrets- Keep secrets out of VCS: add .env to .gitignore and provide .env.example with keys and descriptions- For sensitive setup in CI/dev, use a secrets manager (Vault, GitHub Actions secrets)- In local, document how to copy .env.example to .env and generate dev keys (scripts/bin/generate-secrets.sh)

5) Setup scripts & one-command start- Provide scripts/setup.sh to: - Check Docker installed - Copy .env.example -> .env - Create Python virtualenv or install node deps inside containers - Run DB migrations and seed minimal dev data- One-command start: make start or ./scripts/start-dev.sh which runs: - docker-compose up --build -d - docker-compose exec backend ./scripts/wait_for_db.sh && docker-compose exec backend ./scripts/run_migrations.sh - prints app URLs and dev credentials

6) Developer experience polish- Add VS Code launch configs for debugging (attach to container)- Health-check endpoints and make logs accessible (docker-compose logs -f)- Automated smoke test script to verify services up- In README: expected ports, credentials, troubleshooting

Why this works:- Docker Compose pins images and volumes so environment is identical across machines.- .env.example + setup scripts prevent secret leakage and make configuration explicit.- One-command script automates repetitive steps (build, migrate, seed), lowering onboarding friction so contributors can start coding quickly.

HardSystem Design

54 practiced

You manage a large monorepo used by multiple teams. Propose a CI/CD architecture to reduce redundant builds and tests, support independent service deployment, and manage shared libraries. Include caching, detecting impacted components, dependency-graph-based builds, and permission boundaries.

Sample Answer

Requirements:- Avoid redundant builds/tests across teams in a large monorepo- Allow independent service deployment- Efficient shared-library management- Support caching, impacted-change detection, dependency-graph builds- Enforce permission boundaries per team/service

High-level architecture:- A central CI orchestration layer (controller) + distributed build executors (k8s or cloud agents) + artifact/cache store + metadata service with dependency graph + auth/permission service + deployment pipelines per service.

Core components:1. Dependency Graph Service- Maintains up-to-date graph from workspace manifests (package.json, BUILD, pom.xml).- Exposes APIs: impacted_nodes(changed_files), build_order(nodes).

2. Change Detector- On PR/commit, maps changed files → affected packages/services via graph.- Produces a minimal job set: unit tests for changed packages, integration tests for impacted services.

3. Build Orchestrator- Creates DAG of tasks (lint, unit test, build, package, integration test).- Uses remote cache/registry and memoization to skip tasks whose inputs match cached outputs (content-hash keyed).

4. Remote Build Cache & Artifact Store- Content-addressable storage (CAS) for compiled artifacts, test results, Docker layers.- Store signatures: (repo-hash, toolchain-version, env) → artifact.- Support layer caching for container builds (buildkit registry).

5. Executors- Ephemeral workers in k8s with pre-warmed toolchain images; fetch cached artifacts to avoid rebuild.

6. Deployment Pipelines & Permissions- Per-service CD pipelines that consume artifacts; deployment gated by service owners.- RBAC enforced in orchestrator and artifact registry: teams can read shared libs/artifacts but write/publish only to allowed scopes.- Signed artifacts and provenance to ensure only authorized builds promote to prod.

Caching & correctness:- Use hermetic builds: declare inputs (source files, deps, toolchain). Hash inputs to look up cache.- Cache invalidation when toolchain or build scripts change.- Test result caching only valid if deterministic; always re-run integration/e2e unless covered by recorded golden manifests.

Scalability & performance:- Partition graph and cache namespace by shard; run impacted builds in parallel per DAG.- Use incremental graph updates on commit to avoid full repo scans.- Cache warmers for frequent branches.

Trade-offs:- Upfront cost to maintain accurate dependency graph and hermetic builds.- Stronger developer ergonomics if build/test commands are standardized.- Conservative approach: prefer correctness (rebuild when in doubt) over subtle cache bugs.

Example flow (PR):1. Change Detector computes impacted nodes → Orchestrator creates DAG.2. Executors fetch artifacts from CAS; skip cached steps.3. Run unit tests for changed modules; run integration tests for services impacted.4. On successful build and owner approvals, publish signed artifact and trigger service CD with RBAC.

This design minimizes redundant work, enables independent deployments, manages shared libraries centrally with access control, and scales through caching and graph-driven parallelism.

HardTechnical

69 practiced

A Kubernetes service is being OOMKilled intermittently in production. Provide a deep debugging and remediation plan: collecting memory profiles or heap dumps, reproducing the issue locally with load or synthetic traffic, analyzing pprof or flamegraphs, short-term mitigations, long-term fixes, and methods to verify the leak is resolved.

Sample Answer

Approach summary: treat this as a reproducible memory leak investigation with three parallel tracks — data collection in prod (non-invasive), controlled reproduction, and mitigation + fixes. Iterate: collect evidence → reproduce → root-cause → fix → verify.

1) Collect in-production evidence (non-disruptive)- Capture OOM/Kube events: kubectl describe pod, kubectl get events, check Kubelet logs.- Gather memory metrics from Prometheus/Grafana (container RSS, JVM/Go heap, GC pause/allocs).- Enable coredumps or heap dumps on OOM: for JVM set -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps; for Go enable GODEBUG=madvdontneed=1 and pprof endpoint.- Collect process-level snapshots: kubectl exec into pod and run top, pmap (linux), /proc/<pid>/smaps, or gcore (if allowed).- Save last N logs and stack traces around OOM.

Example commands:

bash

kubectl describe pod mysvc-abc
kubectl logs mysvc-abc --previous
kubectl exec -it mysvc-abc -- top -b -n1
kubectl exec -it mysvc-abc -- cat /proc/1/smaps > /tmp/smaps

2) Reproduce locally/in-staging- Create a scaled-down environment with identical JVM/Go/build flags, same container image and config (limits/requests, env vars).- Use traffic replay or synthetic load (wrk, vegeta, JMeter) emulating production patterns (payload sizes, concurrent users, long-lived connections).- Gradually ramp load while monitoring memory, GC, and pprof/heap dump endpoints.

3) Collect detailed profiles- For Go: enable net/http/pprof and capture heap/alloc profiles during increasing load:

bash

curl http://localhost:6060/debug/pprof/heap > heap.prof
go tool pprof -http=:8080 /path/to/binary heap.prof

- For JVM: capture heap dump (.hprof) and use jmap/jstack, jcmd GC.heap_info, and analyze with Eclipse MAT or VisualVM.- Produce CPU and memory flamegraphs (Flamegraph tool + pprof or async-profiler for JVM).

4) Analyze- Look for growing retained sizes, leaking goroutines/threads, pinned buffers, caches without eviction, native allocations (direct ByteBuffers, C libraries).- Identify allocation sites dominating growth (pprof top, allocation stacks, MAT dominator tree).- Correlate with application logic: singletons, global caches, channel/goroutine leaks, connection pools, unbounded queues.

5) Short-term mitigations- Add resource limits/requests if missing; set reasonable memory requests < limits to let scheduler work.- Configure OOMScoreAdj or JVM flags to prefer killing less critical components.- Restart strategy: increase liveness probe grace to avoid churn, or use controlled rolling restarts to avoid thrashing.- Apply temporary caps on cache sizes, reduce concurrency, or throttle traffic at ingress (Istio/Ingress rate limiting).

6) Long-term fixes- Fix root cause: free unused references, implement eviction policies, bound caches/queues, close connections, cancel contexts, fix goroutine leaks.- Add unit/integration tests reproducing leak pattern (load tests, long-run soak tests).- Improve observability: expose pprof/JMX endpoints, add custom mem metrics, alert on sustained memory growth (e.g., > 20% increase over 10m).

7) Verification- Run long-duration soak tests with same traffic profile; capture periodic heap/pprof snapshots and assert no monotonic growth in retained heap or number of goroutines/threads.- Deploy canary with increased monitoring and automated rollback if memory trend exceeds threshold.- Post-deploy: monitor for several retention windows (GC cycles, full heap cycles) before marking resolved.

Key trade-offs and notes:- Heap dumps can be heavy — schedule off-peak and store securely.- Avoid intrusive profiling in high-traffic prod unless you can tolerate overhead.- Use feature flags to roll out code fixes gradually.

This plan provides an evidence-driven path from symptom to fix and verification, balancing immediate reliability and long-term robustness.

EasyTechnical

55 practiced

Describe the Git branching and merge strategy you used in a team project (for example Git Flow or trunk-based). Explain how feature branches were created and merged, how code reviews were enforced, whether you rebased or merged, and how branch protections and CI checks were configured.

Unlock Full Question Bank

Get access to hundreds of Hands On Projects and Problem Solving interview questions and detailed answers.

Join thousands of developers preparing for their dream job.