Technical Skills and Tools Questions

A concise but comprehensive presentation of a candidate's core technical competencies, tool familiarity, and practical proficiency. Topics to cover include programming languages and skill levels, frameworks and libraries, development tools and debuggers, relational and non relational databases, cloud platforms, containerization and orchestration, continuous integration and continuous deployment practices, business intelligence and analytics tools, data analysis libraries and machine learning toolkits, embedded systems and microcontroller experience, and any domain specific tooling. Candidates should communicate both breadth and depth: identify primary strengths, describe representative tasks they can perform independently, and call out areas of emerging competence. Provide brief concrete examples of projects or analyses where specific tools and technologies were applied and quantify outcomes or impact when possible, while avoiding long project storytelling. Prepare a two to three minute verbal summary that links skills and tools to concrete outcomes, and be ready for follow up probes about technical decisions, trade offs, and how tools were used to deliver results.

HardSystem Design

31 practiced

You're tasked with migrating a monolithic web application to microservices as SRE lead. Produce a migration plan focused on reliability: decomposition approach (strangler pattern), traffic routing (API gateway, sidecars), observability migration (traces, metrics, logs), deployment strategy (canary/dark launch), defining SLOs per service, and rollback/chaos testing. Also list gating criteria you would use to pause or revert migration if reliability degrades.

Sample Answer

Requirements & constraints:- Maintain >=99.95% availability during migration, no customer-visible regressions, support rollback within 15 minutes, keep error budget per product bounded.

High-level approach (Strangler Pattern):- Incrementally extract capabilities by user-visible verticals (e.g., auth, billing, search) starting with low-risk, well-tested domains.- Keep monolith as canonical until service passes reliability gates; route selected traffic to new service.

Traffic routing:- Front door: API Gateway (rate-limiting, auth, routing rules) + service mesh (Envoy) for intra-cluster service discovery, retries, and mTLS.- Use sidecars for observability, circuit-breaking, and per-call tracing metadata.- Start with path-based routing -> weighted routing for canaries -> full cutover.

Observability migration:- Traces: propagate distributed trace IDs from gateway; instrument new services with OpenTelemetry; ensure sampling parity.- Metrics: expose Prometheus metrics; define service-level and endpoint-level metrics (latency p50/p95/p99, success rate, throughput, resource usage).- Logs: structured JSON logs shipped to centralized ELK/Tempo/OTel backend; correlate logs with trace IDs.- Create dashboards showing monolith vs new service comparative KPIs.

Deployment strategy:- CI/CD builds container images, deploy via Kubernetes.- Canary + dark launch: mirror production requests to new service (no response) while canary serves small % of real traffic.- Progressive rollout: 1% -> 5% -> 20% -> 100% with automated health gates.

SLOs & error budgets:- Define SLOs per service: e.g., availability 99.9% (5m windows aggregated), latency p95 < 300ms, error rate <0.1%.- Allocate error budget and require remediation when spent >20% in rolling 28d window.

Rollback & chaos testing:- Automated rollback on gate failure: traffic weight shift, gateway routing revert, and quick image rollback.- Run chaos experiments (kill pods, network latency, DB failover) in staging, then in limited production during low-traffic windows using Chaos Monkey/Chaos Toolkit.- Run failure-injection in sidecars to validate graceful degradation.

Gating criteria to pause/revert:- SLO breaches: sustained error rate above threshold or >20% of error budget spent in 7 days.- Latency regressions: p95 increase >2x baseline for >10 minutes.- Increased customer-impacting incidents: number of SEV incidents doubles vs baseline in 24 hours.- Resource exhaustion: CPU/memory OOMs or increased GC pauses causing timeouts.- Data correctness issues: any loss/corruption detected.- Alert storms: >3x alert rate or >50% of alerts are new post-migration.

Operational practices:- Runbooks and playbooks for each service, automated rollback scripts, regular run drills, and postmortems with corrective actions.- Communication plan: stakeholder notifications at each rollout step, and an on-call escalation path.

Trade-offs:- Strangler minimizes blast radius but increases operational complexity and latency from network hops; mitigated via optimized sidecars and observability.

EasyTechnical

30 practiced

Outline the stages you would include in a CI/CD pipeline for a backend service, and name at least one tool for each stage (choose from Jenkins, GitHub Actions, GitLab CI, CircleCI). For each stage (build, unit tests, integration tests, security scans, image build, registry push, deploy, smoke tests) describe expected artifacts, how you handle secrets, and the gating policy for deployments (manual approval, automated canary promotion, metric-based gating).

Sample Answer

Build (tool: GitHub Actions)- Artifact: compiled binaries/jars, build metadata (commit, version), build logs- Secrets: use GitHub Actions Secrets or HashiCorp Vault; inject at runtime, never commit- Gating: build must succeed before tests run (automated)

Unit tests (tool: CircleCI)- Artifact: test reports (JUnit XML), coverage reports- Secrets: CI project env vars or Vault short-lived tokens- Gating: fail-on-test-failure; no deploy unless coverage threshold met

Integration tests (tool: GitLab CI)- Artifact: integration test reports, environment snapshots (DB dumps), logs- Secrets: dynamic test credentials from Vault/CI secrets, ephemeral test infra (k8s namespaces)- Gating: required green; flaky tests quarantined; blocking for staging deploy

Security scans (tool: Jenkins)- Artifact: SCA reports (SBOM), SAST findings, dependency vulnerability reports- Secrets: scanner service credentials in Jenkins credentials store or Vault- Gating: critical/high vulnerabilities block promotion; medium require ticket or exception

Image build (tool: GitHub Actions)- Artifact: OCI image with immutable tag (sha), SBOM layered into image- Secrets: Docker registry credentials from secrets manager, build args not persisted- Gating: image must pass image scanning and provenance checks

Registry push (tool: CircleCI)- Artifact: pushed image in registry with signed metadata (notary/cosign)- Secrets: registry creds via CI secrets or ephemeral tokens- Gating: push only after image build + scans succeed

Deploy (tool: GitLab CI)- Artifact: release manifest, deployment logs, k8s manifests/helm charts- Secrets: kubeconfigs from Vault, RBAC-limited tokens, use GitOps (Flux/Argo) with signed manifests- Gating: manual approval to deploy to prod OR automated canary pipeline to staging

Smoke tests (tool: Jenkins)- Artifact: smoke test results, endpoint responses, basic latency/health metrics- Secrets: read-only API keys from secrets manager- Gating: automated canary promotion: deploy small percentage, run smoke + metric checks (error rate, latency); metric-based gating promotes to 100% if SLOs met within window, otherwise automatic rollback and alerting

Overall gating policy recommendation:- Stages chained: build → unit → integration → security → image build → push → deploy → smoke- Enforce automated gates for tests/scans; use manual approval for high-risk production deploys or implement metric-based automated canary promotion (telemetry-driven: error rate, p95 latency, success rate). Secrets centralized in Vault/Secrets Manager with short-lived tokens; principle of least privilege; audit all secrets access.

EasyTechnical

26 practiced

For a typical HTTP microservice, list the core metrics you would instrument and monitor (examples: request_rate, error_rate, latency P50/P95/P99, CPU/memory usage, GC metrics, queue depth). For each metric, explain why it matters and how you would choose an alert threshold so alerts are actionable and not noisy.

Sample Answer

Core metrics I’d instrument for an HTTP microservice, why they matter, and how to set actionable alert thresholds:

1. Request rate (requests/sec)- Why: traffic volume, detect traffic spikes or drops.- Alerting: Alert on sudden % change (e.g., >50% drop or >200% spike in 5m) vs baseline to avoid noisy alerts from normal diurnal patterns. Use ML/seasonal baselining if traffic varies.

2. Error rate (5xx and important 4xx)- Why: indicates user-facing failures or bad clients.- Alerting: Alert when error rate > SLO threshold (e.g., >1% of requests over 5m) AND absolute error count > a minimum (e.g., >10 errors/5m) to avoid noise when traffic is low.

3. Latency: P50/P95/P99 (and mean)- Why: performance impact and tail latency affecting UX.- Alerting: Alert on sustained breaches of SLOs (e.g., P95 > 300ms for 10+ minutes) or P99 spikes. Combine with request rate to suppress when low traffic.

4. Saturation: CPU, memory, disk, network- Why: resource exhaustion causes outages.- Alerting: Alert on sustained high utilization (CPU >85% for 10m) or memory >90% + rising trend. Avoid brief spikes using duration windows.

5. GC metrics (pause time, frequency)- Why: GC pauses cause latency spikes.- Alerting: Alert if GC pause > threshold (e.g., >200ms) or increased GC frequency leading to latency regression.

6. Queue depth / backlog (worker queue, request queue)- Why: growing queues indicate downstream slowness or resource exhaustion.- Alerting: Alert when queue length > capacity threshold or growing rate > X/min for Y minutes.

7. Thread/pool utilization and connection counts- Why: thread exhaustion or connection leaks degrade service.- Alerting: High thread utilization (>80%) or connection count near max + growth trend.

8. Up/down health checks and deployment metrics- Why: availability and rollout health.- Alerting: Immediate alert on failed health checks on >N instances or failure percentage >threshold during deployment.

9. Dependency health (DB latency/errors, external APIs)- Why: failures often originate from dependencies.- Alerting: Alert when dependency error rate or latency crosses thresholds or correlation with service errors rises.

Guiding principles for thresholds:- Tie alerts to SLOs and error budgets where possible.- Use relative and absolute conditions (percentage + minimum count).- Require sustained violation (e.g., 5–10 minutes) to reduce flapping.- Suppress alerts during planned maintenance and deployments, or use deployment-aware rules.- Classify alerts: P1 (immediate paging), P2 (investigate next 15–30m), and use runbooks with each alert.

Instrument with Prometheus metrics + labels, aggregate in Grafana dashboards, and iterate thresholds after observing historical patterns and post-incident reviews.

HardTechnical

29 practiced

Your Prometheus environment is suddenly overwhelmed by a high-cardinality metric spike, causing CPU and memory exhaustion. What immediate steps do you take to stabilize the system (throttle ingestion, disable heavy scrapes, identify and delete offending series), and what long-term changes would you implement (metric naming conventions, label cardinality monitoring, aggregation, SDK guardrails, remote_write to long-term systems) to prevent recurrence?

Sample Answer

Immediate (stabilize within minutes)- Put Prometheus in “safe” mode: pause ingestion by stopping or throttling promtool/process (systemctl stop prometheus) or scale down targets in ServiceDiscovery. If using Kubernetes, cordon/scale down the Prometheus StatefulSet to free CPU.- Disable heavy scrapes quickly: edit scrape_configs (or use relabel_config to drop targets) and reload (curl -X POST http://PROM:9090/-/reload) or patch ServiceMonitor to 0 replicas. Temporarily increase scrape_interval and reduce scrape_timeout for noisy jobs.- Identify offending series: query head-series and cardinality metrics: - prometheus_tsdb_head_series, prometheus_tsdb_head_chunks - top series by count: use PromQL: topk(50, count by(__name__, job, instance)({__name__=~".+"})) or script using series API /api/v1/series?match[]=<pattern>&start=&end=- Delete problematic series (if safe): use TSDB admin-delete-series api (Prometheus >=2.11): - curl -X POST -g 'http://PROM:9090/api/v1/admin/tsdb/delete_series?match[]={job="noisy-job"}' - run curl -X POST http://PROM:9090/api/v1/admin/tsdb/clean_tombstones then restart Prometheus.- If under OOM risk, offload remote_write (Thanos/ Cortex) and spin up read-only replica to preserve data.

Short-term mitigations- Add relabel_rules to drop high-cardinality labels (e.g., request_id, session_id) before ingestion: - action: replace/labeldrop/labelmap examples in scrape_configs.- Apply recording rules to aggregate by meaningful dimensions and reduce raw series.

Long-term changes (prevent recurrence)- Metric design and SDK guardrails: - Enforce conventions: high-cardinality data must be logs/traces, not labels; use label cardinality limits per metric. - Provide libraries/linters (promtool, custom CI checks) that fail builds when new metrics exceed cardinality thresholds. - Educate devs: document naming (dot vs underscore), label semantics, and examples.- Cardinality monitoring & alerting: - Export and alert on prometheus_tsdb_head_series growth, per-job series count, and new-series-per-minute. - Integrate cardinality exporter (or custom job) and alert thresholds that trigger automated relabeling or rollbacks.- Aggregation & sampling: - Add recording rules at scrape/ingestion time (via rule files or remote processing) to compute aggregates (p50/p95, counts) and drop raw high-cardinality labels. - Use histograms/summaries appropriately and avoid label misuse.- Ingestion architecture: - Use remote_write to long-term stores (Thanos, Cortex) configured with downsampling/retention; keep Prometheus for recent raw metrics only. - Configure write_sharding, max_samples_per_send, queue_config to throttle bursts.- Automation & runbooks: - Automated playbook to throttle/disable scrapes, delete series safely, and restore. - CI gate that runs synthetic cardinality tests on new changes.- Governance: - Require design review for any metric that could add >N labels or new cardinality class. - Periodic audits (monthly) of top-K cardinality producers and cleanup.

Why this order:- Immediate steps reduce load fastest (stop scrapes, delete heavy series). Admin API + relabeling are safe short-term fixes. Longer term focuses on prevention via design, tooling, and architecture (aggregation, remote_write, SDK guardrails), which stop recurrence without operational firefighting.

HardSystem Design

27 practiced

You are responsible for migrating a large monorepo build system to Bazel to gain reproducible builds and caching. Produce a migration plan: create BUILD files for a small subset, define hermetic toolchains and external dependency handling, integrate remote caching and CI, update developer workflows (local vs remote builds), and outline rollback and success metrics if adoption fails. Address handling language-specific toolchains (Java/Go/Python).

Sample Answer

Requirements & constraints- Goal: reproducible hermetic builds, fast incremental builds via caching, CI integration, minimal dev friction, rollback safe.- Non-functional: support Java/Go/Python toolchains, remote cache (gRPC), CI agents, auth, SLOs for build latency/availability.

High-level migration plan (phases)1. Pilot (2–4 weeks): pick 1–3 representative services (Java backend, Go CLI, Python lib). Create BUILD files, hermetic toolchains, CI job, remote cache proof-of-concept.2. Incremental rollout (8–12 weeks): onboard teams by repo slice, provide docs & training.3. Scale & optimize (ongoing): tune remote cache, CI concurrency, monitoring.4. Maturity & rollback plan establishment.

Sample minimal BUILD files (monorepo subset)

python

# java/service/BUILD
java_library(
  name = "service_lib",
  srcs = glob(["src/main/java/**/*.java"]),
  deps = [
    "//third_party:guava",
  ],
)

java_binary(
  name = "service_bin",
  main_class = "com.example.Main",
  runtime_deps = [":service_lib"],
)

python

# go/tool/BUILD
go_library(
  name = "tool_lib",
  srcs = ["*.go"],
  importpath = "company.com/tool",
  visibility = ["//visibility:public"],
)
go_binary(
  name = "tool",
  embed = [":tool_lib"],
)

python

# python/lib/BUILD
py_library(
  name = "py_lib",
  srcs = glob(["**/*.py"]),
  deps = ["@pypi__requests//:pkg"],
)

Hermetic toolchains & external deps- Use Bazel toolchain rules: - Java: use rules_jvm_external + pinned Maven artifacts via lockfile; use a hermetic JDK Docker image for CI and preinstalled JDK via toolchain rule for local dev. - Go: use rules_go with module_mode = "vendor" or use `GOMODCACHE` pinned modules; provide a Bazel toolchain that bundles a specific go toolchain binary (containerized). - Python: use rules_python with PEX/wheels built into the Bazel cache; pin versions in requirements.txt and maintain a wheel cache in CI and remote cache.- External deps: mirror third-party artifacts into an internal artifact repository (Nexus/Artifactory) and use rules to fetch only from internal mirror. Generate lockfiles and commit to repo.

Remote caching & CI integration- Remote cache: set up Bazel Remote Cache (e.g., Buildbarn, Bazel Remote Cache, or Google Remote Build Execution) behind gRPC with strong auth (mTLS + token).- CI: create a canonical Bazel CI pipeline: - Step A: bazel test //... with remote cache read/write. - Step B: on release, run remote build execution to produce binaries/artifacts.- Implement cache warming and metrics export (Prometheus): cache hit/miss, bytes transferred, request latency.- Authentication & federation: short-lived tokens minted by CI (OIDC) to access cache; use VPC peering or VPN for on-prem agents.

Developer workflows- Local fast feedback: developers run `bazel build //... --config=local` which uses local sandboxed toolchains; enable remote cache read-only to speed up builds. Provide wrapper scripts: `./bazelw build ...` to ensure correct bazel version.- Remote CI builds: CI runs `bazel build/test --config=ci --remote_cache=... --remote_executor=...` to write outputs.- For iterative dev, enable `--experimental_action_listener` for fast iteration or use `ibazel` for incremental rebuilds.- Local hermeticity: ship hermetic dev images (Docker or devcontainers) with same toolchain binaries to eliminate "works-on-my-machine".

Rollback & failure handling- Feature-flag migration per-team: default to old build system; switch small teams first.- Rollback steps: - Disable remote cache writes and switch CI back to legacy builders. - Revert BUILD changes via PRs; maintain both build systems in parallel for up to X weeks.- Safety: CI gate for production builds until stable; require signature for production-promotions.

Success metrics & SLOs- Adoption: % of services migrated / % of developers using Bazel wrappers.- Cache efficiency: hit ratio >= 70% (target adjustable), median build time reduction >= 30% for incremental builds.- Reliability: remote cache availability >= 99.5%; CI build median latency within SLA.- Developer experience: time-to-first-green <= baseline, number of support tickets < threshold.- Quality: no increase in flaky test rate; reproducible build verification pass rate.

Monitoring & alerting- Expose: cache hit/miss, latency, auth failures, storage usage, CI job durations.- Alerts: cache service down, hit ratio drop >20% sustained, build latency regression.

Trade-offs & risks- Upfront cost: authoring BUILD files and pinning deps is work-heavy; mitigate by automating BUILD generation (Gazelle for Go, buildifier tools).- Toolchain drift: solve with pinned container images and CI-enforced lockfiles.- Remote cache security: ensure encryption and least privilege.

Operational runbook (short)- On cache outage: set Bazel env to local-only read/writes; scale workers; notify teams; failover to read-only cache.- On bad hermetic toolchain release: pin previous toolchain, block rollout via CI gates.

This plan balances a safe incremental migration, hermetic reproducibility across Java/Go/Python, measurable success metrics, and clear rollback paths while emphasizing monitoring and developer ergonomics.

Unlock Full Question Bank

Get access to hundreds of Technical Skills and Tools interview questions and detailed answers.

Join thousands of developers preparing for their dream job.