Assess a candidate's practical and conceptual understanding of technology stacks, including major programming languages, application frameworks, databases, infrastructure, and supporting tools. Candidates should be able to explain common use cases and trade offs for languages such as Python, Java, Go, Rust, C plus plus, and JavaScript, including differences between compiled and interpreted languages, static and dynamic type systems, and performance characteristics. They should discuss application frameworks and libraries for frontend and backend development, common web stacks, service architectures such as monoliths and microservices, and application programming interfaces. Evaluate understanding of data storage options and trade offs between relational and non relational databases and the role of structured query language. Candidates should be familiar with cloud platforms such as Amazon Web Services, Google Cloud Platform, and Microsoft Azure, infrastructure components including containerization and orchestration tools such as Docker and Kubernetes, and development workflows including version control, continuous integration and continuous delivery pipelines, testing frameworks, automation, and infrastructure as code. Assess operational concerns such as logging, monitoring and observability, deployment strategies, scalability, reliability, fault tolerance, security considerations, and common failure modes and mitigations. Interviewers may probe both awareness of specific tools and the candidate's depth of hands on experience, ability to justify technology choices by evaluating trade offs, constraints, and risk, and willingness and ability to learn and evaluate new technologies rather than claiming mastery of everything.
HardSystem Design
87 practiced
Design a centralized secrets management solution for applications running across AWS and Kubernetes. Compare using HashiCorp Vault vs cloud-native KMS solutions. Describe secret rotation, fine-grained access controls, secure injection into containers, audit logging, and disaster recovery for secrets.
Sample Answer
Requirements and constraints:- Centralized secrets for apps across AWS (EC2, Lambda) and Kubernetes clusters (EKS / on-prem k8s).- Strong access control, automatic rotation, secure injection into containers, auditability, low-latency reads, and disaster recovery (RTO/RPO targets).- Multi-region/high-availability, minimal developer friction.High-level architecture:- Central secrets service (either HashiCorp Vault clustered or cloud-native KMS+secret store) fronted by an auth/sidecar layer for k8s and SDK adapters for AWS services.- Auth backends: AWS IAM, OIDC (k8s ServiceAccount), and LDAP/SAML for humans.- Storage: encrypted storage backend (consul or HA storage for Vault; for KMS approach, secrets stored in AWS Secrets Manager/Parameter Store with KMS CMKs).- Audit pipeline to centralized SIEM (CloudWatch/CloudTrail + Vault audit/device logs).- DR: cross-region replication of encrypted secrets and offline export capability.Comparison: HashiCorp Vault vs Cloud-native KMS/SMS- Vault strengths: advanced dynamic secrets (DB credentials, AWS STS, PKI), policy language (HCL), secrets leasing and built-in rotation, multi-cloud parity, secret versioning, namespaces for multi-tenant isolation.- Cloud-native (AWS KMS + Secrets Manager/SSM): tight AWS integration, managed HA, simpler ops, lower maintenance, IAM fine-grain RBAC, lower latency inside AWS. Weaknesses: feature parity across clouds limited; cross-cloud complexity increases.- Recommendation: Use Vault if you require dynamic secrets, multi-cloud uniformity, or complex transformation. Use cloud-native for AWS-first environments to reduce ops.Secret rotation- Dynamic secrets where possible (Vault DB/DBCredential, AWS RDS rotation). Lease-based model: clients request credential, Vault issues short-lived creds; rotation automatic when lease expires.- For static secrets, enforce scheduled automatic rotation via orchestration (Vault’s rotation or Secrets Manager’s rotation lambda) and CI/CD hooks to update deployments.Fine-grained access control- Vault: use policies (ACLs), namespaces, and identity-based auth (OIDC, AWS IAM). Bind k8s ServiceAccount to Vault role via Kubernetes auth.- AWS: IAM policies, resource-based policies, and Secrets Manager resource policies; use IAM Roles for Service Accounts (IRSA) for k8s.- Principle of least privilege, RBAC for humans, automated provisioning for service identities.Secure injection into containers- Sidecar or CSI driver: for Vault, use Vault Agent Injector or CSI Secrets Store + provider for Vault to mount or inject secrets as files or environment variables with short TTLs. For AWS, use Secrets Store CSI provider with provider for AWS or native retrieval using IRSA in-app SDK.- Avoid plaintext env vars when possible; prefer in-memory mounts, tmpfs, or file mounts with strict file permissions. Rotate file tokens frequently and use mTLS between sidecar and secrets server.Audit logging- Enable and centralize audit logs: Vault audit devices to syslog/JSON to SIEM. On AWS, enable CloudTrail and Secrets Manager/SSM event logs. Send logs to centralized analytics (ELK, Splunk, Sumo).- Log sensitive events (read, create, revoke) but redact secret values. Monitor anomalies (unusual read rates, failed auths) and alert.Disaster recovery- Regular encrypted backups of secret storage with versioning; store copies in another region (or different cloud). For Vault: use raft-storage auto-replication (DR replication) or snapshot backups of backend.- Test recovery runbooks: restore into new cluster, re-establish auth methods, rotate master encryption keys if necessary.- For cloud-native: enable cross-region replication for Secrets Manager or replicate KMS keys with multi-region keys. Ensure IAM role recovery paths.Trade-offs and final guidance- Vault offers richer features for multi-cloud and dynamic secrets but increases operational burden. AWS native lowers ops but locks you to AWS features.- For an engineering org with multi-cloud needs or advanced secret use-cases choose Vault. For AWS-centric teams prioritizing managed services and low ops, use AWS KMS + Secrets Manager with IRSA and Secrets Store CSI.- In all cases enforce automated rotation, least privilege, sidecar/CSI injection, centralized audit, and run DR drills.
HardSystem Design
83 practiced
Architect a cross-region CI/CD platform that supports 50 microservices with progressive delivery (canary and blue/green), automated rollbacks, an artifact registry, and audit/compliance controls (approval gates, audit logs). Describe components, deployment workflow, how regional failover should work, and how to coordinate database migrations during releases.
Sample Answer
Requirements & constraints:- 50 microservices, multi-region (active-active), progressive delivery (canary + blue/green), automated rollback on bad metrics, artifact registry, approval gates & audit logs, DB migration coordination, compliance controls.High-level architecture:- CI: centralized pipelines (GitHub Actions/Jenkins) per team, build artifacts (immutable images) and publish to Artifact Registry (ECR/ArtifactRegistry/Artifactory).- CD control plane: Global orchestration (Spinnaker or ArgoCD + Argo Rollouts) that manages per-region delivery agents (regional Argo instances) to keep API and policy consistent while deployments run locally per region.- Service mesh: Istio/Linkerd for traffic shifting, telemetry, mTLS.- Progressive delivery: Argo Rollouts / Flagger + service mesh for canary/blue-green, with metrics adapters to Prometheus, Grafana and a policy engine (OPA/Gatekeeper).- Observability & SLOs: Prometheus for metrics, Jaeger for tracing, ELK/Splunk for logs, Cortex/Thanos for global metrics.- Audit & approvals: Approval workflows via CD tool integrated with IAM/SSO and an approvals service; all approval events and pipeline actions are logged to immutable audit store (WORM-capable) and SIEM.- Secrets & config: Vault/KMS + GitOps for declarative config, signed manifests for compliance.Deployment workflow:1. Developer pushes PR → CI builds, runs unit + integration tests, creates signed artifact + SBOM, pushes to artifact registry.2. CI triggers CD: CD runs pre-deploy checks (policy/OAuth approval gates if required). For automated pipelines, human-approval gates enforced per environment with RBAC and timeboxed approvals.3. Progressive rollout: - Canary: Argo Rollouts config shifts 1–5–25–50% traffic based on metric windows. Metrics: error rate, latency p50/p95, CPU, business KPIs. If metrics degrade beyond thresholds, automated rollback is triggered. - Blue/Green: New environment provisioned; smoke tests + canary subset before switching route via mesh/ingress.4. Post-deploy: CD records deployment metadata, diff of manifests, artifact digest, approver IDs, timestamps to audit logs.Regional failover:- Active-active by default. Use global DNS/GLB (Route53/GCP Multi-Regional LB) with health checks + weighted routing.- Health checks combine not just L4 but application-level readiness (health endpoints augmented by mesh probes).- If a region becomes unhealthy: - Global control plane adjusts weights to shift traffic away to healthy regions (automated runbook in CD). - Stateful services: rely on cross-region read replicas and a promoted primary via controlled failover automation (see DB section).- Chaos/DR rehearsals automated via pipelines to validate failover.Database migration coordination:Principles: backward- and forward-compatible changes, zero-downtime where possible, explicit orchestration.Pattern:1. Prepare: Add additive changes first (new columns, tables), deploy services that can write both formats (dual-write) or read-old-or-new.2. Migrate data: Run online migrations via a migration service (k8s Job) that performs batched, idempotent updates against replicas; track progress in migration metadata table.3. Switch: Update services to start reading new fields (feature flag rollout) and run canary to validate.4. Cleanup: Once all regions and clients use new schema, run safe cleanup migrations to remove old columns.Coordination across regions:- Migrations are orchestrated from the global control plane with per-region execution windows and pre-checks (replica lag, schema drift). Use leader election for the migration orchestration job to avoid concurrent conflicting runs.- For stateful primaries, use promoted leader with a short coordinated maintenance window; for multi-master stores (DynamoDB/CockroachDB) rely on their cross-region replication semantics and perform schema changes compatible with CRDB.- For relational DBs (Postgres): use a promoted writer approach—promote a regional writer only when required; prefer online-safe migrations (ADD COLUMN with default NULL, backfill asynchronously). Use read replicas for cutover testing.Compliance, security & traceability:- Immutable artifacts (content-addressable), signed and scanned (Snyk/Trivy), enforce policy via admission controllers.- Approval gates: role-based approval with timeouts, enforced in pipeline; evidence captured in audit logs.- Audit: every pipeline action, approval, artifact promotion, and rollback logged to SIEM with immutable retention policies.- Access: SSO + least privilege IAM, mTLS inside mesh, secrets via Vault with audit logging.Failure & rollback behavior:- Automated rollback triggered when SLOs breach during canary windows; rollback is also gated by human approver if metrics ambiguous.- Rollbacks run region-locally first; global rollback orchestration reduces blast radius.- All rollbacks and forward actions produce audit events and roll-forward artifacts for forensics.Scalability & operational notes:- Multi-tenant control plane scales horizontally; regional agents reduce WAN dependency.- Maintain a library of reusable pipeline templates and service-health checks.- Regular DR and migration drills, plus documented runbooks and automated smoke-test suites.This design balances safety (progressive delivery, automated rollback, approvals, audits) with availability (active-active regions, global control plane, safe DB migration patterns) and is practical for a software engineer responsible end-to-end.
HardSystem Design
81 practiced
Architect a near-real-time analytics pipeline: capture DB changes via CDC (Debezium) into Kafka, process streams with Flink or Spark Structured Streaming, persist raw events in S3, and load aggregated results into a data warehouse (e.g., BigQuery or Redshift). Discuss schema evolution handling, exactly-once or at-least-once semantics, late-arriving data, and monitoring/alerting for stream processing.
Sample Answer
Requirements (functional & non-functional)- Capture DB changes with low latency (<5s), retain raw events in S3, provide rolling aggregates in DW (BigQuery/Redshift) within minutes.- Durability, fault tolerance, scalable to high throughput, support schema evolution, and clear processing semantics (prefer exactly-once for aggregates).High-level architecture- Source DB -> Debezium (CDC) -> Kafka (topic per table/schema) -> Stream processor (Flink / Spark Structured Streaming) -> Sink 1: Raw events to S3 (partitioned by table/date) -> Sink 2: Aggregates to DW (via streaming load or batch micro-batch).Core components & responsibilities- Debezium: capture change events (including schema metadata, op type, and timestamps).- Kafka: durable, partitioned log; use compacted topics for schema registry offsets if needed.- Schema Registry (Confluent or Apicurio): centralize Avro/JSON Schema versions; require producer to register schemas.- Stream Processor: - Flink (preferred for true exactly-once with connectors + two-phase commit) or Spark Structured Streaming with idempotent sinks. - Enrich/validate events, assign event-time, watermarking, windowed aggregations, and write results.- S3: raw event lake, partitioned by event date and topic; write in parquet, include schema version and CDC metadata.- Data Warehouse: load aggregated results via streaming ingestion API or atomic batch loads.Schema evolution- Use Schema Registry with backward/forward/compatible policies. Embed schema id in each Kafka message and file metadata in S3.- Stream jobs read schema id and use Avro/Protobuf for robust evolution; implement converters when incompatible changes occur (e.g., column rename handled via mapping layer).- For breaking changes, deploy a migration plan: dual-write or transformation job to normalize old/new schemas, and use feature flags to roll out consumers.Processing semantics- Exactly-once for aggregated results: - Flink + two-phase commit sinks (Kafka/BigQuery sink supporting transactions) ensures exactly-once. - If using Spark, aim for idempotent writes (upsert with deterministic keys, or use DW staging + dedup by unique event id).- At-least-once for raw event persistence to S3 is acceptable if files are immutable and include event unique IDs; dedupe downstream using event IDs.- Include event unique IDs (source LSN/offset + table + pk) to deduplicate.Late-arriving data & windowing- Use event-time semantics with watermarks. Configure acceptable lateness based on business SLA (e.g., 10 minutes).- Maintain side outputs for late events: either reprocess windows (Flink allows late panes) or write late events to a separate stream/bucket and run backfill jobs to reconcile aggregates.- For out-of-order long-tailed late data, run periodic corrective batch jobs that recompute affected aggregates from raw S3.Monitoring & alerting- Metrics: input/output lag, processing throughput, watermark progress, state size, checkpoint durations, failure/retry counts, offsets lag per partition.- Logs: structured logs with correlation IDs (event IDs, job/partition).- Alerts: - High consumer lag (Kafka lag > threshold) - Checkpoint failures or long checkpoint duration (> threshold) - Watermark stall or advancing too slowly - Sink errors or high retry rates - S3 write failures or increasing small-file counts- Tools: Prometheus + Grafana dashboards, alerting in PagerDuty/Slack, use Kafka Cruise Control for partitioning health, and DataDog/Cloud monitoring for DW ingestion metrics.- Observability: trace events end-to-end (Kafka offset -> Flink job -> DW load), capture lineage in catalog (e.g., Glue/Data Catalog).Operational considerations & trade-offs- Flink gives stronger exactly-once guarantees and better low-latency windowing; Spark is easier if team already runs structured streaming and uses micro-batches.- Balance retention vs cost for raw events in S3 (keep hot recent partitions, cold-archive older).- Testing: simulate failovers, schema changes, and late data with integration tests and canary jobs.- Security: encrypt data at rest/in transit, IAM roles for connectors, redact PII before landing.This design provides durable raw event storage, robust schema evolution handling, deterministic aggregate semantics, strategies for late data, and a monitoring stack to detect and respond to processing issues.
EasyTechnical
86 practiced
Explain the differences between unit tests, integration tests, and end-to-end tests. For a typical web application, provide one concrete example test for each type and explain how you would organize test execution in CI to provide fast developer feedback while ensuring reliability.
Sample Answer
Unit tests, integration tests, and end-to-end (E2E) tests target different scopes and trade-offs between speed, isolation, and confidence.- Unit tests: isolate a single function/class, fast and deterministic. Example: for a web app, a Jest test that verifies a date-formatting utility returns correct strings for edge inputs. Run in-memory, mock external calls. Purpose: catch logic bugs quickly.- Integration tests: exercise interactions between multiple components (e.g., API layer + database + auth). Example: a test that starts the app against a test PostgreSQL (or a Dockerized fixture), makes an HTTP POST to /orders, and asserts DB state and response codes. Purpose: validate contracts between modules.- End-to-end tests: simulate real user flows through the full stack (UI, backend, DB, third-party services). Example: a Cypress test that logs in, adds an item to cart, completes checkout, and verifies confirmation. Purpose: verify the system works end-to-end, including UX and deployment config.CI organization for fast feedback and reliability:- Follow the test pyramid: lots of fast unit tests, fewer integration, very few E2E.- On every PR: run unit tests and linters (fast, block PRs on failures). Run a subset of critical integration tests (smoke) in parallel.- On merge to main or scheduled: run full integration suite and E2E. Run E2E in a stable, identical environment (staging) and mark flaky tests for quarantine.- Use parallelism, test caching, and Dockerized test DBs to speed runs. Fail-fast on unit failures; allow non-blocking flaky-test alerts with tracking tickets.- Collect test artifacts (screenshots, logs) and require E2E pass before release to production. This balances rapid developer feedback with high confidence before deploy.
HardTechnical
129 practiced
Discuss strategies to maintain data consistency across microservices in a shopping cart system: compare eventual consistency with compensating transactions, distributed transactions (2PC), and event-sourcing. For each approach explain implementation complexity, failure modes, and how you would handle customer-visible anomalies.
Sample Answer
Start by clarifying the consistency surface in a shopping-cart system: operations include add/remove items (cart service), inventory reservation (inventory service), payment capture (payment service), and order lifecycle (order service). The user-visible invariants: items shown in cart reflect available inventory and final checkout should not oversell.1) Eventual consistency (async updates, pub/sub)- How: Cart service emits events (CartUpdated, CheckoutInitiated). Inventory and order services subscribe and update asynchronously. Use idempotent handlers, tombstones, and versioning (optimistic concurrency).- Implementation complexity: Low–medium. Simple pub/sub and retry logic; needs careful idempotency and reconciliation.- Failure modes: Out-of-order events, message loss, long propagation delays leading to stale UI (user sees item available but out-of-stock at checkout).- Customer anomalies: Show real-time availability hints, reserve on checkout initiation with short TTL, show “quantity not guaranteed” warnings, and reconcile with optimistic UI corrections (notify user, offer alternatives/partial fulfillments).2) Compensating transactions (sagas)- How: Model checkout as a saga of local transactions with forward actions and compensating actions (reserve inventory → charge payment → confirm order; if payment fails, release inventory).- Implementation complexity: Medium. Requires saga coordinator (or choreography), clear compensating logic, and robust state machine.- Failure modes: Partial failures where compensations fail or are delayed; out-of-order compensations; long-running locks/resources.- Customer anomalies: Provide transactional status UX (processing → payment failed → item released), allow retries, reserve nominal inventory on checkout to reduce surprises, and design compensations to be user-friendly (e.g., auto-retry payments, coupon for inconvenience).3) Distributed transactions (2PC)- How: Global coordinator runs prepare/commit across services using XA/2PC semantics.- Implementation complexity: High. Requires transactional resource managers, tight coupling, and orchestration.- Failure modes: Blocking during coordinator failure, reduced availability, poor scalability, complex recovery.- Customer anomalies: Generally avoids anomalies at the cost of higher latency and outages. Use only for very small critical scopes (e.g., accounting ledger). For shopping carts, avoid 2PC for core flow; if used, add timeouts and fallback compensations.4) Event sourcing (and CQRS)- How: Persist every intent as immutable events (ItemAdded, CheckoutRequested). Projections build read models; other services react to event stream.- Implementation complexity: High. Needs event store, projections, schema evolution, and tooling for replays.- Failure modes: Projection lag, event schema migrations, complex debugging, eventual divergence between projections.- Customer anomalies: Show "eventually consistent" UI with clear timestamps and loading states; allow replay-based reconciliation (rebuild projection) and manual fixes. Use snapshotting for performance and design idempotent consumers.Trade-offs & recommendation:- For shopping cart, use a hybrid: event-driven sagas for checkout flow (good balance of scalability and correctness) + short-lived inventory reservations to minimize user-visible failures. Use event sourcing for audit/complex business rules where replayability is valuable (e.g., promotions, fraud). Reserve 2PC for rare, tightly-scoped accounting integrations only. Always design UX to surface provisional states, provide explicit retries/compensations, and implement monitoring/alerts for reconciliation queues.
Unlock Full Question Bank
Get access to hundreds of Technology Stack Knowledge interview questions and detailed answers.
