Infrastructure Implementation and Operations Questions

Hands on design, deployment, and operational management of infrastructure components and services. This includes setting up and configuring load balancers, database replication and high availability, caching layers, networking and network security, service discovery and routing, container deployment and orchestration, monitoring and observability, logging and alerting, backup and disaster recovery strategies, and secrets management in runtime. Candidates should be able to walk through concrete implementations, explain trade offs, demonstrate troubleshooting and performance tuning, and show how infrastructure components integrate to meet availability, scalability, and security requirements.

MediumTechnical

49 practiced

You observe frequent cache stampedes on a single hot key in Redis used to serve enrichment lookups. Propose three mitigation strategies (architectural and operational), explain implementation steps and trade-offs for each, and how you'd measure success.

Sample Answer

Situation: A single hot key in Redis is causing frequent cache stampedes for enrichment lookups, leading to high origin load and latency. Here are three mitigation strategies with implementation steps, trade-offs, and success metrics.

1) Staggered/Mutexed Refresh (Lock + Dogpile prevention)- Implementation: On cache miss, try to acquire a short-lived Redis lock (SET key:lock NX PX 1000). If acquired, recompute and set the cache; if not, return stale value or block briefly and retry. Optionally implement “refresh-ahead”: proactively refresh keys before TTL expiry by a background worker.- Trade-offs: Extra complexity and potential single-lock bottleneck; must handle lock expiry and crashes to avoid data loss. Returning stale values trades freshness for availability.- Success metrics: Reduced origin QPS during bursts, lower error rates, and lower 95/99th tail latency for requests; monitor lock contention rate.

2) Cache Sharding / Key Splitting (fan-out keys)- Implementation: Instead of one monolithic hot key, split into N subkeys (hash by request attributes or add time/version shards). Use consistent hashing or prefix with a shard id computed from request properties. Aggregate results if needed.- Trade-offs: Increased memory footprint and complexity in lookups/aggregation; potential uneven shard distribution requiring rebalancing.- Success metrics: Reduced per-key QPS and lock contention, more even Redis CPU usage, improved throughput; monitor per-key QPS distribution and memory use.

3) Edge/Local LRU + Multi-tier Caching (client-side + Redis + origin)- Implementation: Add a local in-process LRU cache (e.g., Caffeine) or CDN/edge cache in front of Redis to absorb bursts. Set short local TTLs and fall back to Redis on miss. Optionally introduce a small warmed "hot" cache populated periodically by background refresh jobs.- Trade-offs: Risk of stale data and cache coherence complexity; needs more memory on many nodes; invalidation becomes harder.- Success metrics: Fewer requests reaching Redis (redis QPS down), lower end-to-end latency, improved cache hit ratio at local tier.

Measuring success across strategies:- Metrics: Redis QPS, origin service QPS, cache hit ratio (local and Redis), 95/99th latency, error rate, and CPU/memory on Redis nodes.- Validation: Load-test with synthetic bursts simulating peak traffic; confirm origin load and tail latency drop and no regression in data correctness.

HardTechnical

74 practiced

Implement a Terraform module conceptually that provisions an AWS Auto Scaling Group for Spark worker nodes using a mixed instances policy (spot + on-demand) and lifecycle hooks to gracefully drain Spark executors before instance termination. Describe the module inputs, outputs, and key resource blocks you would include (IAM, ASG, launch template, lifecycle hook).

Sample Answer

Approach: create a reusable Terraform module that builds a Launch Template (mixed instance types + spot overrides), an Auto Scaling Group with MixedInstancesPolicy, IAM role/policy for instance actions (SSM, cloudwatch, drain script), a Lifecycle Hook that triggers a Lambda (or SSM Run Command) to gracefully drain Spark executors before EC2 termination.

Example key snippets (conceptual):

Inputs:- name_prefix, vpc_subnet_ids, security_group_ids- launch_instance_types (list), on_demand_percentage, spot_allocation_strategy- ami_id, instance_profile_name, spark_master_endpoint, drain_timeout_seconds- desired_count, min_size, max_size, instance_types_overrides

Outputs:- asg_id, launch_template_id, autoscaling_group_name, iam_instance_profile

Key resources (conceptual HCL):

hcl

resource "aws_launch_template" "spark_workers" {
  name_prefix = "${var.name_prefix}-lt-"
  image_id    = var.ami_id
  instance_type = var.default_instance_type
  iam_instance_profile { name = aws_iam_instance_profile.spark.name }
  user_data = base64encode(templatefile("${path.module}/userdata/drain.sh.tpl", {
    spark_master = var.spark_master_endpoint
  }))
  tag_specifications { ... }
}

resource "aws_autoscaling_group" "spark_workers" {
  name                 = "${var.name_prefix}-asg"
  max_size             = var.max_size
  min_size             = var.min_size
  desired_capacity     = var.desired_count
  mixed_instances_policy {
    launch_template {
      launch_template_specification { launch_template_id = aws_launch_template.spark_workers.id, version = "$Latest" }
    }
    instances_distribution {
      on_demand_percentage_above_base_capacity = var.on_demand_percentage
      spot_allocation_strategy = var.spot_allocation_strategy
    }
    override = [
      for t in var.launch_instance_types : { instance_type = t }
    ]
  }
  vpc_zone_identifier = var.vpc_subnet_ids
  tag { key = "Name" value = "${var.name_prefix}-worker" propagate_at_launch = true }
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_iam_role" "instance" { ... }          # allow ssm:SendCommand, ec2:Describe*, cloudwatch logs
resource "aws_iam_instance_profile" "spark" { role = aws_iam_role.instance.name }

resource "aws_autoscaling_lifecycle_hook" "pre_terminate" {
  name                   = "${var.name_prefix}-drain-hook"
  autoscaling_group_name = aws_autoscaling_group.spark_workers.name
  lifecycle_transition   = "autoscaling:EC2_INSTANCE_TERMINATING"
  heartbeat_timeout      = var.drain_timeout_seconds
  default_result         = "CONTINUE"
  notification_target_arn = aws_sns_topic.drains.arn  # or SQS / EventBridge for Lambda
  role_arn = aws_iam_role.lifecycle.arn
}

Drain flow & reasoning:- Lifecycle hook pauses termination; sends notification (SNS/EventBridge) to a Lambda or step-function.- Lambda calls SSM RunCommand targeting the instance to run a drain script (user-data places spark utilities). The drain script uses spark/yarn APIs to decommission executors, wait for in-flight tasks, then signals completion by calling CompleteLifecycleAction via AWS SDK.- IAM roles: instance profile (SSM, CloudWatch), lifecycle role (autoscaling:CompleteLifecycleAction), Lambda role (ssm:SendCommand, autoscaling:CompleteLifecycleAction).

Edge cases & best practices:- Use multiple instance_type overrides to improve spot flexibility.- Set appropriate heartbeat_timeout > expected drain time + buffer.- Use SSM rather than SSH for robust command execution and security.- Emit metrics/logs to CloudWatch for failed drains and retry logic.- Ensure graceful decommission uses Spark/YARN recommended REST APIs to avoid data loss.

HardTechnical

44 practiced

Compare envelope encryption (data keys wrapped by cloud KMS) versus client-side encryption for a data lake storing PII. Discuss key management complexity, performance overhead, searchability/indexing, auditability, and regulatory compliance implications.

Sample Answer

High-level summary: Envelope encryption (data encrypted with per-object data keys that are themselves wrapped/unwrapped by a cloud KMS) is the operational default for cloud data lakes — it centralizes key control and minimizes client complexity. Client-side encryption (CSE) pushes full crypto responsibility to producers/clients: stronger end-to-end protection but higher operational burden and less platform-native functionality.

Key management complexity- Envelope: KMS handles master keys, rotation, lifecycle, and HSM-backed protection. Data engineers manage key policies, IAM roles, and occasional key rotation orchestration. Lower developer crypto burden.- Client-side: You must securely generate, store, rotate, distribute, revoke data keys across many clients. Requires secrets vaults, secure client libraries, offline key exchange, and handling lost keys -> high complexity.

Performance overhead- Envelope: Minimal overhead at write/read time — unwrap operation per object (or per batch) against KMS; often cached plaintext data key in memory for short duration to avoid per-object KMS calls. Scales well.- Client-side: Encryption/decryption occurs on client CPUs; can add latency at ingestion and extra compute in ETL jobs. No KMS cache benefit unless you implement your own key caching. Potentially higher network/CPU costs.

Searchability / indexing- Envelope: Data stored encrypted at rest but decrypted by compute nodes or analytics engines when authorized, enabling full-text search, indexing, and columnar operations. Works with tokenization or field-level encryption for selective protection.- Client-side: If clients encrypt entire fields with strong randomized encryption, you generally lose search/indexability. Deterministic encryption or searchable encryption schemes exist but reduce security and are complex to implement and scale. Querying often requires decrypting data post-retrieval, impacting performance and analytics workflows.

Auditability- Envelope: KMS provides centralized audit logs (who requested unwrapping, when), integrated with cloud logging for fine-grained access audits. Easier to generate evidence for access reviews.- Client-side: Auditing is fragmented: you must log key usage in your key distribution service and instrument each client. Harder to prove who decrypted data unless you build rigorous telemetry and tamper-resistant logging.

Regulatory compliance implications- Envelope: Meets many regulatory requirements if KMS is configured per controls (role separation, key rotation, export controls). Cloud providers offer compliance certifications (SOC, ISO, FedRAMP). But because cloud can access wrapped keys, consider legal exposure (cross-border, subpoenas).- Client-side: Stronger privacy guarantee (provider can’t decrypt) which helps regulatory needs where data escrow or provider access is disallowed. However, regulators often require ability to perform audits and data subject requests — ensure you can decrypt when necessary. Also, stronger operator burden to demonstrate compliant key management.

Practical recommendation for a data lake with PII- Use envelope encryption as the baseline for all at-rest protection, with KMS-managed CMKs and tight IAM plus audit logging.- For highest-risk fields (SSNs, financial IDs), use application-level (field) encryption at ingestion: either tokenization or client-side deterministic encryption for indexed fields, or client-side field encryption for end-to-end secrecy when provider access must be prevented.- Implement a hybrid pattern: KMS-wrapped data keys issued to trusted ingestion services (server-side) and true client-side encryption only where necessary. Maintain a central key registry, automate rotation, and enforce strict audit and key escrow policies to satisfy compliance.- Validate with legal/compliance teams and document key lifecycle and access procedures; perform regular crypto and key management reviews.

HardTechnical

34 practiced

During a major upgrade of a central metadata service, downstream pipelines produced inconsistent outputs. Outline a thorough postmortem plan: how you'd capture and preserve evidence, perform root-cause analysis, design data repair strategies (backfill, patching), quantify customer impact, and change processes to prevent recurrence.

Sample Answer

Framework: treat this as an incident postmortem with evidence preservation, RCA, remediation (repair), impact quantification, and preventive changes.

1) Capture & preserve evidence- Stop any destructive writes to downstream targets (snapshot mode).- Take immediate immutable snapshots: metadata DB logical dump, HDFS/S3 object versions, pipeline run logs, job dags, config and migration scripts, K/V store revisions, and timestamps.- Export versions of source tables, affected downstream tables, and message queues (Kafka offsets) to isolated storage.- Record environment: deploy manifests, feature flags, and exact binary hashes.- Document timeline from deploy to detection with timestamps.

2) Root-cause analysis- Reconstruct timeline (who/what/when). Correlate deploy events with metadata changes and affected pipeline runs.- Use diffs between pre-upgrade and post-upgrade metadata (schemas, defaults, referential constraints).- Re-run representative failing pipeline runs in a sandbox with controlled inputs to reproduce.- Hypothesize causes (e.g., incompatible schema migration, default-value change, caching invalidation bug, race during migration). Validate with targeted tests and log traces.- Identify contributing factors (insufficient integration tests, missing canary, lack of idempotent migrations).

3) Data repair strategies- Classify affected data: stale, incorrect, duplicated, missing.- Non-destructive patching: write idempotent correction scripts that map bad -> good, using joins against authoritative snapshots. Prefer append-only correction tables plus view-layer remapping for auditability.- Backfill: re-run historical pipeline jobs deterministically from preserved Kafka offsets or snapshot timestamps. Use orchestration (Airflow) with deduplication and watermarking to avoid double-processing. Parallelize via date/partition.- Hotfix for recent windows: targeted patch jobs for only affected partitions to reduce cost/time.- Verify via checksums/row-counts and data-quality rules before promoting repairs.

4) Quantify customer impact- Define metrics: number of impacted downstream runs, affected users/dashboards, data consumers, SLA breaches.- Measure scope: rows affected, percentage of daily volume, number of dependent models/reports.- Business impact: revenue-affecting reports, ML model performance degradation (drift), support tickets.- Provide timeline of exposure (how long incorrect data was visible).

5) Prevent recurrence (process & technical)- Enforce migration safety: staged migrations (backfill -> dual-write -> cutover), explicit compatibility checks, automated pre-deploy contract tests.- Add canary deployments and consumer-side compatibility tests; run full integration tests against synthetic and sampled production data.- Improve observability: data lineage, end-to-end data quality SLA metrics, alerts on schema/default changes, and downstream discrepancy detectors (row-count diffs, cardinality, checksum).- Change management: migration playbooks, mandatory runbooks, rollback plans, DB migration approval for metadata-affecting changes.- Practice drills and runbooks; schedule post-deploy verification window before enabling full traffic.

Outcome: deliver a timeline-backed postmortem, tested repair plan with verifiable QA, quantified impact for stakeholders, and prioritized engineering actions (tests, canary, migration patterns) tracked to completion.

EasyTechnical

46 practiced

Explain cache-aside, write-through, and write-back caching strategies. Give concrete examples of when each is appropriate in a data platform: e.g., metadata cache for query planning, frequently-read lookup tables, or buffering writes for bursty ingestion.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Implementation and Operations interview questions and detailed answers.

Join thousands of developers preparing for their dream job.