Technical Background and Skills Questions

Provide a clear, evidence based overview of your technical foundation and demonstrated credibility as a technical candidate. Describe programming and scripting languages, frameworks and libraries, databases and data stores, version control systems, operating systems such as Linux and Windows, server and hardware experience, and cloud platforms including Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Explain experience with infrastructure as code tools, containerization and orchestration platforms, monitoring and observability tooling, and deployment and continuous integration and continuous delivery practices. Discuss development workflows, testing strategies, build and release processes, and tooling you use to maintain quality and velocity. For each area, explain the scale and complexity of the systems you worked on, the architectural patterns and design choices you applied, and the performance and reliability trade offs you considered. Give concrete examples of technical challenges you solved with hands on verification details when appropriate such as game engine or platform specifics, and quantify measurable business impact using metrics such as latency reduction, cost savings, increased throughput, improved uptime, or faster time to market. At senior levels emphasize mastery in three to four core technology areas, the complexity and ownership of systems you managed, the scalability and reliability problems you solved, and examples where you led architecture or major technical decisions. Align your examples to the role and product domain to establish relevance, and be honest about gaps and areas you are actively developing.

HardTechnical

84 practiced

Design a key-management and envelope-encryption scheme for multi-tenant data stored across S3 and databases. Requirements: per-tenant isolation, key rotation, high QPS support, and audit logging. Explain KMS integration, caching design to reduce KMS calls, procedures for rotation and emergency key revocation, and recovery when a key is compromised.

Sample Answer

Requirements clarification:- Per-tenant isolation: each tenant’s data keys must be logically separated and access-controlled.- Key rotation, emergency revocation, high QPS: minimize KMS latency while keeping security.- Strong audit trail for all key operations.

High-level design (envelope encryption):- For every object/row/blob: generate a unique data encryption key (DEK, AES-256).- Encrypt payload with DEK (AES-GCM).- Encrypt (wrap) the DEK with the tenant’s customer master key (CMK) in KMS — store ciphertext-DEK with the object/row in S3 metadata or DB column.- Store metadata: tenant_id, key_version, ciphertext-DEK, algorithm, timestamp, and audit fields.

KMS integration:- Use a per-tenant CMK (either one CMK per tenant or one CMK per tenant-group). Manage CMKs in KMS (AWS KMS example).- Use KMS GenerateDataKey API to get plaintext DEK + ciphertext-DEK when creating data.- Use KMS Decrypt to unwrap ciphertext-DEK to plaintext DEK on read.- All KMS calls logged to CloudTrail/Audit logs; tag requests with tenant context for filtering.

Caching design to reduce KMS calls (high QPS):- Primary approach: local in-process DEK cache (memory-only). Cache decrypted plaintext DEKs keyed by (tenant_id, key_version) with strict TTL (e.g., 1–5 minutes), LRU, and max-size. Plaintext DEKs never persisted to disk. Evict on process exit. This dramatically reduces Decrypt calls for hot tenants.- Use AWS Encryption SDK’s Caching Cryptographic Materials Manager or build similar: cache cryptographic materials (plaintext DEKs + metadata) with strict usage limits (max bytes encrypted, max time).- For multi-node sharing: avoid caching plaintext across network. If a cross-node cache is required, store only ciphertext-DEKs and let each node call KMS Decrypt on first access, then use local cache.- Harden cache: allocated in secure memory, zeroed on eviction, container isolation, strict IAM that only process role can call KMS.

Rotation procedure:- CMK rotation: avoid in-place rotation that invalidates existing ciphertexts. Recommended patterns: 1. Versioned keys: create new CMK (or a new key version alias tenant_cmks/tenantA/v2). 2. New writes use new CMK/version to GenerateDataKey and key_version recorded in metadata. 3. Re-wrap (re-encrypt) existing ciphertext-DEKs: use a background rewrap job that: - Reads object metadata, calls KMS Decrypt with old CMK to get plaintext DEK (requires KMS access). - Calls KMS Encrypt/GenerateDataKey? (or rewrap via KMS ReEncrypt if supported) with new CMK to produce new ciphertext-DEK, update metadata atomically. - Validate checksums and update audit log. 4. Option: lazy rewrap — on first read after rotation, decrypt with old CMK (via KMS), then re-encrypt DEK with new CMK and update object.- Ensure rewrap jobs are idempotent and rate-limited to avoid KMS throttle.

Emergency revocation and compromised key recovery:- If a CMK is compromised (key material leaked or IAM compromised), take immediate containment actions: 1. Disable the compromised CMK in KMS (fast), and revoke IAM credentials that allowed KMS operations. 2. Create replacement CMK(s) and update alias (tenantA/current -> tenantA/new). 3. For immediate reads: if CMK is disabled, Decrypt fails. Use rewrap procedure only if you can still decrypt (if attacker has CMK material, you cannot guarantee confidentiality of old ciphertexts). 4. Recovery/rekey strategy: - If compromise limited to CMK metadata (not key material), rewrap as normal. - If key material definitely leaked (attacker can decrypt past ciphertexts), assume previous ciphertexts are exposed. Mitigation is: - Generate new DEKs and re-encrypt data (decrypt payload using either backed-up plaintext DEKs or by calling KMS Decrypt if KMS still usable) then re-encrypt DEK with new CMK. - If you cannot safely decrypt because KMS was disabled, restore from a secure backup taken before compromise (if allowed) or use application backups that include encrypted DEKs and rewrap those after rotation. - Rotate tenant credentials, rotate downstream secrets, and force clients to use new keys.- Post-incident: audit, rotate all affected keys, perform forensics via CloudTrail/KMS logs, notify stakeholders.

Audit logging and monitoring:- Log every GenerateDataKey, Decrypt, ReEncrypt, Enable/Disable/Rotate action to centralized audit (CloudTrail + SIEM).- Emit application-level audit when metadata updated (who, job id, old_key_version -> new_key_version).- Monitor KMS error rates, latency, and unusual patterns (e.g., spikes in Decrypt from unusual principals).- Tag logs with tenant_id and correlation_id.

Operational considerations / trade-offs:- Per-tenant CMKs gives strongest isolation but increases management overhead and KMS quotas; grouping tenants can reduce quota pressure.- Caching plaintext DEKs in-process trades a small RISK for large performance gains. Keep TTLs short and enforce usage caps (bytes encrypted, ops).- Use async background rewrap to avoid large migration windows; lazy migration reduces immediate load.- For very high QPS and ultra-low latency, consider using a dedicated cryptographic materials service running in a hardened environment (e.g., Nitro Enclaves or a managed KMS cache) that exposes secure APIs and is itself access-controlled and audited.

Example lifecycle (write/read):- Write: GenerateDataKey(CMK-tenantA) -> get plaintext DEK + ciphertext-DEK -> encrypt payload with DEK -> store payload + ciphertext-DEK + key_version.- Read: check local cache for (tenantA,key_version) -> if cached plaintext DEK, decrypt payload locally; else call KMS Decrypt(ciphertext-DEK) obtain plaintext DEK, cache it briefly, then decrypt payload.

This design meets per-tenant isolation, supports rotation and emergency procedures, scales via local caching of DEKs, and preserves an auditable trail of all key operations.

MediumTechnical

78 practiced

How do you manage branching and deployment for DAGs (e.g., Airflow) and schema migrations to ensure minimal disruption? Explain strategies for versioning DAGs, ensuring backward compatibility, testing in staging, and promoting changes to production without causing duplicate runs or schema mismatches.

Sample Answer

Situation: In my last role I maintained dozens of Airflow DAGs and coordinated schema changes for downstream warehouses. To avoid outages, I used a disciplined branching, testing and deployment pattern that ensured backward compatibility and prevented duplicate runs.

Strategy (branching & versioning):- Feature branches for work; PRs with DAG + migration changes together.- Version DAGs explicitly in code and Airflow (dag_id includes semantic version or timestamp like my_dag_v2) so old and new can coexist during transition.- Keep tasks idempotent and add run-level dedupe (e.g., use external run registry or unique constraints on job_id/run_date) to prevent double processing.

Schema migrations (safe deploy order):- Follow expand-then-contract: 1. Deploy code that tolerates both schemas (read old columns, write new columns). 2. Run background/ETL jobs to backfill/populate new columns. 3. Switch readers to new columns. 4. Remove old columns in a later deploy.- Use migration tools (Alembic/Flyway) with transactional, reversible scripts; include checks for long-running migrations and perform offline/partitioned rollouts for large tables.

Testing & staging:- CI: unit tests for operators, integration tests using Dockerized Airflow and a fork of the staging DB. Run linters and DAG import tests on PR.- Staging runs: deploy DAG version to staging with production-like schedules but run off-peak and with mocked upstream data. Use end-to-end test runs and assert downstream table schemas and row counts.- Smoke tests after migration scripts: validate schema, row counts, and sample queries.

Promotion to production:- Use CI/CD pipelines to promote only merged tags; artifact is a release commit that includes DAG and migration.- Order: first deploy backward-compatible DAG code to prod, then apply migration scripts in a controlled window (or run them in read-replica-first mode). Monitor metrics and job health.- Blue/Canary: enable new DAG version for a small subset of partitions/dates or a canary run. Once stable, pause the old DAG or retire it after a grace period.- Prevent duplicate runs: ensure scheduler only activates one dag_id; use Airflow REST to pause/unpause atomically, and rely on idempotent tasks and unique constraints.

Monitoring & rollback:- Add health checks, alerts on task failures, schema drift detectors, and dashboards for run counts.- Rollback plan: if migration causes errors, revert to code path that reads old schema (supported by initial backward-compatible deploy), and run rollback migration if necessary.

Why this works:- Explicit versioning + expand-then-contract migrations avoid breaking readers.- CI + staging with end-to-end tests catch logic/schema mismatches early.- Idempotency, dedupe mechanisms and controlled promotion prevent duplicate processing and ensure minimal disruption.

HardTechnical

83 practiced

Explain how HyperLogLog and Count-Min Sketch work for cardinality and frequency estimation in streaming systems. Cover error bounds, mergeability, memory trade-offs, and provide the math or pseudo-code for calculating expected error and merging sketches. Give a real-world use-case where you applied one of these.

Sample Answer

HyperLogLog (HLL) and Count–Min Sketch (CMS) are compact probabilistic summaries for streaming cardinality and frequency estimation respectively.

HyperLogLog — cardinality- Idea: hash each item to a uniformly random bitstring; use the position of the leftmost 1 (leading zero count +1) to estimate rarity. Keep m = 2^p registers R[0..m-1]; for hash h(x), index = first p bits, rho = leadingZeros(remaining)+1, R[index] = max(R[index], rho).- Estimate: raw E = alpha_m * m^2 * (1 / sum(2^{-R[i]})), where alpha_m is bias-correction constant. Expected relative error ≈ 1.04 / sqrt(m). So error ∝ 1/√m (memory m registers).- Mergeability: element-wise max of registers: R_merged[i] = max(Ra[i], Rb[i]).- Memory trade-off: each register stores small integer (log log n); p controls m and error.

Pseudo:

python

# p bits -> m=2^p
def add(h):
  idx = h >> (64-p)
  rho = leading_zeros((h << p) | 1) + 1
  R[idx] = max(R[idx], rho)
def merge(Ra,Rb): return [max(a,b) for a,b in zip(Ra,Rb)]

Count–Min Sketch — frequency- Structure: a d x w array with d hash functions. On add(x, +c): for i in 1..d, C[i][h_i(x)] += c. Estimate count = min_i C[i][h_i(x)].- Error bound: with w = ceil(e/ε) and d = ceil(ln(1/δ)), estimate ≥ true_count and overestimation ≤ ε * total_count with probability 1−δ. So space O((1/ε) log(1/δ)).- Mergeability: element-wise addition of matrices.- Trade-offs: CMS is biased (only overestimates) but constant time and merges trivially; to reduce variance need larger w or use conservative update.

Pseudo:

python

def add(x,c):
  for i in range(d):
    C[i][h_i(x)%w] += c
def estimate(x):
  return min(C[i][h_i(x)%w] for i in range(d))
def merge(A,B): return [[a+b for a,b in zip(rowA,rowB)] for rowA,rowB in zip(A,B)]

Real-world use-case (I implemented): at my previous company I used HLL in Kafka-stream processors to estimate unique active users per day across regions. We maintained per-partition HLLs, merged them at window close (element-wise max), and stored compact HLL blobs in a time-series store. This cut storage by ~1000x vs raw user lists, provided ±1.6% error (p=14), and enabled fast rollups for analytics dashboards. For heavy-hitter detection in the same pipeline we used CMS (ε=1e-3, δ=1e-6) to track top event types with mergeable sketches for parallel processing.

EasyTechnical

88 practiced

Define key data quality metrics a team should track (completeness, uniqueness, validity, timeliness, accuracy). Describe a data-quality test you implemented (for example: null-rate threshold, FK existence, distribution drift) and how you automated remediation or alerting when violations occurred.

Sample Answer

Key metrics to track- Completeness: fraction of non-null values per column (null-rate). Targets per column based on business rules.- Uniqueness: % of unique keys (primary key or candidate key) and duplicate counts.- Validity: conformity to schema/regex/type/range (e.g., email format, date ranges).- Timeliness: latency between event time and availability in downstream store; SLA breach rate.- Accuracy: correctness vs. trusted source (sampling or reconciliation with gold table), e.g., percent match.

Concrete data-quality test I implementedSituation: In a customer ingestion pipeline we saw downstream analytics skew from month to month.Test: I implemented a null-rate threshold and foreign-key (FK) existence check using Great Expectations.

Example GE expectation (simplified):

python

# python
expect_column_values_to_not_be_null = {
  "expectation_type": "expect_column_values_to_not_be_null",
  "kwargs": {"column": "customer_id", "mostly": 0.995}
}
expect_fk = {
  "expectation_type": "expect_column_values_to_be_in_set",
  "kwargs": {"column": "customer_id", "value_set": "reference_customer_ids"}
}

Automation & remediation- Scheduled checks in Airflow after ETL task; failing checks mark DAG XCOM and trigger downstream steps.- Alerting: failures post as formatted message to Slack and create an incident in PagerDuty with context (failing expectation, sample rows, job/run id).- Remediation: for transient null-rate spikes I automated a backfill DAG that re-ingested source batches for the affected time window; for FK failures I quarantined offending rows into a landing table and triggered a lightweight data-owner notification workflow with instructions and a pre-built SQL to reconcile.- Metrics surfaced in Prometheus/Grafana (per-check pass rate, MTTR).

Result: Alerts reduced silent data issues; automated backfills fixed ~70% of pipeline failures without manual intervention and MTTR dropped from days to under 3 hours. Key learnings: set realistic thresholds, include sample rows in alerts, and ensure humans can opt out of auto-remediation.

EasyTechnical

105 practiced

Explain ETL vs ELT and describe a concrete incremental extraction strategy you implemented. Include how you tracked offsets or bookmarks, handled schema drift, validated the incremental loads (row counts, checksums), and how you recovered from a failed incremental load.

Sample Answer

ETL vs ELT (brief): ETL extracts data, transforms it in a processing layer, then loads the cleaned data to the target. ELT extracts and loads raw data first (typically to a data lake/warehouse), then runs transformations inside the target (leverages warehouse compute, easier replay/backfill).

Concrete incremental extraction strategy I implemented:Situation: Ingesting transactional rows from a Postgres OLTP into a Redshift DW daily.

Approach:- Extraction: used high‑watermark (last_updated timestamp) + primary key tie-breaker. A metadata “bookmarks” table stored last_successful_timestamp and last_id to avoid missing rows with identical timestamps.- Bookkeeping: pipeline wrote per-source offsets to a durable metadata table in the control DB after successful commit. For Kafka CDC pipelines we used offsets stored in Kafka Connect’s committed offsets.- Schema drift: registered source schemas in a lightweight schema registry (JSON schema). On each run we compared incoming schema to registry; compatible additive changes (new nullable columns) auto-added to staging; breaking changes (type change, removed column) triggered alerts and automated creation of a transformed mapping job with schema migration PR for engineers.- Validation: after load to staging, ran row-count compare (source vs staged) and checksum/hash (MD5 of concatenated stable columns) per-partition. Results logged to an audit table. Thresholds: zero-tolerance for row loss; checksum mismatch caused retry.- Recovery: pipelines are idempotent—loads to staging used COPY into temp table + transactional swap (or upsert using MERGE). On failure, we: 1) Did not advance the bookmark. 2) Investigated logs and re-ran extraction for the exact bookmarked window. 3) If partial load occurred, we truncated staging partition and re-ingested. For CDC streams we re-consumed from last committed offset.Result: This produced reliable daily incrementals with clear observability; incidents were recoverable within 30–60 minutes and required minimal manual intervention.

Unlock Full Question Bank

Get access to hundreds of Technical Background and Skills interview questions and detailed answers.

Join thousands of developers preparing for their dream job.