Ownership and Project Delivery Questions

This topic assesses a candidate's ability to take ownership of problems and projects and to drive them through end to end delivery to measurable impact. Candidates should be prepared to describe concrete examples in which they defined goals and success metrics, scoped and decomposed work, prioritized features and trade offs, made timely decisions with incomplete information, and executed through implementation, launch, monitoring, and iteration. It covers bias for action and initiative such as identifying opportunities, removing blockers, escalating appropriately, and operating with autonomy or limited oversight. It also includes technical ownership and execution where candidates explain technical problem solving, architecture and implementation choices, incident response and remediation, and collaboration with engineering and product partners. Interviewers evaluate stakeholder management and cross functional coordination, risk identification and mitigation, timeline and resource management, progress tracking and reporting, metrics and impact measurement, accountability, and lessons learned when outcomes were imperfect. Examples may span documentation or process improvements, operational projects, medium sized feature work, and complex or embedded technical efforts.

HardSystem Design

30 practiced

Design a telemetry system that can compute and surface pipeline-level SLOs (freshness, completeness, accuracy) across 1000 pipelines. Describe the data model for telemetry events, aggregation windows, storage choices, query patterns for dashboards, and how to link an SLO breach back to candidate root causes.

Sample Answer

Requirements & constraints:- Track 1,000 pipelines; compute SLOs: freshness (latency), completeness (expected vs ingested), accuracy (validation pass-rate).- Near-real-time alerts (1–5 min), historic queries (90+ days), dashboards, root-cause linking to pipeline runs/components.- High-cardinality dims (pipeline, dataset, partition, region, cluster).

High-level architecture:- Telemetry producers (ETL runners, schedulers, validators) emit structured events to a streaming bus (Kafka/PubSub).- Stream processors (Flink/Spark Structured Streaming) enrich, window-aggregate, write to OLAP (BigQuery, Snowflake, ClickHouse) and to a time-series DB (Prometheus or M3) for low-latency SLO metrics.- Metadata & lineage service (Data Catalog + lineage store) and tracing (Jaeger/OTel) for RCA.

Telemetry event data model (JSON/Avro):- event_type: run_start/run_end/ingest_record/validation_result/heartbeat- pipeline_id, run_id, task_id- dataset_id, partition_key (date/hour)- record_count_expected, record_count_observed- bytes_expected, bytes_observed- validation_checks: [{check_name, status, failed_rows, sample_hash}]- timestamps: event_time, producer_time- source_host, cluster, image_version, git_commit- error_code, stack_trace (optional), tags

Aggregation windows:- Real-time sliding: 1m/5m for alerting- Rollup windows: 1h, 6h, 24h for dashboards- Daily/hourly partitioning keyed by pipeline_id + partition_key for completeness/freshness

Storage choices:- Raw events: partitioned object store (S3/GCS) + catalog (Parquet/ORC)- Near-real-time SLO metrics: TSDB for freshness/latency, and OLAP (BigQuery/ClickHouse) for completeness/accuracy aggregates and ad-hoc queries.- Index lineage & traces in a graph DB or dedicated lineage store for fast traversal.

Query patterns for dashboards:- KPI queries: pipeline-level SLO time-series (TSDB) for graphs; heatmaps for top offenders (OLAP)- Drill-down: join OLAP aggregates with raw events on run_id/partition_key, filter by time window, validation_check- Ad-hoc RCA: query recent failed validation samples, error codes, and task-level durations from raw events

Alerting & breach -> root cause linking:- On SLO breach (from stream processor), emit alert with context: pipeline_id, window, metric delta, top failing validation checks, run_ids.- Attach pointers: run_id(s), sample keys, trace_id(s) and lineage path to upstream datasets/components.- Automated RCA steps: - Correlate timestamped traces & task logs (trace_id in events) to find slow/failed tasks. - Use lineage store to surface upstream pipelines/feeds that contributed missing partitions or anomalous data volumes. - Run automated impact analysis: compare expected vs observed record_count by upstream source, schema drift detection (hashes), and config changes (git_commit).- Provide interactive dashboard with “Investigate” that executes prepared queries: recent runs, task-level latencies, validation failure samples, upstream producer health.

Operational considerations & trade-offs:- Use streaming for low-latency SLOs; OLAP for correctness and ad-hoc queries.- Retention: keep high-resolution TSDB short (30–90d), OLAP/Parquet long-term.- Sampling: keep full validation summaries but sample failure rows for storage cost control.- Backpressure & cost: batch writes to OLAP, compress Parquet, use partition pruning.

This design gives near-real-time SLO visibility across 1k pipelines and deterministic links from SLO breaches to candidate root causes via run_ids, traces, lineage, and sampled failure evidence.

EasyTechnical

37 practiced

Explain the difference between an SLA and an SLO in the context of data pipelines. Provide two concrete examples of SLOs for batch and streaming jobs and describe how you would translate them into alerts and an escalation path.

HardTechnical

37 practiced

A critical pipeline depends on several third-party services that intermittently fail, causing data loss. Propose a resilient architecture and operational plan to eliminate data loss and meet at-least-once delivery guarantees. Include buffering, durable handoffs, retry strategies, idempotency design, and runbook practices.

Sample Answer

Requirements:- At-least-once delivery, zero data loss, tolerates intermittent third-party failures, bounded latency SLAs, observable and operable.

Resilient architecture (high-level):Producer -> Ingress buffer (durable, ordered) -> Durable processing / transformation -> Durable handoff queue -> Deliverer -> Third-partyUse: Kafka (or cloud: Kinesis/SQS+Dynamo for checkpointing) for ingress; object store (S3/GCS) as long-term dead-letter; relational DB or transactional store for metadata/checkpoints.

Key components & reasoning:- Ingress buffering: Kafka topic with replication and retention holds raw events until consumed successfully. This prevents immediate loss when downstream flaps.- Durable handoffs: After transformation, write an “intent” record to a transactional store (e.g., DB row or Kafka topic) before attempting delivery — provides exactly-traceable state.- Delivery worker: Reads handoff, attempts delivery, marks status atomically. Use consumer groups and commit only after marking delivered.- Persistent retries: Exponential backoff with jitter, configurable retry policy, and retry queue (separate Kafka topic or DLQ with retry timestamps). Use rate-limiting to protect third-party.- Idempotency: Design idempotent payloads — include unique event_id, sequence numbers, and stable keys. Support deduplication at third-party if possible; otherwise enforce client-side idempotency by storing delivered event_ids in a fast store (Redis or DynamoDB with TTL and conditional writes).- Backpressure and circuit breaker: Monitor error rate; when threshold exceeded, open circuit and switch to batch/backup export to object store for asynchronous manual or bulk replay.- Durable archival: All outgoing attempts and final states persisted to S3/GCS for forensic replay.

Operational plan & runbook:- SLOs/alerts: alert on consumer lag (Kafka lag), retry queue growth, error rates, circuit breaker trips, delivery latency.- Runbook steps: (1) Detect: high lag or DLQ increase. (2) Triage: check third-party health, auth issues. (3) Mitigate: open circuit, increase retries, scale workers, or pause delivery and failover to bulk export. (4) Recovery: once third-party healthy, replay from handoff/archival with controlled rate. (5) Postmortem: record root cause, update thresholds.- Automation: Auto-scale workers based on lag; automated replay tool with dry-run and idempotency checks.- Testing: Chaos tests that simulate third-party failures, and DR drills to validate replay and runbook effectiveness.

Trade-offs:- At-least-once may yield duplicates — idempotency/deduplication handles correctness at cost of additional storage and checks.- Latency vs durability: longer retention increases durability but may delay final delivery.

This design ensures no events are dropped, supports safe retries, and provides clear operational guidance for recovery.

MediumTechnical

25 practiced

You are asked to migrate a set of ETL workflows from an on-prem Hadoop cluster to AWS using EMR, S3, and AWS Glue. Provide a high-level migration plan that covers discovery, testing, performance validation, data movement strategy, risk mitigation, rollback plan, and a staging-to-production cutover checklist.

Sample Answer

Discovery & assessment- Inventory workflows, datasets, dependencies, schedules, SLAs, HDFS layout, Hive/HCatalog schemas, UDFs, custom jars, and job resource profiles (CPU, memory).- Prioritize by business impact and complexity.

Design & mapping- Target stack: S3 as data lake (raw/curated), EMR for Spark/MapReduce, AWS Glue Data Catalog for metadata, Glue jobs for lightweight ETL/ETL orchestration or AWS Step Functions for orchestration.- Map HDFS/Hive → S3 paths, Hive metastore → Glue Catalog (schema differences), convert HiveQL → Spark/Glue ETL where needed.

Data movement strategy- Initial bulk: DistCp from HDFS → S3 on an EMR temporary cluster (or use AWS DataSync/Snowball for very large datasets).- Continuous delta: export Kafka/CDC or scheduled incremental jobs; set up partitioned writes to S3 and use manifest/atomic commits (write to staging prefix then rename).- Preserve checksums and timestamps; validate row counts and sample hashes.

Testing strategy- Unit test converted jobs with small data, integration tests against staging Glue Catalog and S3, end-to-end validation for pipelines.- Data quality tests: row counts, column-level checksums, null/uniqueness constraints.- Security tests: IAM, KMS encryption, bucket policies.

Performance validation- Baseline: capture job runtimes and resource usage on-prem.- Benchmark EMR instance types, EMR versions, Spark configs (executor/driver sizing, dynamic allocation, shuffle settings), and S3 consistency/tuning (EMRFS vs S3A).- Run load tests with production-like data; measure latency, throughput, cost. Tune partitions, file sizes (128–512MB), use Parquet/ORC and compression.

Risk mitigation & rollback- Run in parallel: keep on-prem cluster live while validating outputs from cloud pipelines.- Canary runs for high-risk pipelines.- Maintain immutable raw snapshots on S3 and keep on-prem HDFS backups for fallback.- Rollback: if critical failures post-cutover, re-point consumers back to on-prem Hive/warehouse and stop Glue/EMR jobs.

Staging→Production cutover checklist- Pre-cut: Glue Catalog schemas synced; IAM roles, KMS keys, bucket policies applied; monitoring/alerts (CloudWatch, EMR Ganglia, Glue metrics) configured; cost controls and tags applied.- Data: latest full/partition copy completed; data quality checks passed; sample queries validated.- Jobs: converted jobs tested, retries and failure handling configured, DAG orchestration validated, SLA alarms set.- Cutover day: disable on-prem schedulers for migrated jobs; run final delta export; run cloud ingestion and compare row counts and hashes; switch downstream consumers to new S3/Glue endpoints; monitor closely for 48–72 hours.- Post-cut: decommission schedule, archive HDFS for 30–90 days, run a post-mortem and cost/perf review.

Key considerations- Security (VPC endpoints for S3, least-privilege IAM), compliance, data lineage (Glue Catalog + Lake Formation), and cost optimization (Spot + autoscaling).

HardSystem Design

34 practiced

Design an access control and data governance workflow that enables self-serve data discovery but prevents unauthorized access to PII. Include data classification, automated discovery, approval workflows, policy enforcement, audit trails, and how to scale the process across many datasets and teams.

Sample Answer

Requirements & constraints:- Enable self-serve discovery for analysts/scientists while preventing unauthorized PII access (least privilege).- Automated classification and continuous discovery across many datasets.- Policy-driven enforcement, approval workflows, full audit trail for compliance (GDPR/CCPA).- Scale to thousands of datasets, multiple teams, cloud-native stack.

High-level architecture:Catalog & Discovery → Classification & Tagging → Policy Engine → Access Workflows → Enforcement Layer → Audit & Monitoring

Components and responsibilities:1. Metadata Catalog (e.g., AWS Glue Data Catalog, Amundsen/DataHub) - Stores dataset schemas, owners, tags, sensitivity levels, sample lineage. - Provides search API/UI for self-serve discovery.

2. Automated Discovery & Classification - Batch/stream scanners (Spark jobs or Lambda functions) run on schedule and on ingestion: sample data, infer schema, run PII detectors (regex, ML models for names/SSNs, entropy checks). - Output: sensitivity tags (PII, Confidential, Public) and confidence scores written to catalog.

3. Data Classification Service - Validation rules + human-in-the-loop: classification that’s >threshold accepted; medium-confidence triggers owner review via ticketing (Jira/ServiceNow) integrated with catalog.

4. Policy Engine (e.g., Apache Ranger, OPA) - Centralized, tag-based policies (role, purpose, dataset sensitivity, allowed operations). - Support conditional rules: e.g., analysts can query anonymized aggregates of PII, data scientists with approved use-case and training can access masked data.

5. Approval & Access Workflow - Self-serve request UI in the catalog: user requests access specifying purpose and duration. - Automated checks: training/compliance status, manager approval, business justification. - If approved, ephemeral credentials issued (short-lived IAM roles, token-bound to purpose) and mapped to dataset-level policies.

6. Enforcement Layer - Enforce at compute & storage layers: table-level masking, row-level filters, column-level encryption. - Implement techniques: dynamic data masking, tokenization, differential privacy for exports, view-based access for Redshift/BigQuery/Snowflake, Spark job runner that injects filters. - Use data virtualization or query gateway to rewrite queries to enforce policies.

7. Audit, Logging & Monitoring - All requests, approvals, queries, data accesses logged to immutable store (WORM S3) and SIEM. - Periodic attestation by owners, anomaly detection for unusual access patterns, automated alerts for policy violations.

Data flow (example):New dataset ingested → scanner classifies columns as PII/confidence → tags written to catalog → if high-confidence PII, create masked view and default restrict policy → dataset visible in catalog (metadata-only) → analyst requests access → approval flow validates training & purpose → temporary role granted with scoped permissions → queries routed through policy enforcement layer → access logged.

Scaling strategies:- Tag-based policy model avoids per-dataset rules.- Use autoscaling serverless scanners (Spark on Kubernetes or EMR autoscaling) and incremental classification using change-data markers.- Cache frequently-evaluated policies in distributed cache (Redis) at enforcement gateways.- Delegate owner approvals via federated workflows; provide SDKs and CI checks so teams embed classification in pipelines (shift-left).

Trade-offs & considerations:- ML classifiers reduce manual effort but need feedback loops to avoid false negatives; implement owner review and periodic reclassification.- Masking vs. access: masking preserves utility but may limit advanced modeling—enable controlled secure enclaves for vetted use-cases.- Performance: query-rewrite/enforcement introduces latency; mitigate with pushdown filters and materialized masked views for heavy workloads.

Key metrics:- Time-to-classify new dataset, percent datasets with tags, time-to-grant access, number of policy violations, audit coverage, false-positive/negative rates for PII detection.

This design balances self-serve productivity with tight, scalable PII protection by combining automated discovery, tag-based policies, approval workflows, enforcement at multiple layers, and comprehensive auditing.

Unlock Full Question Bank

Get access to hundreds of Ownership and Project Delivery interview questions and detailed answers.

Join thousands of developers preparing for their dream job.