Assess a candidate's practical and conceptual understanding of technology stacks, including major programming languages, application frameworks, databases, infrastructure, and supporting tools. Candidates should be able to explain common use cases and trade offs for languages such as Python, Java, Go, Rust, C plus plus, and JavaScript, including differences between compiled and interpreted languages, static and dynamic type systems, and performance characteristics. They should discuss application frameworks and libraries for frontend and backend development, common web stacks, service architectures such as monoliths and microservices, and application programming interfaces. Evaluate understanding of data storage options and trade offs between relational and non relational databases and the role of structured query language. Candidates should be familiar with cloud platforms such as Amazon Web Services, Google Cloud Platform, and Microsoft Azure, infrastructure components including containerization and orchestration tools such as Docker and Kubernetes, and development workflows including version control, continuous integration and continuous delivery pipelines, testing frameworks, automation, and infrastructure as code. Assess operational concerns such as logging, monitoring and observability, deployment strategies, scalability, reliability, fault tolerance, security considerations, and common failure modes and mitigations. Interviewers may probe both awareness of specific tools and the candidate's depth of hands on experience, ability to justify technology choices by evaluating trade offs, constraints, and risk, and willingness and ability to learn and evaluate new technologies rather than claiming mastery of everything.
HardTechnical
75 practiced
A production ML feature used by models drops in accuracy by 20% immediately after a pipeline change. Walk through a debugging playbook across ingestion, transformation, joins, aggregation, and downstream feature consumption. Include how to use sampling, lineage, unit tests, shadow/canary runs, backfills, and rollback strategies to isolate and fix the issue and prevent recurrence.
Sample Answer
Situation: A production ML feature suddenly causes model accuracy to drop 20% right after a pipeline change. Below is an actionable debugging playbook (ingestion → transformation → joins → aggregation → consumption) plus isolation, fix, rollback, and prevention steps tailored for a Data Engineer.1) Triage & scope- Confirm incident window (timestamp of pipeline change) and affected model(s)/deployments using monitoring alerts and model telemetry.- Record feature lineage to identify all upstream jobs, tables, sources touched by the change.2) Ingestion checks- Sample raw input for the impacted timeframe vs. baseline: schema, null rates, value distributions, event counts, late/duplicate records.- Quick checks: file counts, broker lags (Kafka), checksum/hash comparisons, and ingestion job logs for errors.3) Transformation checks- Replay transformation step on sampled raw data for both pre-change and post-change code paths.- Compare intermediate columns and deterministic hashes. Run unit tests for transformation functions.- Look for off-by-one, timezone, casting, rounding, or imputation behavior differences.4) Joins and keys- Validate join keys cardinality and foreign-key integrity (missing keys, many-to-many explosion).- Run small-sample LEFT/ANTI joins to detect unexpected row loss/duplication.- Examine join order/partitioning changes that could change shuffle behavior.5) Aggregation checks- Recompute aggregations on sample partitions; compare window boundaries, GROUP BY semantics, late-event handling, watermark changes.- Confirm consistent bucketing/hashing if feature store uses partitioned files.6) Downstream feature consumption- Verify serialization, type casting, feature encoding, and normalization applied at serving-time. Compare feature vectors used for training vs. inference.- Shadow-run the new pipeline to generate features into a parallel feature-store path and run the model offline to measure accuracy before full rollout.7) Isolation methods- Canary / shadow runs: Feed production traffic to both old and new pipelines in parallel. Compute model metrics for both.- Unit and integration tests: Add regression tests asserting statistical invariants (means, missing-rate thresholds, unique-counts).- Sampling: Use stratified sampling across users/time to focus expensive checks.8) Remediation & rollback- If canary shows regression, immediately switch inference to previous stable feature path (rollback). Use feature-store versioning to restore previous feature artifact.- Patch: fix transformation/join/aggregation bug, run targeted backfill over affected time window on a staging feature-store, validate with shadow model evaluation.- Backfill strategy: perform idempotent, partitioned backfill; validate checksums and statistics per partition before promoting.9) Prevent recurrence- Enforce CI: unit tests for transforms, integration tests for joins/aggregations, and data contracts (schema + distribution checks).- Automated lineage + drift alerts: detect schema, cardinality, and distribution shifts with thresholds.- Feature-store versioning & immutability: require canary approvals and automated shadow-run pass before switching default.- Run periodic replay tests and keep small rollback window with automated feature rollback playbooks.Result-oriented checklist (quick):- Map lineage → run targeted samples → compare hashes/statistics → shadow-run → rollback if regression → backfill + validate → promote.This approach isolates root cause quickly, minimizes user-impact, and builds safeguards to prevent similar regressions.
HardTechnical
84 practiced
Design an S3-backed data lake partitioning and compaction strategy for hundreds of terabytes of Parquet files queried by Athena and Spark. Cover partition key selection, directory layout conventions, compaction (small-file mitigation), scheduling compaction jobs, Glue/Hive catalog management, and techniques to reduce query scan costs and latency.
Sample Answer
Requirements & constraints:- Hundreds of TB of Parquet on S3, read by Athena (serverless SQL) and Spark (ETL/analytics).- Minimize query scan cost and latency; avoid many small files; support incremental writes and time-series queries.Partition key selection:- Primary: event_date (date, yyyy=,mm=,dd=) — most queries filter by time.- Secondary / optional: country or data_source for high-cardinality filters only when used frequently.- Avoid high-cardinality columns (user_id) as partitions — use them as clustered/sort keys in Parquet.Directory layout conventions:- s3://bucket/datalake/<domain>/table/event_date=YYYY-MM-DD/country=US/<dataset_files>.parquet- Use stable prefixes (avoid random UUID paths) to allow efficient prefix listing.- Include _SUCCESS markers and manifest files for snapshot consistency.Compaction (small-file mitigation):- Use Spark jobs to coalesce small Parquet files into larger files ~256MB (tunable).- Maintain partition-level compaction: compact within a partition to avoid rewriting whole table.- Write atomically: write to a temp prefix, validate, then move/rename via S3 object copy+delete or use EMRFS S3-optimized atomic rename pattern.Scheduling compaction jobs:- Two-tier approach: - Nearline micro-compaction: run continuously or hourly for recent partitions (merge files <32MB). - Offline full compaction: daily/weekly batch for older partitions (merge to ~256MB).- Trigger compaction by thresholds: number of files per partition, total partition size, or file-size histogram.- Use AWS Step Functions or Airflow to orchestrate; autoscale Spark cluster (EMR or EKS).Glue / Hive catalog management:- Use AWS Glue Catalog as single source of truth.- Register partitions programmatically: partition discovery after compaction via MSCK REPAIR or Glue API to add partitions (avoid full scans).- Store partition metadata and table statistics (column-level stats) to enable predicate pushdown and cost-based optimizations.- Versioned schema evolution: use schema registry or store schema in Glue table properties.Techniques to reduce scan cost & latency:- Predicate pushdown: ensure queries filter on partition columns first.- Parquet column pruning: include only needed columns; encourage SELECT column_list.- Sort/cluster data within Parquet files (Z-order or sort by high-selectivity cols) to reduce IO.- Use file-level statistics & min/max values (write stats during compaction) so Athena/Spark skip row groups.- Use Glue table parameters to set splittable and compression settings (snappy).- Maintain per-partition manifests for Athena to avoid listing thousands of small files.- Caching: use Athena workgroup query result caching and Spark caching for hot datasets.Trade-offs & monitoring:- Larger files reduce metadata overhead but increase rewrite cost for updates.- Monitor: file-size distribution, partition count, query bytes scanned, job runtimes. Alert when thresholds exceeded.- Start with date-based partitioning + periodic compaction; adjust secondary partitions and file-size target based on query patterns and cost metrics.
HardTechnical
84 practiced
Compare Terraform and Pulumi for managing a complex, multi-account cloud environment supporting a data platform. Discuss testing and CI integration, language expressiveness and team ramp-up, secrets handling, drift detection, community modules, and multi-cloud support. Recommend which tool you'd pick and why for a team that values testability and code reuse.
Sample Answer
High level: both Terraform and Pulumi are IaC for multi-account, multi-cloud environments, but they trade-off declarative HCL + huge ecosystem (Terraform) vs. imperative/general-purpose languages + richer abstraction/testability (Pulumi). For a data platform where many accounts, complex networking, and repeatable reusable patterns matter, here's a focused comparison.Testing & CI- Terraform: strong ecosystem for integration testing (Terratest in Go), acceptance tests via terraform plan/apply in CI, and policy-as-code with Sentinel/OPA. Terraform Cloud/Enterprise provide state locking, remote runs, and policy checks.- Pulumi: first-class unit testing using your language's testing frameworks (pytest/jest/go test), mocks for cloud providers, and Automation API for headless runs in CI. Easier to unit-test logic and small components.Language expressiveness & ramp-up- Terraform (HCL): declarative and predictable; modules are simple but HCL is limited for complex logic (workarounds with for_each, count, or external data). Faster for teams used to DevOps workflows.- Pulumi: use Python/TypeScript/Go/C#: full language features, sophisticated abstractions, libraries, and standard testing—steeper ramp if team lacks software-engineering background but excellent for data engineers comfortable in Python.Secrets handling- Terraform: supports remote state encryption and integrations (Vault, AWS SSM/Secrets Manager). Providers/plugins exist; must avoid baking secrets in state/output.- Pulumi: built-in secrets management with providers (KMS, Vault) and encrypted state; secrets can be typed and handled in program code more naturally.Drift detection- Terraform: plan shows drift; state vs. real world; Terraform Cloud offers drift detection and notifications.- Pulumi: preview detects drift; Pulumi service can show diffs; imperative resources can make certain drifts less predictable if side effects occur.Community modules & ecosystem- Terraform Registry: massive, vetted modules for networking, IAM, managed services—huge time saver.- Pulumi Registry: growing, many libraries but smaller surface; for niche cloud services Terraform often has earlier community coverage.Multi-cloud & multi-account- Terraform: battle-tested for large, complex multi-account setups; robust backends, workspaces, and terragrunt/automation patterns for account bootstrapping.- Pulumi: supports multi-cloud in code, and its language abstractions make sharing patterns across clouds easier, but some providers lag in features.RecommendationFor a data engineering team that values testability and code reuse I’d pick Pulumi if the team is comfortable with a general-purpose language (especially Python) because:- Unit and integration testing is simpler and more natural (use existing test tooling, mocks).- Code reuse via packages, classes, and language-level abstraction reduces duplication across accounts and pipelines.- Secrets and CI integration are straightforward using the Automation API and native secrets support.However, if you need maximum ecosystem coverage, many vetted community modules, or your org already standardizes on Terraform (Terraform Cloud, large module library), choose Terraform and invest in Terratest + a strong module structure (and Terragrunt) to get testability and reuse.Implementation note: either way, use remote state with locking, enforce policy-as-code, create reusable account-bootstrapping modules/packages, and add CI gates (plan/preview approval, automated unit tests, and drift monitoring).
EasyTechnical
76 practiced
Explain differences between relational (row-based) databases and non-relational stores (key-value, document, wide-column) and when to prefer columnar analytical stores. Relate to OLTP vs OLAP workloads, transactionality, query patterns, indexing, and when a data engineer should choose NoSQL for parts of a pipeline.
Sample Answer
Relational (row-based) databases vs non-relational stores — core differences and when to use columnar analytical storesDefinition and data model:- Relational (row-based RDBMS): fixed schema, normalized tables, rows represent full entities. Strong ACID transactions and SQL.- Non-relational (NoSQL): - Key-value: simple get/put by key, ultra-fast lookups. - Document: semi-structured JSON-like documents (MongoDB), flexible schema, good for nested data. - Wide-column: sparse, column-family oriented (Cassandra, HBase), designed for massive writes and partitioned access.OLTP vs OLAP:- OLTP (transactional systems): many small, concurrent reads/writes, low latency, need strong consistency and transactions → relational DBs are typical.- OLAP (analytical systems): large scans, aggregations, read-heavy, columnar access patterns → columnar analytical stores (e.g., Snowflake, BigQuery, ClickHouse) excel.Transactionality & consistency:- RDBMS: ACID by default; good for money transfers, inventory.- NoSQL: often offer eventual consistency or tunable consistency; some provide transactions (document DBs), but generally weaker than RDBMS.Query patterns & indexing:- Row-based RDBMS: index on columns to speed selective queries; efficient when retrieving full rows.- Columnar stores: store columns together enabling vectorized scans and compression — excellent for aggregation over few columns across many rows.- Key-value: limited query patterns (key lookups); add secondary indexes at cost.- Document/wide-column: support queries on fields/column-families but less flexible than SQL joins.When a data engineer should choose NoSQL in a pipeline:- High write throughput, massive scale, and schema flexibility are required (telemetry, event ingestion).- Low-latency key-based lookups (session stores, caches).- When denormalization and embedding reduce joins and simplify upstream processing.- For componentized microservices where ownership of data and elastic scaling matter.- Use NoSQL as landing or operational store; then ETL into a columnar OLAP warehouse for analytics.When prefer columnar analytical stores:- Batch analytics, BI dashboards, time-series aggregations, and ad-hoc SQL over large datasets where scanning and compression speed matter.- Combine with an ETL pipeline that transforms raw, event-driven NoSQL data into curated, columnar tables for reporting.Trade-offs summary:- Choose RDBMS for transactional integrity and complex relational queries.- Choose NoSQL for scale, flexibility, and simple access patterns.- Use columnar OLAP stores for analytics and aggregation performance; move data via ETL/CDC from operational stores.
HardTechnical
84 practiced
Describe how you would secure a Kafka-based streaming platform to meet SOC2 or similar compliance requirements. Cover encryption in transit and at rest, authentication mechanisms (SASL, mTLS), authorization (ACLs, RBAC), audit logging, key rotation, network segmentation, and operational processes for security patches and incident response.
Sample Answer
Requirements:- Protect customer data to SOC2 standards: confidentiality, integrity, availability, strong audit trails, least privilege, timely patching and incident response.- Support multi-tenant producers/consumers, high throughput, low latency.Encryption- In-transit: enable TLS for all broker, client, inter-broker, controller, and REST endpoints. Require TLS 1.2+ and strong ciphers; disable insecure protocols.- At-rest: encrypt topic data and logs using disk-level encryption (LUKS/KMS) or filesystem encryption provided by cloud (EBS/EFS with KMS). Ensure backups and snapshots are encrypted.Authentication- Use mutual TLS (mTLS) for broker-to-broker and option for client mTLS where X.509 identity is required.- For application simplicity and wide client support, enable SASL mechanisms (SCRAM-SHA-256/512 or OAUTHBEARER for OIDC integration). Prefer OIDC/SAML-backed tokens for short-lived credentials and centralized identity (Okta/Azure AD).Authorization- Enable Kafka ACLs (authorizer.class.name = kafka.security.authorizer.AclAuthorizer) for topic/consumer-group-level controls; enforce least privilege.- For larger orgs, implement RBAC mapped from central identity (via OIDC groups → Kafka ACL provisioning) and manage via automated infra-as-code (Terraform/Ansible) to avoid manual errors.Audit logging- Enable broker audit logs for authentication, authorization failures, config changes, and ACL modifications. Ship logs to immutable, centralized SIEM (Splunk/ELK/Cloud SIEM) with retention policy per SOC2.- Capture client access events (consumer offsets, producer writes) and correlate with metadata.Key management & rotation- Use centralized KMS (AWS KMS, Cloud HSM) for CA and data encryption keys. Maintain short-lived certs where possible; automate certificate issuance and rotation (Vault/ACME).- Have automated key/cert rotation pipelines: generate -> distribute -> roll certificates across brokers and clients with zero-downtime strategies (graceful restart and dual-key acceptance window).- Rotate KMS data keys periodically and re-encrypt archives as needed.Network segmentation- Place brokers in private subnets, limit management ports to secured jump hosts or bastions. Use VPC service controls, network ACLs, and security groups to restrict producer/consumer access.- Isolate control plane (Zookeeper/KRaft) from data plane; use private peering or service endpoints for cross-account access. Enforce TLS & mTLS at load balancers/gateways.Operational processes- Patching: maintain an image pipeline for brokers, test in staging, and perform rolling upgrades with health checks. Track CVEs, prioritize critical patches within SLA (e.g., 30 days for critical).- Configuration drift: enforce via IaC and GitOps; run compliance checks (infrastructure scanning).- Monitoring: broker metrics, authorization failures, unusual throughput spikes; alerting thresholds to SOC2 requirements.- Incident response: documented runbooks for auth compromise, data exfiltration, and broker compromise. Tabletop exercises quarterly; chain-of-custody for forensic logs; communication plan and SLA for breach notification.- Change control & approvals: audited PRs for ACL/config changes, scheduled maintenance windows, and separation of duties.Additional controls- Limit sensitive data in topics (PII masking/encryption at application level), enforce schema registry and compatibility, and use topic-level retention & compaction policies to limit exposure.Result: These combined controls (crypto, identity, least privilege, immutable audit logs, network isolation, automated rotation and patching, and tested IR processes) align Kafka operations with SOC2 controls while preserving throughput and availability.
Unlock Full Question Bank
Get access to hundreds of Technology Stack Knowledge interview questions and detailed answers.
