Hadoop Ecosystem & Related Tools Questions

Overview of the Hadoop ecosystem components (e.g., HDFS, MapReduce, YARN) and related tools (Hive, Pig, HBase, Sqoop, Flume, Oozie, Hue, etc.). Covers batch and streaming data processing, data ingestion and ETL pipelines, data warehousing in Hadoop, and operational considerations for deploying and managing Hadoop-based data pipelines in modern data architectures.

MediumTechnical

40 practiced

A MapReduce job processing 100GB of logs is slower than expected. List and explain concrete configuration and code-level optimizations you would apply to reduce runtime and resource usage. Consider mappers/reducers, combiners, compression for shuffle and outputs, IO formats, speculative execution, and memory settings.

Sample Answer

Situation: A 100GB MapReduce job is slower than expected. Below are concrete configuration and code-level optimizations I’d apply, why they help, and example settings.

1) Right-size mappers & reducers- Reduce/Increase reducers based on job: controls shuffle parallelism. Start with reducers = max(1, min(500, totalInputSize / 256MB)).- Use mapreduce.job.reduces to tune; avoid single reducer hotspot.

2) Use a Combiner- Implement a combiner that mirrors reducer logic to reduce shuffle volume (e.g., sum/count). Register with job.setCombinerClass(MyReducer.class).- Reduces network I/O and reducer load.

3) Compression (shuffle + output + input)- Enable intermediate map output compression to shrink shuffle: mapreduce.map.output.compress=true mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec- Compress final outputs with a splittable codec (e.g., Snappy with container formats like Parquet/Avro or LZO with index).- Use codecs with low CPU (Snappy, LZ4) for faster throughput.

4) Optimize IO formats & storage layout- Use binary compact formats (Parquet/Avro) rather than text logs; support predicate pushdown and column pruning.- If logs are text, use SequenceFiles with compression to speed reads.

5) Split sizing & input format- Tune input split size to balance parallelism: mapreduce.input.fileinputformat.split.maxsize and split.minsize to avoid too many tiny maps or too few huge maps.- Implement a custom InputFormat if parsing expensive.

6) Memory & sort buffer tuning- Increase mapreduce.task.io.sort.mb (e.g., 256-512MB) to reduce spills.- Set mapreduce.map.memory.mb and mapreduce.reduce.memory.mb to avoid OOMs; tune mapreduce.map.java.opts to -Xmx appropriately (e.g., 0.8 * container memory).- Increase mapreduce.reduce.shuffle.parallelcopies (default 5) if network allows.

7) Speculative execution- Disable speculative execution for reducers if skew causes duplicate work: mapreduce.map.speculative=false, mapreduce.reduce.speculative=false. Alternatively enable only for maps to handle slow nodes.

8) Partitioning & skew handling- Implement custom Partitioner to evenly distribute keys.- Handle heavy keys with “hot key” mitigation: salt keys or use two-phase aggregation (pre-aggregate in mapper/combiner then final reduce).

9) Code-level improvements- Avoid expensive object creation in map()/reduce(); reuse objects, use primitive counters where possible.- Stream parsing (avoid building huge in-memory structures).- Use in-mapper combining for cheap pre-aggregation when safe.

10) Locality & cluster considerations- Ensure HDFS block locality; increase dfs.blocksize (e.g., 256MB) for larger sequential reads.- Monitor network/disk IO and GC; adjust JVM opts to reduce GC pauses (CMS/G1 flags).

11) Monitoring & iterative tuning- Use job counters, logs, and ResourceManager UI to identify bottlenecks (spill counts, shuffle bytes, GC time, task failures).- Iterate: profile, change one setting at a time, measure improvements.

Result: These actions reduce network shuffle, minimize spills, balance task load, and cut CPU/IO overhead—often turning a slow job into one that completes significantly faster with lower resource use.

HardTechnical

75 practiced

Design an approach to achieve exactly-once semantics when writing streaming stateful results from Spark Structured Streaming to HBase. Cover checkpointing, idempotent writes, write-ahead logs or Kafka transactions, how to handle job restarts, and what guarantees are realistic in the presence of network partitions.

Sample Answer

Approach summary- Achieve practical exactly-once by combining Spark Structured Streaming checkpointing, an external commit protocol (per-batch atomic marker), and idempotent HBase writes (conditional upserts keyed by primary key + batch id). Use a write-ahead staging or Kafka transactions when possible to make commits atomic end-to-end.

Detailed design1. Checkpointing (Spark)- Use Spark Structured Streaming with a stable checkpointLocation so Spark persists offsets and progress. This ensures deterministic replay of micro-batches after failures.- Use foreachBatch rather than unsupported custom sinks: foreachBatch gives batchId and deterministic retry semantics.

2. Per-batch atomic commit protocol- For each micro-batch (given batchId): a. Stage writes into a durable WAL/staging area (HDFS/parquet) or produce to a transactional Kafka topic with producer transactions using batchId as transaction id. b. Apply idempotent upserts to HBase referencing batchId (see next). c. Once all writes succeed, record batchId as committed into a small metadata table (HBase row or ZK/DB). This is the single source of truth for "applied batches".- On restart, check committed batchId: skip re-applying batches <= committedBatchId; re-run failed batches.

3. Idempotent HBase writes- Design row key such that re-applying same logical record overwrites deterministically (Put with same rowKey and same value), or use a column storing lastAppliedBatchId and only perform Put if incoming batchId > stored batchId using CheckAndPut/CheckAndMutate for atomic compare-and-set.- Alternative: write each record with composite key (businessKey + batchId) and run a compaction/aggregation periodically to collapse duplicates.

4. Write-ahead log vs Kafka transactions- WAL approach: before applying to HBase, write the batch payload to HDFS (WAL). Apply HBase from WAL; mark committed only after HBase writes succeed. WAL provides crash-safe replay.- Kafka txns approach (if Kafka is source or broker available): use Kafka producer transactions: write results into a transactional topic and include offsets; commit the transaction only after HBase commit; or the reverse: read from Kafka with exactly-once semantics and use an external coordinator to atomically commit both Kafka offsets and HBase batch commit (two-phase commit emulated).

5. Handling job restarts- Spark replays micro-batches deterministically based on checkpoint offsets.- On startup, compare incoming batchId to committedBatchId. If <= committed, skip. If > committed but you find staged WAL for that batch, re-run apply step idempotently.- Ensure commit of batchId is the final atomic step; crash between apply and commit must be handled by replaying apply which is idempotent.

6. Realistic guarantees under network partitions- HBase itself provides single-row atomicity but no distributed transactions: you can ensure exactly-once per-row using conditional writes, but cross-row multi-row transactional semantics are not guaranteed.- Under network partitions you face CAP trade-offs: you can guarantee consistency by blocking (unavailable) or allow writes that may later be retried/duplicated. Realistic promise: "exactly-once" per logical key (idempotent upserts) and "at-least-once" across the cluster for multi-row atomicity unless you implement additional coordination (external transaction manager).- Use retries with backoff and monitoring; accept that in severe partitions either writes will fail (preferable) or duplicates may appear and must be cleaned via idempotency/compaction.

Practical recommendations- Implement batchId-based conditional writes (CheckAndPut) in HBase for per-key exactly-once.- Persist commit markers in durable store (HBase meta table or ZK).- Use WAL staging for durability if Kafka transactions unavailable.- Add monitoring and periodic reconciliation jobs to detect inconsistencies and repair using WAL.- Document failure modes: exactly-once for individual keys; multi-row atomicity only achievable with extra coordination and will trade availability during partitions.

MediumSystem Design

51 practiced

You run a mixed workload YARN cluster. How would you configure CapacityScheduler queues and resource guarantees to ensure critical nightly ETL jobs always finish on time while allowing ad-hoc interactive queries during the day? Discuss queue configuration, max-capacity, preemption, and fair-share trade-offs.

Sample Answer

Requirements & constraints:- Nightly ETL jobs (critical) must finish by SLA window (e.g., 2–6am).- Daytime interactive/ad‑hoc queries should be responsive but best-effort.- Multi-tenant cluster with mixed Spark/Hadoop workloads.

High-level approach:- Create two top-level CapacityScheduler queues: etl (guaranteed) and interactive (elastic).- Use subqueues if needed per team (etl.prod, etl.nonprod, interactive.data-science).

Queue configuration:- etl queue: capacity = 30–50% (guaranteed share), maxCapacity = 100% (allow use of idle resources), a high priority for applications, ACLs limited to ETL service accounts.- interactive queue: capacity = 50–70%, maxCapacity = 100%, allow many small, preemptible containers, enable user-level subqueues to isolate noisy users.

Preemption & reservations:- Enable preemption for the etl queue with a conservative reclaim timeout (e.g., preemptionTimeoutMs = 5–10m) so long-running interactive queries can finish short tasks or checkpoint.- Use the CapacityScheduler’s application-level preemption (kill containers) only as last resort; prefer yarn reservations to reserve resources ahead of nightly window.- Schedule reservations for ETL time window: create a Reservation System plan so resources are guaranteed at reservation start.

Fair-share trade-offs:- Capacity guarantees ensure ETL completes; but strict capacities reduce daytime responsiveness. Setting high maxCapacity to 100% lets interactive use idle hours.- Preemption provides guarantees but can disrupt interactive users—mitigate by: - Grace periods and notifications - Prefer preemption of large, long-running containers (e.g., big Spark executors) and use graceful executor decommissioning (Spark dynamic allocation + graceful shutdown)- Consider fair-share within interactive queue (queue-level fair share) to avoid starvation between users.

Operational practices:- Monitor queue usage, tail latency of ETL, and preemption events; tune capacities seasonally.- Use workload-aware autoscaling (add/remove nodes) or cloud spot/ephemeral nodes for daytime elasticity.- Add admission control (max running apps per user, queue) and set application prioritization for critical ETL jobs.

Why this works:- Guarantees via capacity/reservations protect SLAs.- High maxCapacity + elastic use maximizes utilization.- Controlled preemption and graceful decommissioning balance SLA enforcement with user experience.

HardTechnical

42 practiced

Propose a 3-year cost model comparing on-prem Hadoop cluster vs cloud-based lake (S3 + EMR). Include CAPEX/OPEX breakdown, storage and compute costs, data egress, staffing, licensing, and operational risk. Explain which metrics and sensitivity analyses you would present to finance and leadership to support the recommendation.

Sample Answer

Approach/framework:- Build a 3-year total cost of ownership (TCO) model comparing On‑Prem Hadoop vs Cloud Lake (S3 + EMR). Model annualized CAPEX, recurring OPEX, per-GB storage/compute, data transfer, staffing, licensing, and risk/contingency. Present base, best, worst-case scenarios and sensitivity to key drivers.

Assumptions (example, make editable):- Data size today: 200 TB, growth 50%/yr → Year3 ~675 TB- On‑prem hardware life: 4 years; procurement lead 3 months- On‑prem cluster: 50 nodes (32 vCPU, 256 GB RAM), usable storage after replication/RAID: 300 TB- Cloud: S3 storage, EMR serverless/spot for compute- Compute workload: 5,000 node-hours/month equivalent- Staff: 2 sysadmins (on‑prem), 1 cloud engineer + 1 data platform engineer (cloud)- Discount rate: 8%

On‑Prem CAPEX:- HW purchase (servers, networking, racks, power, cooling): $1.2M upfront (annualized over 4 yrs ≈ $300k/yr)- SAN/NAS or HDFS replication overhead storage factor 1.5 → additional capacity cost included- Software licenses/support (Cloudera/HDP legacy): $120k/yr- Data center space, power & cooling: $60k/yr

On‑Prem OPEX:- Staffing: 2 admins + 0.5 SRE (loaded) ≈ $300k/yr- Maintenance, spare parts, upgrades: $50k/yr- Backup/DR, networking ops: $40k/yr- Security/compliance overhead: $30k/yrTotal On‑Prem approx Year1: ~$900k; growth drives incremental hardware refresh in Year2/3

Cloud (S3 + EMR) CAPEX:- Minimal — tooling/setup & migration project: $150k (Year0/1)

Cloud OPEX:- S3 storage: $0.023/GB-month → at avg stored TBs across year (~350 TB avg Year1) ≈ $96k/yr growing to ~$185k by Year3- EMR compute: spot/on‑demand mix; assume effective $0.03 per vCPU‑hour → for 5,000 node‑hrs/month with 32 vCPU ≈ 5k*32*24*30*0.03 ≈ $3.5M/yr (use autoscaling, job batching, spot to reduce to ~$700k/yr)- Data egress: 5 TB/month external egress @ $0.09/GB ≈ $5k/yr; inter-AZ negligible- Managed services (Glue, Athena, monitoring): $80k/yr- Staffing: 2 cloud engineers / platform ≈ $260k/yrTotal Cloud approx Year1: ~$1.2M (but leverages elasticity—costs align with usage; Year2/3 grows with data but can be optimized)

Operational risk & qualitative factors:- On‑prem: higher upfront risk (procurement delays), single-site failure risk unless DR invested, slower scale up, heavy ops burden.- Cloud: vendor lock-in, data egress cost risk, security/compliance considerations, but higher agility, faster time-to-insight, and better resilience.

Metrics and sensitivity analyses to present:- 3-year NPV and payback period- Yearly and cumulative TCO broken into CAPEX vs OPEX- Cost per TB stored, cost per node-hour, and cost per ETL job- Sensitivity to: data growth rate (±20–50%), compute hours (±30%), spot availability (affects EMR cost), egress volume (±50%), discount rate (±2–5%)- Break-even analysis: at what sustained monthly compute/storage does On‑Prem become cheaper?- Risk‑adjusted costs: include probability-weighted contingencies for hardware failure, migration delay, compliance fines

Recommendation summary (example):- If compute demand is spiky and growth high, Cloud lake wins on agility, predictable ops and lower time-to-value; present optimized cloud cost using spot instances, data tiering (S3 IA/Glacier), and job scheduling to reduce EMR cost.- If organization has stable, very high sustained compute and strict data residency/cost sensitivity, on‑prem may be competitive after Year2 — show the break-even curve.- Deliver model as parameterized spreadsheet so finance can see impacts of changing assumptions; include recommended governance: tagging, cost allocation, reserved capacity commitments if cloud chosen, and a migration/rollback plan to mitigate operational risk.

EasyTechnical

43 practiced

Describe how HDFS NameNode stores filesystem metadata using fsimage and edits. Explain the purpose of checkpointing and what SecondaryNameNode (or CheckpointNode) does in classic Hadoop deployments. Also describe how this changed with JournalNodes and NameNode HA.

Unlock Full Question Bank

Get access to hundreds of Hadoop Ecosystem & Related Tools interview questions and detailed answers.

Join thousands of developers preparing for their dream job.