Junior Data Engineer Interview Preparation Guide - FAANG Standards

Decision process overview:

• Quantify baseline demand: measure average and peak vCPU, memory, storage, and runtime for your long-running ETL clusters over 3–12 months. Separate steady-state (baseline) vs burst workload.
• Cost vs flexibility trade-off: reserved instances / committed-use discounts (CUDs) buy capacity at lower cost for predictable baseline; on‑demand gives maximum flexibility for unpredictable needs; spot/preemptible give lowest cost but can be terminated.

When to use reserved/CUDs:

• Use reserved/CUDs to cover 60–90% of predictable, long-running capacity (steady ETL pipelines, always-on schedulers, critical ingestion). They reduce cost (30–70%) and simplify capacity planning.
• Choose contract length and convertible vs standard based on expected growth and workload constraints.

When to use on‑demand:

• Use on‑demand for unpredictable spikes, experiments, short-lived bursts, or when instance interruption is unacceptable and you need rapid elasticity.

When to use spot/preemptible:

• Appropriate for fault-tolerant, stateless, or checkpointed workloads: batch ETL stages that can restart, speculative processing, re-processing non-critical datasets, or worker nodes behind durable storage. Yield large savings (up to 80%).
• Not appropriate for critical single-node services, low-latency jobs that can’t tolerate interruptions, or pipelines without fast restart/ checkpointing.

Risk mitigation patterns for spots:

• Checkpointing and idempotent processing (persist intermediate state to durable storage).
• Hybrid cluster: run master/coordinator on reserved/on‑demand nodes; worker executors on spot instances.
• Autoscaling groups with mixed instance policies and capacity-optimized allocation.
• Graceful termination handling (capture termination notice, flush in-flight work).
• Retry/backoff strategies and queue-based task distribution (e.g., message queues, Kubernetes jobs).
• Priority fall-back: if spot capacity unavailable, automatically launch on‑demand nodes to meet SLA.

Scenarios:

• Large nightly ETL that reads from S3, writes to a data lake, and uses Spark: use reserved instances for driver/master and spot for executors + checkpointing + fallback to on‑demand if spot shortage threatens completion.
• Real-time ingestion pipeline with tight SLA: use reserved or on‑demand only.
• Ad-hoc reprocessing after schema change: use spot massively for cost efficiency if job can be restarted.

Summary rule: reserve capacity for predictable, critical components; use on‑demand for unpredictability or when termination risk is unacceptable; use spot when workloads are interruptible and you implement checkpointing, hybrid architecture, and graceful failover.

Direct answer

Batch and streaming are two ways of answering the same question (what does the data say right now) at different points on the latency/cost/complexity curve. Batch reads a large accumulated chunk on a schedule; streaming processes each event as it arrives. Pick batch by default and reach for streaming only when a specific business decision genuinely needs data fresher than the next scheduled run can provide.

Structured elaboration

Latency. Batch is minutes to a day (whatever the schedule is); streaming is sub-second to a few seconds, bounded mainly by how long you wait for late data (the watermark).

Throughput. Batch amortizes overhead across a huge chunk, so it is usually the cheaper way to move the same total volume of data. Streaming pays a small per-event tax (serialization, network hops, state lookups) on every single record, so the same total throughput costs more compute.

Cost. A batch job runs, finishes, and releases its compute. A streaming job holds compute (and often memory-resident state) 24/7 whether or not there's a burst of traffic, so streaming infrastructure has a real always-on cost floor that batch does not.

Operational complexity. Batch failure recovery is simple: rerun the job. Streaming failure recovery has to reason about partial state, checkpoints, exactly-once vs at-least-once delivery, and consumer lag, which means more moving parts, more monitoring surface, and a team that has to understand event-time semantics, not just SQL.

Correctness. Batch naturally sees all the data for a period before computing anything, so late-arriving records are simply part of the input. Streaming has to make an explicit decision about how long to wait for late data (allowed lateness) before it emits a result, which means a streaming aggregate can legitimately differ from the eventual batch recompute of the same period.

Worked example

A nightly-refreshed revenue dashboard is a clean batch case: the business consumes it once a day, a few hours of latency is invisible to the user, and a failed run just reruns. A fraud-detection system that has to block a card swipe before it completes is a clean streaming case: by the time a batch job would even start, the transaction has already succeeded or failed. A hybrid shows up constantly in practice: a company might run streaming only for the handful of metrics that trigger pages or block a transaction, and leave everything else (the other 90% of reporting) on batch, because paying the always-on cost and operational overhead of streaming for a dashboard nobody checks more than once a day has no payoff.

Trade-offs and pitfalls

The most common mistake is treating this as a technology choice instead of a latency-requirement choice: teams reach for Kafka and Flink because streaming sounds more modern, then discover they've taken on 24/7 operational burden for data nobody looks at more than once a day. The second most common mistake is the opposite: assuming batch is always simpler, when a batch job that has grown large enough to blow its nightly window is itself an operational risk. The right first question is always "what decision does this data drive, and how fresh does it actually need to be to drive that decision correctly," not "which technology is state of the art."

Direct answer. Confirm duplication is the cause (rather than a logic bug) by comparing the row count immediately after the join to the row count you'd expect from the smaller side alone, and by checking, on a small sample, whether specific keys legitimately have multiple matches on both sides; fix it either by aggregating one side down to uniqueness before the join, or by deduplicating the joined result afterward in a way that doesn't silently discard rows you actually need.

Structured elaboration. An unanticipated many-to-many relationship means a join key you expected to be unique on at least one side actually has multiple matching rows on BOTH sides for some values, which multiplies rather than merely combines: two matching rows on each side for the same key produce four joined rows, not two, which then inflates any downstream SUM or COUNT computed from that joined result. To confirm this is the cause, pick a specific key value, count its rows on each side of the join independently, and multiply those counts together; if that product matches the number of joined rows you're seeing for that key, you've confirmed genuine multiplicative duplication rather than, say, a join condition that's simply too loose.

Two realistic fixes, and they aren't interchangeable: pre-aggregate one side down to one row per join key BEFORE the join (appropriate when you only actually need a single value per key from that side, like a "most recent" or "total" per key); or deduplicate the JOINED result afterward using a window function or an explicit grouping, which is appropriate when you genuinely need attributes from multiple matching rows on the many-side and the "duplication" is actually correct given the relationship, just not what a naive downstream SUM assumed.

Worked example. A product joined to promotions where a product can have multiple active promotions and a promotion can apply to multiple products is a genuine many-to-many; a report computing "total sales per product" that naively joins in promotions and then sums a sales column will multiply each product's real sales figure by however many active promotions it happens to have, a bug that's invisible on products with exactly one promotion and only becomes obvious (and often only gets NOTICED) on products with several.

Trade-offs and pitfalls. The riskiest version of this bug is exactly the one described in the worked example: it's silently correct for the common case (one match per key) and silently wrong only for the less common case (multiple matches), which means it can ship, look fine in testing, and only surface as a real discrepancy once someone happens to look at a key with genuine multiplicity, often much later and much harder to trace back to its root cause.

Complexity

Detecting this costs a handful of targeted counting queries against a specific suspect key value, not a full reprocessing of the dataset, so confirmation is cheap even though the underlying bug can be expensive to have shipped.

Edge cases

A key with exactly one match on both sides produces exactly one joined row and looks completely correct, which is precisely why this bug tends to survive testing against a small, low-multiplicity sample and only appears once the real data includes genuine multi-match keys.

Direct answer

Sort the intervals by their start value, then sweep through them once, keeping a running "last merged interval": if the next interval's start is at or before that interval's end (an overlap, or an exact touch), extend the running interval's end to cover it; otherwise, the next interval begins a new, separate merged interval. This is O(n log n) time, dominated entirely by the initial sort, and O(n) space for the output.

Structured elaboration

Why sort by start first. Without imposing an order, a single forward pass cannot be trusted to have already seen every interval that might overlap the one currently being examined; an interval appearing later in the input array could easily start earlier than one already processed. Sorting by start guarantees that once the sweep has moved past a given point, every remaining interval starts at or after it, which is exactly what makes a single linear sweep afterward sufficient.

The merge step itself. Initialize the output with the first (now sorted) interval. For each subsequent interval [s, e], compare s against the END of the last interval currently in the output. If s <= last.end, the two intervals overlap or exactly touch, so extend the last interval's end to max(last.end, e), using max rather than just assigning e directly, because a later interval can be fully NESTED inside the one already being extended (a smaller end than the current running interval), and assigning e unconditionally would incorrectly shrink coverage that the merged interval already legitimately spans. If s > last.end, there is a genuine gap, and [s, e] starts a new entry in the output.

Complexity. The sort is O(n log n) and dominates the total cost; the sweep afterward visits each interval exactly once, O(n). The output requires O(n) space in the worst case (no intervals overlap at all, so every input interval becomes its own output entry).

Touching versus overlapping, a design decision worth naming. Whether two intervals that exactly touch (one's end equals the next one's start, such as [1,4] and [4,5]) should merge is a real semantic choice, not an automatic consequence of the algorithm. Using <= in the overlap check treats a touch as mergeable (matching how meeting-room-style problems usually intend adjacency: back-to-back meetings occupy no coverage gap), while a strict < would keep them as separate, back-to-back intervals; a senior answer should state explicitly which convention is being used rather than leave it implicit.

Worked example

def merge_intervals(intervals):
    if not intervals:
        return []
    ordered = sorted(intervals, key=lambda pair: pair[0])
    merged = [list(ordered[0])]
    for start, end in ordered[1:]:
        last = merged[-1]
        if start <= last[1]:
            last[1] = max(last[1], end)
        else:
            merged.append([start, end])
    return merged

print(merge_intervals([[1, 3], [2, 6], [8, 10], [15, 18]]))

# Independent cross-check: a genuinely different method (an event-counting sweep
# line) rather than a second copy of the same merge logic.
def sweep_line_reference(intervals):
    if not intervals:
        return []
    events = []
    for s, e in intervals:
        events.append((s, 0, 1))   # start: delta +1, processed before ends at same coord
        events.append((e, 1, -1))  # end:   delta -1
    events.sort()
    result = []
    active = 0
    run_start = None
    for pos, _, delta in events:
        if active == 0 and delta == 1:
            run_start = pos
        active += delta
        if active == 0:
            result.append([run_start, pos])
    return result

import random
random.seed(3)
hand_picked = [
    [[1, 4], [4, 5]],             # touching intervals
    [[5, 8], [1, 3]],             # unsorted input
    [[2, 5], [1, 10]],            # fully nested interval
    [[1, 2]],                     # single interval
    [],                           # empty input
    [[5, 6], [1, 2]],             # unsorted, non-overlapping
    [[1, 3], [2, 6], [8, 10], [15, 18]],
]
mismatches = 0
for tc in hand_picked:
    if merge_intervals(tc) != sweep_line_reference(tc):
        mismatches += 1
for _ in range(3000):
    n = random.randint(1, 6)
    intervals = [[a, a + random.randint(0, 5)] for a in (random.randint(0, 15) for _ in range(n))]
    if merge_intervals(intervals) != sweep_line_reference(intervals):
        mismatches += 1
print(f"7 hand-picked cases + 3000-trial random sweep (seed 3): mismatches = {mismatches}")

Output:

[[1, 6], [8, 10], [15, 18]]
7 hand-picked cases + 3000-trial random sweep (seed 3): mismatches = 0

matching the question's own example exactly: [1,3] and [2,6] overlap (2 <= 3) and merge into [1,6]; [8,10] and [15,18] each start after the previous merged interval's end, so they remain separate. This was additionally cross-checked against the independently-implemented sweep-line reference above (treating each interval as a +1/-1 event at its start/end and rebuilding runs where the event count returns to zero), which is a genuinely different method rather than a second copy of the same merge logic, on 7 hand-picked cases (touching intervals, unsorted input, a fully nested interval, a single interval, empty input, and unsorted non-overlapping intervals) plus a 3,000-trial random sweep (seed 3); the printed line confirms zero mismatches across all of it, giving real cross-validation rather than testing the code against itself.

Trade-offs and pitfalls

The most consequential bug is assigning last[1] = end directly instead of last[1] = max(last[1], end) when merging: this passes every test case where intervals only partially overlap or extend each other, and only fails on a fully nested interval, which is easy to omit from a quick manual test set and then breaks silently in production by shrinking coverage that should have been preserved. A second common bug is skipping the initial sort, either because the candidate assumes input is already given in start order (sample inputs in problem statements are often, misleadingly, already sorted) or forgets that an unsorted input makes a single forward sweep unreliable; always sort explicitly rather than relying on an assumption about input order. Finally, treat the touching-versus-overlapping boundary as a decision to state out loud (<= merges touching intervals, < does not) rather than an arbitrary implementation detail, since which one is correct depends on what the intervals represent.

Situation: As a data engineering lead planning a quarter, I balance customer-facing features (new datasets/analytics), platform improvements, and technical debt.

Framework (concise):

• Assess impact: score by business value (revenue/decision velocity), user impact (number of teams/queries affected), and risk (data quality/regulatory exposure). Use a 1–5 score for each.
• Estimate effort: quick T-shirt sizing (S/M/L) + confidence level; decompose into tickets for parallel work.
• Assign owners: pair an engineer (code/ops owner) with a stakeholder (data consumer or PM) who owns acceptance criteria and ROI tracking.
• Prioritize by value/effort ratio, critical risk first, then high-value low-effort items.
• Communication: present a prioritized roadmap with three buckets—must-fix (safety/regulatory), high-ROI refactors, and new features—plus trade-offs and timelines. For execs highlight business impact and risk; for PMs show dependencies and expected delivery dates.

Concrete example:
We have one SRE-grade pipeline refactor (reduces failures 80%) vs. building a new enrichment that unlocks a single product metric. Bandwidth allows one project. Scores: refactor: impact 4 (low failures -> fewer missed reports), risk 5, effort M; enrichment: impact 3, risk 2, effort M. I pick the refactor, assign a senior engineer + product analyst for validation, commit to a follow-up minor sprint for the enrichment, and communicate to PMs/execs that delaying the enrichment reduces short-term insight but avoids ongoing incident costs and SLA breaches.

Direct answer

A shuffle occurs whenever an operation needs rows sharing a key, or requiring a global order, to be physically co-located on the same partition: groupByKey, reduceByKey/aggregateByKey, join (unless one side is broadcast), distinct, sortByKey/orderBy, and explicit repartition/coalesce(shuffle=True). Shuffles are expensive because they combine three genuinely costly operations at once: network transfer (data physically moves between executors), serialization/deserialization (every record is serialized to write shuffle files and deserialized to read them back), and disk I/O (shuffle data is written to and read from disk, not kept purely in memory, plus a sort step on the read side for most shuffle implementations).

Structured elaboration

Operations that trigger a shuffle, organized by why. Aggregation-by-key operations (groupByKey, reduceByKey, aggregateByKey, combineByKey) need every value for the SAME key co-located to combine them. Join operations (join, cogroup) need matching keys from BOTH sides co-located, unless one side is small enough to broadcast (that alternative in depth). Set/uniqueness operations (distinct) need to compare every occurrence of a value against every other occurrence, wherever they originated. Ordering operations (sortByKey, orderBy) need a GLOBAL order across the whole dataset, which requires knowing the relative rank of every row against every other row, impossible to determine from a single partition's local view alone. Explicit repartitioning (repartition, coalesce with shuffle=True) directly and deliberately redistributes data across a new partition count.

Why each cost component is expensive, concretely.

• Network. Shuffle data crosses executor (and often physical node) boundaries; network bandwidth, even on fast cluster interconnects, is typically the slowest hop in the entire read-transform-write pipeline compared to local memory or even local disk access.
• Serialization. Every record written to a shuffle file is serialized (converted to bytes) on the map side and deserialized on the reduce side; this is real, non-trivial CPU cost per record, which is exactly why serializer choice (Kryo versus Java) directly affects shuffle-heavy job performance.
• Disk spill. Shuffle write output is written to LOCAL disk (not kept purely in executor memory) specifically so it can survive the writing executor's later removal, and the reduce side often needs an additional SORT step (for shuffle implementations requiring sorted output, relevant to the HashAggregate-vs-SortAggregate distinction) which itself may spill to disk if the data does not fit in the allocated sort buffer.

Spark UI / metric fields confirming a shuffle-heavy stage. In the Stages tab, per-stage summary metrics directly expose: Shuffle Read Size / Records and Shuffle Write Size / Records (non-trivial, non-zero values here directly confirm the stage involves a shuffle at all, and their magnitude relative to the stage's INPUT size indicates how much of the stage's total cost is shuffle-related versus pure computation); Shuffle Spill (Memory) and Shuffle Spill (Disk) (non-zero values here specifically indicate the shuffle's working set exceeded available execution memory and had to spill, a stronger signal of a shuffle-cost problem, not just a shuffle's PRESENCE); the query plan's Exchange node (visible via explain(), the direct correspondence between Exchange nodes and wide dependencies/stage boundaries) confirms exactly where in the LOGICAL pipeline a shuffle was inserted, complementing the Stages tab's RUNTIME view of shuffle cost.

Worked example

df.groupBy("customer_id").agg(F.sum("amount")).explain()

The physical plan shows an Exchange hashpartitioning(customer_id, 200) node between the scan and the final HashAggregate, confirming a shuffle occurs at exactly the groupBy step (the wide dependency), partitioned into 200 partitions (Spark's shuffle-partitions default, unless overridden). Running the equivalent job and inspecting the Stages tab afterward would show a Shuffle Write metric on the map-side (partial-aggregate) stage matching roughly the post-partial-aggregation data volume, and a Shuffle Read metric on the reduce-side (final-aggregate) stage matching that same volume, the two numbers that should roughly agree with each other (what one stage writes as shuffle output, the next stage reads as shuffle input) and, together, are the direct evidence "this stage is shuffle-heavy" rather than an assumption.

Trade-offs and pitfalls

• The 3 concrete costs of a shuffle, stated together as a checklist worth memorizing: (1) network transfer of the shuffled bytes between executors, (2) serialization/deserialization CPU cost on both the write and read side, (3) disk I/O for the shuffle files themselves plus any spill during the reduce-side sort/combine step; all three compound, which is why a shuffle is categorically more expensive than a narrow transformation touching the identical VOLUME of data, not merely "somewhat slower."
• Common mistake: treating "shuffle occurred" (a binary fact, confirmable via the plan's Exchange node alone) and "shuffle is a performance problem for THIS job" (a magnitude question, requiring the Stages tab's actual size/spill metrics) as the same finding; many correct, necessary shuffles are NOT the bottleneck for a given job, and chasing shuffle elimination for its own sake, on a shuffle that is not actually costly relative to the rest of the job, is wasted effort.
• Common mistake: checking only Shuffle Read/Write SIZE and missing the Spill metrics specifically; a stage can have a large but well-provisioned shuffle (fits comfortably in execution memory, no spill) that performs fine, while a SMALLER shuffle that spills heavily (undersized memory relative to that specific stage's needs) can be the actual bottleneck; size alone does not tell the whole story, spill does.
• Confirming a shuffle via explain()'s plan-level Exchange node and confirming its COST via the Stages tab's runtime metrics are complementary, not redundant: the plan tells you WHERE a shuffle exists in the logical pipeline, the runtime metrics tell you HOW MUCH it actually cost for a specific run against specific data.

Direct answer

The pipeline needs to treat the SFTP (SSH File Transfer Protocol) transfer, the validation, and the commit into the partitioned dataset as three distinct, separately-verifiable steps, because collapsing them (writing directly into the final partitioned location as bytes arrive) is exactly what turns a partial or corrupted transfer into bad data a downstream query can already see.

Structured elaboration

Transfer and verification

• Pull the file from SFTP into a staging location first, never directly into the final partitioned dataset, so a failed or partial transfer never becomes visible to a downstream reader.
• Verify the transfer completed correctly using a checksum: if the SFTP source can provide one (many partner feeds publish a companion checksum file), compare against it; otherwise compute your own checksum on the staged file and compare its size against what the source reports, at minimum.

Schema and checksum validation

• Validate the staged file's schema against what you expect before touching the partitioned dataset at all: column presence, types, and a row-count sanity check against recent history for this same daily drop.
• A checksum mismatch or a schema-validation failure halts the pipeline at this stage, before anything downstream is touched, rather than partially loading a suspect file.

Atomic commit

• Write the converted, partitioned Parquet output to a temporary path, then atomically move or rename it into the final partition location only once the write is fully complete and verified, so a reader never sees a partially-written partition.
• On object stores without a true atomic rename across prefixes, achieve the same effect by writing to a versioned or staged prefix and updating a pointer (a manifest or a catalog entry) only after the write finishes, rather than writing in place.

Partitioning strategy

• Partition by the file's own delivery date, since that is the natural, stable grain at which this daily file arrives, and keep the partition scheme simple and predictable so downstream consumers can reason about "yesterday's partition" without needing pipeline-internal knowledge.

Metadata-catalog updates

• Register the new partition with your catalog (Glue, Hive metastore, or equivalent) only after the atomic commit succeeds, so the catalog and the actual data on disk never disagree about what partitions exist.

Retry and backoff

• Retry a failed SFTP transfer with backoff, since transient network issues on a large 100 GB transfer are common; but do not retry a checksum or schema-validation failure blindly, since that is very likely a genuine problem with the source file that a retry will just reproduce identically.

Cost

• Compress and use a columnar format (Parquet) for the final dataset to reduce both storage cost and downstream query cost; keep the raw staged copy for a bounded retention window (long enough to support reprocessing after a bug fix) rather than indefinitely, since 100 GB/day of uncompressed staged copies adds up fast.

Worked example

A 100 GB file lands on the SFTP endpoint overnight. The transfer job pulls it into a staging bucket, verifying the transferred byte count and checksum against the source's own manifest; a mismatch (a partial transfer from a dropped connection) triggers an automatic retry of just the transfer, not the whole pipeline. Once verified, a validation pass checks the file's schema against the expected columns and confirms the row count falls within the normal range for this daily drop (catching, for example, a file that is unexpectedly only 10% of its usual size, a signal the partner's own export may have failed partway). The conversion job then writes partitioned Parquet output to a temporary prefix, and only after that write completes and is itself verified does a final atomic operation move it into the dated partition and register it with the metadata catalog, so a downstream query against "today's partition" either sees the complete, correct data or does not see the partition at all, never a partial one.

Trade-offs & pitfalls

• Writing directly into the final partitioned location "to save a copy step" is the single most common way this kind of pipeline produces a partial-partition incident; the staging-then-atomic-commit pattern costs extra storage and time but is what actually prevents that class of failure.
• Retrying a checksum failure automatically, without distinguishing it from a transient transfer failure, wastes time re-downloading a file that is going to fail the checksum identically every time if the source file itself is genuinely corrupted.
• Keeping the raw staged copy indefinitely "just in case" quietly becomes a real cost line at 100 GB/day; define and enforce a retention window deliberately rather than letting it default to forever.
• A catalog update that happens BEFORE the underlying data write is fully durable is a subtle but real bug: a reader could see a partition registered in the catalog moments before the actual files are consistently readable from the object store, depending on the store's consistency model.

Schema: sensor_readings(sensor_id, ts TIMESTAMP, value, quality, ingestion_ts, PRIMARY KEY(sensor_id, ts DESC) or use (sensor_id, ts) with clustering by sensor_id+ts DESC). Columns: sensor_id, ts, value (numeric), status/quality, tags, offset. Primary key choice: composite key with sensor_id first to enable contiguous writes and range scans per sensor. Indexing: clustered/sort key on (sensor_id, ts DESC) for fast latest-value and efficient range scans. Maintain a separate latest_values(sensor_id PK, latest_ts, value, quality) table updated via upserts or streaming to serve instant latest queries. Retention: time-based partitioning per month or per sensor bucket, with TTL job to drop/compact partitions; use downsampling aggregates (hourly/daily) stored in rollup tables. For high write throughput: use batch inserts, append-only writes, partitioning, and use LSM-based stores (Cassandra/ClickHouse/TimescaleDB) or write-optimized engines. Trade-offs: separate latest table gives O(1) reads; background retention/compaction minimizes storage.

We can implement a simple singly linked list with a Node class and a LinkedList that maintains head (and optionally tail). Below implementation keeps head and tail to make prepend O(1) and append O(1) as well; if you don't want O(1) append you can remove the tail pointer and traverse from head.

class Node:
    def __init__(self, val):
        self.val = val
        self.next = None

class LinkedList:
    def __init__(self):
        self.head = None
        self.tail = None  # keep tail for O(1) append

    def prepend(self, value):
        node = Node(value)
        node.next = self.head
        self.head = node
        if self.tail is None:      # empty list -> tail must point to new node
            self.tail = node

    def append(self, value):
        node = Node(value)
        if self.head is None:      # empty list
            self.head = node
            self.tail = node
        else:
            self.tail.next = node
            self.tail = node

    def find(self, value):
        cur = self.head
        while cur:
            if cur.val == value:
                return cur
            cur = cur.next
        return None

    def to_list(self):
        out = []
        cur = self.head
        while cur:
            out.append(cur.val)
            cur = cur.next
        return out

Example usage:

ll = LinkedList()
ll.append(10)
ll.prepend(5)
ll.append(20)
print(ll.to_list())        # [5, 10, 20]
print(ll.find(10).val)     # 10
print(ll.find(99))         # None

Notes:

• Time: prepend O(1), append O(1) with tail (O(n) if no tail), find O(n), to_list O(n).
• Space: O(n) for storage.
• Edge cases handled: empty list updates both head and tail; single-element transitions keep pointers consistent. If you remove tail maintenance, be careful to update tail when you pop the last element (not shown here).

Direct answer. Reducing data-processing compute cost by a large percentage without breaking SLAs works best as a prioritized set of levers applied roughly in order of expected-savings-per-effort, validated incrementally rather than applied all at once, so a cost regression can be attributed to a specific change and rolled back cleanly if it risks the SLA.

Structured elaboration.

1. Candidate levers, roughly ordered by typical impact-to-effort ratio. (a) Right-sizing: many clusters/instances are provisioned generously and rarely revisited -- auditing actual utilization against provisioned capacity often finds low-risk, immediate savings. (b) Partitioning: repartitioning or adding partition pruning so jobs scan only the data they actually need (by date, by key) cuts both compute time and data volume moved -- often a large win when the current layout forces full-table or full-partition scans. (c) Caching: caching a reused expensive intermediate result (a join output, an aggregation) across jobs that currently recompute it repeatedly removes duplicated compute entirely, at the cost of a cache-invalidation policy to keep it correct. (d) Storage lifecycle and format: moving cold data to cheaper tiers and adopting better compression/columnar formats reduces ongoing storage cost with minimal processing-logic risk. (e) Spot instances for tolerant workloads (instances reclaimable with short notice at a steep discount, safe for checkpointed batch work). (f) Job consolidation: multiple small, similar jobs run separately often share fixed per-job overhead that consolidation eliminates. (g) Scheduling: shifting non-time-critical batch work to off-peak windows where spot/reserved pricing is more favorable. (h) Data-transfer cost: at large scale, cross-region or cross-service data movement can be a surprisingly large cost line; auditing and reducing unnecessary transfer often yields savings nobody had been tracking.
2. Quick wins vs. longer-term investment. Right-sizing and storage-tier lifecycle policies are typically quick, low-risk wins (config/policy changes, not code rewrites); job consolidation and format migration are medium-effort; deeper architectural changes (moving compute location, redesigning a pipeline's core processing logic) are longer-term investments reserved for when the quick/medium wins are exhausted and cost pressure remains.
3. Measuring improvements and attribution. Roll out changes incrementally (one lever, or a small batch of independent levers, at a time) and measure BOTH cost and SLA-relevant metrics (latency, completion time, error rate) before and after each change, so a regression can be attributed to the specific change that caused it rather than lost in a bundle of simultaneous changes.
4. Cost attribution to teams/pipelines. Tag resources and cost line items by owning team/pipeline BEFORE starting the optimization effort, so savings (and any regressions) can be measured and communicated at the granularity stakeholders actually care about, and so future cost growth can be caught early by the team closest to the cause.
5. Validating savings safely. For any change carrying correctness or SLA risk (job consolidation, format changes, moving compute location), validate against a comparable data volume in a non-production environment first, and roll out to production with a defined rollback plan and monitoring window before declaring the change permanent.
6. Rollout order. Quick wins first (fastest validated savings, builds momentum and buys margin for riskier changes later), then medium-effort changes, with the highest-risk architectural changes last and only if cost targets are not yet met by the earlier tiers.

Worked example. A team facing pressure to cut cost by 40% without breaking SLAs finds via a resource audit that (a) several clusters are provisioned at roughly 2x their peak observed utilization (right-sizing alone plausibly recovers a meaningful fraction of the target), (b) 60% of data older than 30 days sits on hot storage despite being queried less than once a month (moving it to a cheaper cold storage tier recovers further savings at near-zero risk), and (c) three separate small nightly jobs process overlapping data with duplicated per-job overhead (consolidating them removes that duplication), and (d) two of those same jobs re-run an identical expensive join against a slowly-changing dimension on every invocation, a strong caching candidate once the jobs are consolidated. Applying right-sizing, partition pruning, and storage tiering first (lowest risk, fast to validate) captures a large share of the 40% target within the first rollout wave; job consolidation, validated carefully against output correctness given its higher implementation risk, closes most of the remaining gap; if a residual gap remains after these, spot-instance adoption for the most preemption-tolerant remaining workloads is the next lever, reserving any deeper architectural change as a longer-term follow-up rather than a first-wave requirement.

Trade-offs & pitfalls. The temptation under cost pressure is to apply the most aggressive lever (often spot instances or an architectural rewrite) first because it sounds impactful, when right-sizing and storage-lifecycle changes are frequently both cheaper to implement AND lower-risk -- sequence by risk-adjusted impact, not by how dramatic a change sounds. For an ML-specific variant of this problem (cutting cost for a training pipeline), REPRODUCIBILITY is an added constraint the general levers above do not automatically respect: sampling or aggressive caching applied to training data must be validated not to silently change model outputs run-to-run, which is a correctness risk beyond the usual SLA/latency risk this framework otherwise covers.