Spotify Data Engineer Interview Preparation Guide - Junior Level (1-2 Years)

Approach
I would first dedupe to one row per user per active day, because multiple events on the same day should not create extra streak length. Then I would use the gaps-and-islands pattern. An island is a run of consecutive dates. A simple way to label each island is to subtract the row number from the date, because consecutive dates keep the same shifted value.

WITH active_days AS (
  SELECT DISTINCT user_id, activity_date
  FROM raw_events
  WHERE activity_date IS NOT NULL
), islands AS (
  SELECT
    user_id,
    activity_date,
    activity_date - (ROW_NUMBER() OVER (
      PARTITION BY user_id
      ORDER BY activity_date
    ) * INTERVAL '1 day') AS island_key
  FROM active_days
)
SELECT user_id,
       MIN(activity_date) AS start_date,
       MAX(activity_date) AS end_date,
       COUNT(*) AS streak_length
FROM islands
GROUP BY user_id, island_key;

Handling bad data
If there are known missing source days, I would join to a data-quality calendar and exclude those outage dates before building streaks. That prevents a pipeline failure from looking like a user break.

Worked example
For dates 1, 2, 3, 5, the first three share one island. The 5 starts a new one. If day 4 was a confirmed outage, I would not count it as a true break.

Direct answer

Default to full recompute while the dataset is small enough that its cost is negligible and correctness is what you are optimizing for; move to incremental once recompute cost, whether time, compute spend, or contention with other jobs, becomes the actual bottleneck. Incremental is only safe once you can reliably tell exactly what changed since the last run and can apply that change idempotently (a repeated application produces the same result as applying it once); without both of those, incremental will silently drift from what a full recompute would have produced.

Structured elaboration

What has to be true for incremental to be safe:

1. Deterministic, idempotent transform: reapplying the same incremental step twice on the same input must produce the same output, an upsert or replace-by-key, not a blind increment, or a retried or replayed job double-counts.
2. Reliable change detection: you can identify exactly which upstream rows or partitions changed since the last successful run, a watermark, a CDC stream, or an explicit dirty marker, not an approximation.
3. A defined reopening path for late data: if a partition already marked done can still receive new data, there has to be a mechanism to mark it dirty again and reprocess it, or incremental quietly stops matching what a full recompute would have produced for that window.
4. Periodic reconciliation: because incremental correctness depends on the three points above holding continuously, some periodic full recompute, or at minimum a checksum or row-count comparison against one, has to exist to catch silent drift; incremental without a way to verify against ground truth is running on faith.

Worked example

A table has 500 million total rows. On a typical day, only the newest day's partition changes, plus roughly 0.5% of historical rows get reopened by late-arriving corrections:

$0.005 \times 500{,}000{,}000 = 2{,}500{,}000 \text{ reopened rows}$

Full recompute cost is proportional to all 500,000,000 rows scanned every run. Incremental cost is proportional to the new day's rows plus the 2,500,000 reopened rows, roughly 0.5% of the full-recompute cost per run once the change-detection and reopening mechanism above are actually in place. That roughly 200x reduction in rows processed per run is the entire argument for going incremental, and it only holds as long as the 0.5% reopened figure is being tracked correctly; if change detection silently misses reopened partitions, the real error compounds run over run instead of staying pinned at 0.5%.

Trade-offs & pitfalls

Adopting incremental before change detection is actually reliable is the most common wrong turn: it looks correct in testing, where late data is rare, and drifts silently in production, where it is not. Treating incremental as strictly better ignores that it adds real engineering surface, dirty-tracking, idempotent upserts, reconciliation jobs, that is not worth it for a small or rarely changing table. Skipping periodic reconciliation against a full recompute removes the only mechanism that catches drift before a stakeholder notices the numbers do not add up.

Direct answer

Snappy is the right default for high-throughput ETL (extract, transform, load): it trades a lower compression ratio for very fast compression AND decompression, keeping CPU from becoming the bottleneck on read-heavy pipelines. Gzip compresses noticeably smaller but costs meaningfully more CPU for both compression and decompression, better suited to data written once and read rarely (archival, cold storage) than to a pipeline reading the same data repeatedly. LZ4 is even faster than Snappy at a similar or slightly worse compression ratio, a reasonable alternative default where decompression speed is the single dominant concern. For serialization, Kryo should be the default over Java serialization for any performance-sensitive Spark job: it is substantially faster and more compact, at the cost of needing explicit class registration for full efficiency.

Structured elaboration

Compression codec trade-offs.

The "splittable" column matters specifically for distributed processing: for COLUMNAR formats like Parquet/ORC, compression is applied per column-chunk/block, so any of these codecs support parallel reads regardless of the compression choice; the splittability concern is much more significant for raw compressed TEXT files (a single .gz file is NOT splittable by a distributed reader, forcing one task to read the whole file), which is a strong independent argument for landing data in a columnar format rather than compressed raw text in the first place.

Why Snappy or LZ4 for high-throughput ETL specifically. An ETL pipeline typically reads its input, decompresses it, transforms it, and writes output, repeatedly, at scale; decompression happens on every single read, so a slow decompression codec (like Gzip, relatively) directly taxes every downstream stage that touches the data, not just the one job that originally wrote it. Snappy/LZ4's speed advantage compounds across every read of that data for its lifetime, while Gzip's better compression ratio saves storage cost and (marginally) initial read I/O time, a trade that favors Gzip mainly for data written once and read rarely, the opposite of a typical ETL pipeline's access pattern.

Java serialization vs Kryo. Spark, by default, uses Java's built-in serialization for shipping objects between executors (task closures, shuffle data, cached objects), which is broadly compatible (works with any Serializable class with no extra setup) but slow and produces large serialized representations. Kryo (spark.serializer=org.apache.spark.serializer.KryoSerializer) is substantially faster and more compact for the same objects, at the cost of needing classes explicitly registered (spark.kryo.registrationRequired/registerKryoClasses) for its most efficient encoding; without registration, Kryo still works (falling back to a less-optimized encoding path) but does not realize its full performance advantage, and with registrationRequired=true (a stricter mode), an unregistered class causes a hard failure rather than a silent slowdown, catching the gap explicitly rather than leaving performance on the table unnoticed.

Worked example

A nightly ETL job reading 2 TB of raw event data (currently Gzip-compressed JSON, being migrated to Snappy-compressed Parquet) and performing several transform-and-join stages before writing aggregated Parquet output.

Current state (Gzip JSON): every stage that reads the 2 TB pays Gzip's decompression cost, and because raw .gz files are not splittable, the READ stage itself cannot parallelize within a single file the way a columnar, block-compressed format can; this compounds with the format-choice question (Parquet vs raw JSON), since the compression-codec choice and the file-FORMAT choice interact.

Migration target (Snappy Parquet, Kryo serialization): switching the storage format to columnar Parquet with Snappy compression removes the single-file-not-splittable problem (Parquet's internal row-group structure is independently readable in parallel regardless of codec) AND switches to a faster codec; separately, enabling Kryo serialization (spark.serializer=org.apache.spark.serializer.KryoSerializer) with the pipeline's custom case classes/objects registered speeds up every shuffle and every cached-object access across the whole job, not just the initial read. These are two independent, additive levers (storage codec for data at rest, Kryo for in-flight serialization during shuffles/caching), both pointed at the same underlying goal: keep CPU spent on codec/serialization overhead low relative to CPU spent on the actual transform logic, since for a high-throughput pipeline that overhead is paid on every stage, not once.

Trade-offs and pitfalls

• Common mistake: choosing Gzip by default because it "compresses better," without weighing that the pipeline reads the same data repeatedly; for a genuinely write-once-read-many-times ETL pipeline (the common case), the cumulative decompression cost across many reads usually outweighs the one-time storage savings, unless storage cost specifically is the dominant concern (rare at typical ETL data volumes relative to compute cost).
• Common mistake: enabling Kryo but never registering the actual classes used, leaving real performance on the table while believing the switch "already happened"; spark.kryo.registrationRequired=true during development/testing is a good way to surface every unregistered class explicitly rather than discovering the gap only via a profiler.
• Compression codec and file FORMAT are two separate decisions that compound. Choosing Snappy over Gzip helps regardless of file format, but the LARGER win in the worked example above came from moving off raw compressed JSON to columnar Parquet in the first place (that decision in depth); do not let the codec choice alone stand in for the more consequential format choice.
• Recommended defaults, stated plainly: Snappy (or LZ4 if decompression speed is the single dominant concern) for high-throughput ETL storage compression, and Kryo with explicit class registration for Spark's in-flight serialization; reach for Gzip specifically for cold/archival data, not as a general default.

Approach

Build a Point whose __slots__ blocks new attributes from being added, whose coordinates are exposed only through read-only @property accessors (no setters), and whose __eq__/__hash__ agree by both being derived from the same tuple of coordinates. None of __slots__, the properties, or __hash__ alone would make the object safe as a dict key; together they make external mutation impossible (immutability) and make the hash provably stable for the object's lifetime (safety).

Code (Python 3.12)

class Point:
    __slots__ = ("_x", "_y")

    def __init__(self, x, y):
        self._x = x
        self._y = y

    @property
    def x(self):
        return self._x

    @property
    def y(self):
        return self._y

    def __eq__(self, other):
        if not isinstance(other, Point):
            return NotImplemented
        return (self._x, self._y) == (other._x, other._y)

    def __hash__(self):
        return hash((self._x, self._y))

    def __repr__(self):
        return f"Point({self._x}, {self._y})"

p1 = Point(1, 2)
p2 = Point(1, 2)
cache = {p1: "origin-ish"}
print(cache[p2])          # "origin-ish": p2 hashes/compares equal to p1
print(len({p1, p2}))       # 1

try:
    p1.x = 99
except AttributeError as e:
    print("cannot mutate:", e)

Output:

origin-ish
1
cannot mutate: property 'x' of 'Point' object has no setter

Why immutability matters here, made concrete

If the same class exposed x/y as plain mutable attributes instead of read-only properties, mutating an instance after it is already a dict key silently breaks lookups, because the dict placed the entry based on the hash computed at insertion time:

class MutablePoint:
    def __init__(self, x, y):
        self.x, self.y = x, y
    def __eq__(self, other):
        return isinstance(other, MutablePoint) and (self.x, self.y) == (other.x, other.y)
    def __hash__(self):
        return hash((self.x, self.y))

mp = MutablePoint(1, 2)
d = {mp: "value"}
print(mp in d)     # True, right after insertion
mp.x = 999          # mutate the key after it's already stored
print(mp in d)     # False: mp now hashes to a different bucket than the one it lives in
print(len(d))       # 1: the entry is still physically there, just unreachable via mp

Output:

True
False
1

The entry is not lost, it is orphaned: d still has one entry, but no lookup starting from mp's current state will ever find it again, because hash(mp) changed after insertion while the dict's internal bucket placement did not move.

Key points

• __slots__ removes the per-instance __dict__, which both saves memory for many instances and, as a side effect here, prevents adding a new attribute (p1.z = 1) from outside; it does not by itself make existing slot attributes read-only.
• Read-only @property (a getter with no setter defined) is what actually blocks p1.x = 99; combining it with __slots__ for the backing _x/_y fields is what gives full external immutability.
• __hash__ and __eq__ are built from the exact same tuple, (self._x, self._y), so the hash-equality invariant holds by construction: there is no way for two Points to compare equal while hashing differently, or vice versa.

Complexity and edge cases

Construction, equality, and hashing are all $O(1)$ for a fixed number of fields (hashing a fixed-size tuple is $O(1)$ in the number of fields, though hashing each field itself costs whatever hashing that field's type costs).

• The absorbed "read-only mapping type usable as a dict key" variant is the same competency at the container level: types.MappingProxyType wraps an existing dict as a read-only view (it prevents mutation through the proxy, but the underlying dict can still change and the proxy itself is not hashable), whereas a hashable, dict-key-safe mapping needs an immutable backing store, for example built from a frozenset of (key, value) pairs, plus a custom __hash__ over that frozenset.
• NotImplemented (not False) from __eq__ when compared to an incompatible type lets Python fall back to the other object's __eq__, which matters if Point is ever compared against a type that knows how to compare itself to a Point but is not itself a Point.

Direct answer

Depth-first search (DFS) explores a graph by following one path as deep as possible before backtracking; it can be implemented recursively (using the language's own call stack) or iteratively (using an explicit, heap-allocated stack you manage yourself). In production data-processing jobs, iterative DFS is generally preferred because it sidesteps recursion-depth limits, avoids reliance on tail-call optimization that most mainstream runtimes do not provide, and makes checkpointing a long traversal possible.

Structured elaboration

Recursion limits and stack overflow. Most language runtimes cap call-stack depth (Python's default is around 1000 frames; the JVM and most native stacks are also bounded, just at a larger and less predictable size depending on frame size and thread stack settings). A deep or adversarially structured input, a long lineage chain, a deeply nested document, a graph shaped more like a long path than a wide tree, can exceed that limit and crash with a stack overflow or a language-level recursion error, even though the algorithm itself is correct and the WORK involved is entirely reasonable.

Tail-call optimization (TCO). Some languages eliminate the stack growth of a self-recursive call in tail position, making deep recursion as cheap as a loop. Python, Java, and most JVM languages do not do this, so recursive DFS in these languages has a real, unavoidable per-call stack cost that iterative DFS with an explicit stack does not.

Reliability and observability. An explicit stack is just a data structure the code owns: it can be inspected mid-traversal (current depth, current frontier), logged, capped with a hard size limit that fails gracefully instead of crashing the process, and generally treated as first-class program state rather than hidden call-frame bookkeeping.

Checkpointing and resumption. This is the strongest production argument. A long-running traversal, for example a full crawl of a large dependency graph or lineage table, needs to survive a deploy, a crash, or a scale-in event. An explicit stack is just a list of (node id, iteration position) pairs, which can be serialized to durable storage (a queue, a database row, a checkpoint file) and reloaded to resume exactly where the job left off. A recursive call stack cannot be serialized this way at all: the "resume point" lives in interpreter/runtime call frames that have no external representation.

Worked example

Consider a data engineering job that walks a table-lineage DAG (directed acyclic graph) to compute, for every table, the full set of upstream sources it depends on. Two implementation choices, same logic:

Recursive: def visit(table): mark visited; for upstream in table.upstreams(): visit(upstream). Simple to write, but if the lineage graph has a chain of 5,000 tables feeding into each other (a real pattern in mature data warehouses with many small transformation steps), this recursion depth alone can exceed Python's default recursion limit, failing on a perfectly valid, non-cyclic input.

Iterative with an explicit stack, checkpoint-capable: the stack holds (table_id, next_upstream_index) tuples instead of raw table ids, so the traversal can pause after processing any table, serialize the current stack contents plus the visited set to a checkpoint row, and resume later by deserializing that same state and continuing the loop. If the job is killed mid-run by a deploy, the resumed job re-reads the last checkpoint and continues from that exact point instead of restarting the entire lineage walk from scratch.

Trade-offs and pitfalls

• Recursive DFS is not wrong, it is simpler to read and reason about, and for graphs with bounded, known-shallow depth (most business-logic trees, most small configuration graphs) the recursion-limit risk never materializes. Reach for iterative DFS specifically when depth is unbounded, unknown, or the traversal is long-running enough that resumability matters.
• Common mistake: raising the recursion limit (sys.setrecursionlimit in Python) as a fix instead of switching to an iterative approach; this treats the symptom, not the underlying fact that call-stack memory is finite and a sufficiently deep or adversarial input will still exceed whatever new limit is chosen, it also risks a genuine native stack overflow (a process crash, not a catchable exception) rather than a clean language-level error.
• Common mistake: building "checkpointing" on top of a recursive implementation by catching exceptions and retrying from the top; without an explicit, serializable notion of traversal state, a retry after partial progress has no way to skip work already done, so it either redoes everything (wasteful) or needs a separate, bolted-on progress-tracking mechanism that the iterative approach gets for free from its own stack representation.
• Idempotency matters alongside checkpointing. If a resumed traversal might re-process a node whose completion was not yet durably recorded before a crash, downstream writes (aggregations, side effects) should be safe to repeat, since exact-once resumption is hard to guarantee and idempotent-safe-to-retry is usually the more practical target.

Direct answer

Separating the person from the problem means directing your response at the specific piece of work, data, or decision, not at the person's character or competence, and replacing an evaluative claim ("this is sloppy") with an observable one (what you actually saw, and the specific gap). The reframe isn't about softening the message. It's about making the message actionable: a character judgment gives someone nothing to do but defend themselves, while a fact-based statement gives them something to fix, or something specific to correct you on.

Structured elaboration

The core move is to separate the person from the problem: respond to the specific behavior or claim, not to a judgment about who they are. In practice that means distinguishing what you actually observed from the interpretation you layered on top of it.

1. Notice the trigger. A sentence built on an inferred internal state ("you don't care," "you're not thorough") is a judgment. A sentence built on an external, checkable detail (a date, a number, a diff, a specific quote) is fact-based.
2. Convert it. Replace the internal-state claim with what you observed, the specific instance it happened in, and the concrete question or ask that follows from it.
3. Invite correction. End with a question, not a verdict, so the other person can add context you're missing rather than just defend against a conclusion.
4. Keep the pattern separate from the instance. If it's really a recurring pattern that needs addressing, that's a different, private conversation about the trend, still anchored in specific instances, not a trait.

Worked example

Two engineers are deep in a heated architecture debate over which service should own write authority. One says: "You always push for the more complex design because you want something to put on your résumé." That's a character claim with no evidence attached, and it immediately spikes the temperature in the room. If you're facilitating, the live reframe sounds like: "Let's park the why for a second and get to the tradeoff: what specific failure mode are you worried about with the shared-ownership design?" That single sentence does the work live. It drops the interpretation, restates the actual technical question, and hands the conversation back to something answerable. In this case, the redirected conversation surfaces that the real objection was about a specific consistency guarantee under partial failure, which turned out to be addressable in about an hour, not a two-day standoff about who's the better architect.

Trade-offs and pitfalls

• Overcorrecting into vagueness isn't more factual, it's less useful. "There were some concerns" doesn't identify anything a person can act on.
• This is a technique for reducing defensiveness, not for avoiding a hard message. If someone genuinely underperformed, the reframe changes how you say it, not whether you say it.
• Used mechanically, it reads as a script. The goal is genuine curiosity about what happened, not visibly following a formula in front of someone who can tell the difference.
• It assumes reasonable good faith on the other side. If someone repeatedly reframes fact-based feedback as a personal attack no matter how carefully you phrase it, that's a pattern problem, not a phrasing problem, and it needs a direct, documented conversation instead of a softer script.

At a handful of nodes a fixed heartbeat timeout works fine: miss N beats, declare the node down. At several thousand nodes spread across racks and regions, the same fixed threshold produces both false positives (a node under scheduling pressure or behind a congested link gets marked down while still alive) and slow true detections (a threshold loose enough to avoid those false positives takes too long to catch a real failure), because network and scheduling jitter is not uniform across the cluster. A Phi Accrual failure detector replaces the single global timeout with a statistical model of each node's own recent heartbeat behavior, so the question becomes how improbable this silence is given what has actually been observed from that specific node, rather than whether more than T seconds have passed, and it outputs a continuously increasing suspicion level, phi, instead of a binary up-or-down verdict.

Membership and probing, before the suspicion model

• Partial views and gossip: each node keeps a small, mostly random subset of the membership rather than the full list, and periodically gossips membership updates and heartbeats to a few random peers, so state propagation stays cheap instead of all-to-all.
• Direct and indirect probing (SWIM-style; SWIM: a gossip-based membership protocol combining direct pings with an indirect fallback): a node pings a random peer directly, and if it gets no acknowledgment, asks a handful of other random members to ping that peer on its behalf, since one lossy path between two specific nodes should not be read as evidence the target itself is down.
• A minimal membership-manager state machine per monitored node: ALIVE, SUSPECT, DEAD. Failed direct-plus-indirect probes move a node from ALIVE to SUSPECT rather than straight to DEAD, and the suspicion is gossiped with an incarnation number, so a node that is actually alive can refute it by re-announcing itself with a higher incarnation before it is declared DEAD and dropped from membership.

From a binary probe result to a continuous suspicion level

Probing only tells you whether a heartbeat arrived, which is still binary. Phi accrual sits on top of that: instead of one global timeout, each node keeps a short sliding window of the actual inter-arrival times it has observed from a given peer, uses that history to estimate how spread out that peer's heartbeats normally are, and converts the elapsed time since the last heartbeat into a suspicion level using roughly:

where $P_{later}(t)$ is the estimated probability, given that peer's own recent history, that a heartbeat still arrives more than $t$ time units after the previous one. As $t$ grows past what the peer's history says is normal, $P_{later}(t)$ shrinks and phi grows smoothly and without bound, instead of snapping from 0 to 1 at one fixed cutoff.

Why this matters at thousands of nodes: a peer on a congested cross-region link naturally has wider jitter in its own history, so it takes a longer silence to reach the same phi value as a peer on a quiet local link, without anyone hand-tuning a per-peer timeout. It also lets different consumers of the same signal act at different confidence levels: something that only needs to stop routing traffic to a possibly-dead node can act at a low phi and cheaply reverse course if the node turns out fine, while something that removes a node from a lock's quorum permanently can wait for a much higher phi, all from the same underlying measurement.

Worked example: computing phi from a pinned heartbeat history

Take a 6-sample sliding window of observed inter-arrival times, in milliseconds, for one peer: [1000, 1300, 700, 1100, 900, 1200].

Mean $\mu = 1033.33$ ms, population standard deviation $\sigma = 197.2$ ms.

Approximating the interval distribution as Normal($\mu, \sigma^2$), so that $P_{later}(t) = \tfrac{1}{2}\,\mathrm{erfc}\!\left(\tfrac{t-\mu}{\sigma\sqrt{2}}\right)$ (erfc is the complementary error function, a standard tool for turning a distance from the mean into the probability mass remaining in the distribution's tail beyond that point), gives:

Tracing the 1500 ms row end to end: standardize the elapsed time against this peer's own mean and spread, $z = (1500 - 1033.33) / (197.2 \times \sqrt{2}) = 466.67 / 278.88 \approx 1.673$; convert that into a tail probability, $P_{later}(1500) = \tfrac{1}{2}\,\mathrm{erfc}(1.673) \approx \tfrac{1}{2} \times 0.0180 \approx 0.00898$; then take $-\log_{10}$ of that probability, $\phi(1500) = -\log_{10}(0.00898) \approx 2.05$. The other three rows fall out of the same three steps with a different elapsed time plugged in.

At t = 1000 ms, right around this peer's own mean, phi stays near zero: nothing unusual. At t = 1500 ms, phi crosses 2, a mild anomaly worth noting but not worth acting on for most purposes. By t = 2500 ms, phi is in the double digits, meaning the observed silence is astronomically improbable given this specific peer's recent behavior, which is enough statistical confidence for even a conservative consumer to move the node to SUSPECT, or to DEAD if it was already SUSPECT and never refuted with a higher incarnation.

Trade-offs and pitfalls

Phi is not a probability itself, it is a log-scaled confidence score, and reading a given phi value as a direct percentage chance of failure is a common misunderstanding; the number only means something relative to whatever threshold a given consumer chooses. A second pitfall is picking one global phi threshold across a heterogeneous fleet, which reintroduces the exact problem this design was meant to solve, since the entire benefit comes from calibrating to each peer's own recent history rather than from a smarter constant. Third, phi accrual detects the absence of a heartbeat; it cannot distinguish a genuinely dead node from a network partition dropping every packet between two specific nodes while both are alive and healthy elsewhere, so membership decisions that need to be safe, like removing a node from a lock's quorum, still need a consensus-backed or fencing mechanism layered on top of the suspicion signal, not phi alone.

Direct answer. In Amazon Kinesis Data Streams, a stream is divided into shards, each an independent, ordered sequence of records. A producer writes records to a shard (chosen by a partition key), a consumer reads records from a shard in order, and retention determines how long unconsumed records stay available (24 hours by default, extendable up to 365 days). Shard count is the unit of both throughput and parallelism: more shards mean more concurrent producers and consumers can be served, and the total capacity of the stream scales linearly with shard count.

Structured elaboration.

• Shard. The base unit of capacity. Each shard supports up to 1MB/second or 1,000 records/second of writes, whichever limit is hit first, and up to 2MB/second of reads (or 5 GetRecords calls/second per shard, depending on consumer type).
• Producer. Writes records to the stream, specifying a partition key. Kinesis hashes the partition key to determine which shard the record lands on, so records with the same key always go to the same shard, preserving per-key order.
• Consumer. Reads records from one or more shards, in order, using a shard iterator. Kinesis Client Library (KCL) applications typically run one worker per shard to parallelize consumption.
• Retention. How long a record stays in the stream if unconsumed. The default is 24 hours; you can extend it for replay scenarios (reprocessing after a bug fix) at additional storage cost.
• Resharding. Splitting a shard increases capacity for a hot partition key range; merging reduces shard count (and cost) when a stream is over-provisioned. Both operations are typically triggered manually or via auto-scaling policies, not automatically by Kinesis itself.

Worked example. A team's typical reason to reshard is outgrowing either the write throughput (aggregate producer traffic exceeds shard capacity, causing ProvisionedThroughputExceededException errors) or hitting a hot-key problem (one partition key's traffic overwhelms its single shard while others sit idle). Splitting the hot shard into two increases its effective capacity without touching the rest of the stream, while a broad traffic-growth reshard usually means increasing shard count across the board. Since shard count directly drives cost (each shard has an hourly charge regardless of how much of its capacity you actually use), teams periodically merge shards back down during low-traffic periods or seasonal troughs to avoid paying for idle capacity.

Trade-offs and pitfalls. A common mistake is choosing a partition key with low cardinality (for example, a fixed region label for a service with only three regions), which concentrates traffic onto a handful of shards regardless of how many shards the stream has, defeating the purpose of adding more. Resharding is also not instantaneous or free: it briefly affects the shard's availability for writes and requires consumers to be resilient to shard-ID changes as they discover new child shards after a split or merge. Basic cost planning should account for the fact that you pay per shard-hour whether or not that shard's capacity is fully used, so right-sizing shard count to actual traffic (not a generous overestimate "to be safe") directly controls cost.

Direct answer

When standard observability has failed (logs rotated away, metrics missing, dashboards blank), the investigation has to fall back to whatever EVIDENCE still exists outside the normal instrumentation path: OS-level and host metrics that exist independent of the application's own logging, network captures showing what actually crossed the wire, user-visible outputs and error messages as a proxy for internal state, and any third-party system that independently recorded a copy of relevant activity.

Structured elaboration

When the usual "check the logs, check the dashboard" path is unavailable, the fallback sources fall into a few categories, each useful for a different kind of question:

1. OS-level and host metrics (/proc, a live snapshot of what the OS knows about running processes and memory; top/ps history, point-in-time process and resource usage; the systemd journal, the system's own log of service starts, stops, and events; and kernel logs via dmesg, low-level messages from the OS itself, including out-of-memory kills) exist independent of whatever the application chose to log. Of these, dmesg is usually the first one worth checking for a suspected resource-exhaustion incident, since it directly reports OOM kills, and can confirm or rule out resource exhaustion, OOM kills, or a process restart, even with zero application-level logging.
2. Network captures or connection state (existing packet captures if any were running for another purpose, or even just current connection state via ss/netstat if the investigation starts while the issue is still ongoing) can show what actually happened on the wire, useful when application logs would have recorded the SAME information but didn't.
3. User-visible outputs: error messages actually shown to users, HTTP status codes observed by a synthetic monitor or a customer's own tools, timestamps from customer reports, all reconstructable from OUTSIDE the system under investigation.
4. Any third-party or downstream system that independently recorded activity: a CDN's own access logs, a payment processor's transaction log, a client-side error-tracking tool, any system that received a copy of the interaction and kept its own record, even if it's not the system you'd normally check first.
5. Timing correlation across whatever fragments exist, reconstructing a plausible timeline even from incomplete sources: a customer-reported timestamp, a downstream system's transaction timestamp, and a host-level kernel log entry, cross-referenced, can often reconstruct enough of a sequence to form a solid hypothesis even with none of the "normal" evidence intact.

Worked example

An outage where the application's structured logs were rotated away before anyone realized an investigation would be needed, and the metrics dashboard shows a gap during the exact incident window due to an unrelated agent failure. Reconstructing from what remains: dmesg shows an OOM-killer event for the application's process at almost exactly the time customers first reported errors, which is a strong, independent signal pointing at memory exhaustion without needing a single application log line. Cross-referencing against a downstream payment processor's own transaction log (which independently timestamped every request it received from this service) shows request volume to it dropped to zero at the same moment, confirming the process was fully down, not just degraded. Together, these two independent, non-application sources reconstruct a timeline (memory exhaustion, process killed, requests stopped) that's just as actionable as a full log trail would have been, despite neither source being the application's own instrumentation.

System changes made afterward to avoid recurrence: extend log retention specifically to cover a reasonable incident-investigation window (not just cost-driven defaults); add redundant, INDEPENDENT monitoring (a host-level agent separate from the application's own metrics pipeline, so a single pipeline failure doesn't blind you to everything at once); and specifically add OOM/resource-exhaustion alerting sourced from the kernel/host level rather than solely from application-reported metrics, since this incident showed the application-level signal was exactly what went missing.

Trade-offs and pitfalls

The core skill here isn't a specific tool, it's the discipline of treating "our normal observability failed" as its own root-cause-worthy fact (why did logs rotate that fast, why did the metrics agent fail silently) in addition to whatever the underlying incident was, since fixing only the original incident without fixing the observability gap guarantees the NEXT incident will be just as hard to diagnose.

Recommendation

For thousands of short patterns over a very long text, build one Aho-Corasick automaton and read the text once; in production code, use a maintained library for it rather than writing your own. Hand-write it only when you need behaviour a library does not give you or cannot install one. A separate search per pattern wins only when there are very few patterns, or when the text can be indexed once and searched many times.

An Aho-Corasick automaton is a trie (a tree whose paths spell the patterns) with failure links that tell the scan where to continue after a mismatch, so the text is never re-read.

Cost model

Let N be the text length, k the number of patterns, P the total length of all patterns, and z the number of matches you must list.

KMP (Knuth-Morris-Pratt) and Z are linear-time single-pattern searches, and Boyer-Moore skips ahead using the text character where a comparison failed; for this decision only their cost shape matters: each reads about the whole text once, and you pay that once per pattern. The reason the automaton wins at scale is that k leaves the text term: the work per text character is constant on average over the whole run (this is what amortized means: occasional expensive steps are paid for by the cheap ones around them), not proportional to the number of patterns. z is the matches you list. If you only need counts per pattern, you can drop z too: add one to a counter for the node reached after each character, then push counts up the failure links from the deepest nodes to the shallowest. That is the count path, worked on a tiny input after the measured example.

Measured on a pinned example

The program below counts overlapping occurrences of k = 300 random 4-letter patterns over a random 50,000-character text on an 8-letter alphabet, with seeded random numbers. It also asserts that the counts equal those from a separate regular-expression scan per pattern.

import random
import re
from collections import deque


def count_all(patterns, text):
    """Occurrences (overlaps included) of every pattern, in one pass over the text."""
    nxt, fail, end = [{}], [0], {}
    for pid, p in enumerate(patterns):
        node = 0
        for ch in p:
            if ch not in nxt[node]:
                nxt.append({})
                fail.append(0)
                nxt[node][ch] = len(nxt) - 1
            node = nxt[node][ch]
        end.setdefault(node, []).append(pid)
    order, queue = [], deque(nxt[0].values())
    while queue:
        u = queue.popleft()
        order.append(u)
        for ch, v in nxt[u].items():
            f = fail[u]
            while f and ch not in nxt[f]:
                f = fail[f]
            fail[v] = nxt[f].get(ch, 0)
            queue.append(v)
    visits, node, hops = [0] * len(nxt), 0, 0
    for ch in text:
        while node and ch not in nxt[node]:
            node = fail[node]
            hops += 1
        node = nxt[node].get(ch, 0)
        visits[node] += 1                      # only count the visit now
    for u in reversed(order):                  # deepest first: a visit to u is also a visit to its suffix nodes
        visits[fail[u]] += visits[u]
    counts = [0] * len(patterns)
    for node, pids in end.items():
        for pid in pids:
            counts[pid] = visits[node]
    return counts, len(nxt) - 1, hops


def per_pattern(patterns, text):
    return [len(re.findall("(?=" + re.escape(p) + ")", text)) for p in patterns]


if __name__ == "__main__":
    rng = random.Random(11)
    n, k, m = 50_000, 300, 4
    text = "".join(rng.choice("abcdefgh") for _ in range(n))
    patterns = ["".join(rng.choice("abcdefgh") for _ in range(m)) for _ in range(k)]
    counts, nodes, hops = count_all(patterns, text)
    assert counts == per_pattern(patterns, text)
    print(f"text length N={n}, patterns k={k}, total pattern length P={k * m}")
    print(f"automaton: {nodes} trie nodes, {n} transitions + {hops} failure hops = {n + hops} steps")
    print(f"separate scans: {k} x {n} = {k * n} character positions visited")
    print(f"total occurrences counted: {sum(counts)}")

Run in a python:3.12 container (Python 3.12.15), it prints:

text length N=50000, patterns k=300, total pattern length P=1200
automaton: 598 trie nodes, 50000 transitions + 47962 failure hops = 97962 steps
separate scans: 300 x 50000 = 15000000 character positions visited
total occurrences counted: 3564

The automaton does 97,962 steps (50,000 transitions, one trie edge followed per text character, plus 47,962 failure hops, each a fallback along a failure link after a mismatch) for all 300 patterns, versus 300 x 50,000 = 15,000,000 character positions for one full pass per pattern, about 153 times fewer. A skipping algorithm like Boyer-Moore reads fewer than all positions per pattern, so the gap against that baseline is smaller, but with 4-letter patterns the skips are short.

The count path on a tiny input

Patterns he, she, hers against the text ushers. The scan only adds 1 to the node it lands on after each character: s, sh, she, her, hers are each visited once (the first character u leaves it at the root). That alone would count she and hers correctly but miss he, which ended inside she. A visit to a node is also a visit to every node on its failure chain, because those are suffixes of the text just read; she fails to he, so pushing the count from she to he credits he with the occurrence. Processing the nodes deepest first guarantees a node's count is complete before it is pushed further up. Here is that walk-through as code, with every step printed:

from collections import deque


def count_walkthrough(patterns, text):
    nxt, fail, name = [{}], [0], [""]
    for p in patterns:
        node = 0
        for ch in p:
            if ch not in nxt[node]:
                nxt.append({})
                fail.append(0)
                name.append(name[node] + ch)
                nxt[node][ch] = len(nxt) - 1
            node = nxt[node][ch]
    order, queue = [], deque(nxt[0].values())
    while queue:
        u = queue.popleft()
        order.append(u)
        for ch, v in nxt[u].items():
            f = fail[u]
            while f and ch not in nxt[f]:
                f = fail[f]
            fail[v] = nxt[f].get(ch, 0)
            queue.append(v)
    visits, node = [0] * len(nxt), 0
    for ch in text:
        while node and ch not in nxt[node]:
            node = fail[node]
        node = nxt[node].get(ch, 0)
        visits[node] += 1
    print("visits after the scan :", {name[u]: visits[u] for u in order if visits[u]})
    for u in reversed(order):                    # deepest nodes first
        if visits[u]:
            print(f"  push {visits[u]} from {name[u]!r} to its failure target {name[fail[u]]!r}")
        visits[fail[u]] += visits[u]
    print("visits after the push :", {name[u]: visits[u] for u in order if visits[u]})
    return {p: visits[next(u for u in order if name[u] == p)] for p in patterns}


print(count_walkthrough(["he", "she", "hers"], "ushers"))

Output (Python 3.12, python:3.12 container):

visits after the scan : {'s': 1, 'sh': 1, 'her': 1, 'she': 1, 'hers': 1}
  push 1 from 'hers' to its failure target 's'
  push 1 from 'she' to its failure target 'he'
  push 1 from 'her' to its failure target ''
  push 1 from 'sh' to its failure target 'h'
  push 1 from 'he' to its failure target ''
  push 2 from 's' to its failure target ''
  push 1 from 'h' to its failure target ''
visits after the push : {'h': 1, 's': 2, 'he': 1, 'sh': 1, 'her': 1, 'she': 1, 'hers': 1}
{'he': 1, 'she': 1, 'hers': 1}

The last line is the per-pattern count: he once (inside she, at index 2), she once, hers once.

When each wins

• Per-pattern search wins when k is small (a handful of patterns), when the platform's built-in substring search is vectorised native code (compiled code that processes many characters per CPU instruction) and your automaton would be interpreted code, when patterns change constantly and building the trie each time is not amortised, or when the text is static and queried with many different patterns: build a suffix array (a sorted list of all suffix start positions of the text) once, and each pattern then costs about O(m log N): a binary search makes about log N probes and each probe compares up to m characters of the pattern with a suffix. The suffix array route is the heavier tool: it pays off only when the same text is searched many times.
• The automaton wins when the pattern set is large and fixed, the text is long or arrives as a stream (it keeps one state and O(P) memory, and can carry the state across chunks), and you need every pattern in one pass.

When not to hand-write it

Use an optimised library when:

• the pattern set is large or changes often and you want a tested implementation: the aho-corasick crate in Rust, pyahocorasick in Python, and fixed-string multi-pattern modes of command-line tools (grep -F -f patterns.txt) exist for exactly this;
• you need case folding, Unicode handling, word boundaries or regular-expression features, which libraries and engines such as RE2 (Google's regular-expression engine, built to avoid backtracking) handle and a naive automaton does not;
• memory matters: libraries pack the automaton into arrays, where a dict-per-node version in Python spends at least 64 bytes on every node (the size of an empty dict in CPython 3.12, 184 bytes with one entry);
• speed matters: a compiled implementation is much faster than interpreted Python, and the matching loop is the hot loop.

Hand-write it when an interview asks for it, when no suitable library exists for the platform (embedded, restricted runtime), or when you need custom semantics: the leftmost-longest rule, per-pattern weights, or state stored between calls. The leftmost-longest rule means that when several patterns match starting at the same text position, the longest one is the one reported.

A regular-expression alternation p1|p2|...|pk is not a substitute on a backtracking engine such as Python's re: it tries alternatives one after another at each text position (a backtracking engine tries one choice and, if the rest fails, goes back and tries the next), so the cost grows with k. Engines with linear-time guarantees (RE2) compile alternations into an automaton.

Pitfalls

• Counting overlapping occurrences: the count path above includes overlaps and shorter patterns nested inside longer ones, because a visit to a node also counts for its suffix nodes.
• Building the trie from patterns that share long prefixes: fine, nodes are shared.
• Reporting z matches when only counts are needed.

Running the code

python cnt.py
python tiny.py

Save the listings as cnt.py and tiny.py. It uses only the standard library (random, re, collections).