Google Data Engineer Interview Preparation Guide - Junior Level (1-2 Years)

Direct answer: Reach for a window function whenever the downstream consumer needs to see every original row and a group-level or ordered fact attached to it. GROUP BY collapses N rows into one summary row per group, which is correct when you only need the summary, but it permanently discards which specific rows produced that summary, their order relative to each other, and any row that isn't itself the aggregate result. A window function computes the same kind of aggregate or positional fact but writes it back onto every original row instead of replacing them.

Structured elaboration

Situation 1: rolling or trailing time-window features. You need each user-day's trailing 30-day spend as a feature, but you also need to keep the original daily row (date, that day's own spend, any other daily attributes) for model training or a time-series chart. GROUP BY user would collapse all of a user's days into one row and lose the day-by-day granularity entirely; you could GROUP BY user, day, but that just gives you the single day's own spend back, with no way to see the trailing context on that same row.

SELECT user_id, day, amount,
  SUM(amount) OVER (PARTITION BY user_id ORDER BY day
    ROWS BETWEEN 29 PRECEDING AND CURRENT ROW) AS spend_30d
FROM transactions;

What GROUP BY would throw away: every day's individual spend value and its position relative to neighboring days; you'd be left with only a single number per user, not a sequence.

Situation 2: row-to-row comparisons (LAG/LEAD). You need the time since a customer's previous transaction, or the previous transaction's amount, as a feature on the current transaction's row.

SELECT *,
  LAG(ts) OVER (PARTITION BY user_id ORDER BY ts) AS prev_ts,
  ts - LAG(ts) OVER (PARTITION BY user_id ORDER BY ts) AS time_since_prev
FROM events;

What GROUP BY would throw away: there is no group-level aggregate that captures "the specific previous row's value" at all; MIN/MAX/AVG summarize across a whole group, they cannot single out "the one row immediately before this one." GROUP BY has no concept of adjacency between rows.

Situation 3: ranking or flagging within a group, kept at row grain. You need to flag each purchase as a customer's first purchase, or rank each purchase by amount within that customer's history, while keeping every purchase row intact for a transaction log or funnel report.

SELECT *,
  ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY ts) = 1 AS is_first_purchase,
  RANK() OVER (PARTITION BY user_id ORDER BY amount DESC) AS amount_rank
FROM purchases;

What GROUP BY would throw away: the identity of which specific row is the first, or how each row ranks against its peers; GROUP BY user_id with MIN(ts) tells you when the first purchase happened but not which purchase row that was or anything else about it (amount, product, channel), because those attributes belonged to a row that GROUP BY has already discarded.

Key points

• The unifying idea: window functions annotate rows, GROUP BY replaces rows with a summary.
• Whenever "I need this detail row, plus some fact computed across its group or neighbors, on the same output row," that's the window-function signal.
• Deduplication and top-N-per-group patterns follow the same shape: rank within a partition, then filter, rather than aggregating the group away.

Worked example

Fixture for events (situation 2's row-to-row comparison):

Running the LAG query above against this data (verified in DuckDB):

Each user's opening row has no predecessor, so prev_ts and time_since_prev come back NULL; every later row carries the previous row's timestamp and the gap since it, and all five rows stay in the output, rather than being collapsed down to two rows (one per user) the way a GROUP BY user_id would.

Trade-offs & pitfalls

Prefer GROUP BY plus a join back to detail rows over a window function when the aggregate itself is expensive or when only a small number of pre-aggregated values are actually needed downstream (e.g., a small lookup table of category totals used by many queries): materializing the aggregate once and joining is often clearer and cheaper than recomputing a window function over the full detail set every time the query runs. A common wrong turn is reaching for a window function purely out of habit when a true GROUP BY summary was all the report ever needed; that adds unnecessary per-row computation and a wider result set for no benefit.

Direct answer

Move to a dedicated warehouse when ad-hoc analytical queries start measurably hurting the production database's transactional performance, when the same numbers are being computed slightly differently in different spreadsheets, or when reporting needs data joined across sources the production database does not have (a payments processor, a support tool, a marketing platform). Start with the simplest useful version: a small set of tables that are periodically copied out of production into a separate database or a managed cloud warehouse, denormalized just enough to answer the handful of reports people actually run today, not a fully modeled Kimball bus architecture with conformed dimensions across every future business process.

Structured elaboration

Signal one: production impact. A heavy analytical query (a full table scan for a monthly report, say) run directly against the database serving live user traffic can degrade transactional latency for real users; if analysts are being asked to "only run reports at night" or engineers are seeing production incidents traced to a report someone ran, that is a concrete, observable signal, not a vague sense that things feel slow.

Signal two: inconsistent numbers. Once more than one person is computing the same metric independently (one analyst's spreadsheet formula, another's ad-hoc query), small differences in filtering or date handling silently produce different answers to "what was our revenue last month," and nobody notices until two answers are compared in the same meeting. This is the earliest, cheapest form of the exact conformance problem later covered by dimension conflicts across marts; catching it before it compounds is far cheaper than the reconciliation project.

Signal three: joining across sources. Once a report needs to combine production order data with a separate support tool's ticket data and a third-party payment processor's transaction data, there is no single production database to query against anymore; some place has to receive copies of all three and let them be joined together, which is the core job a warehouse exists to do.

The simplest version to build first. Do not start with a full dimensional model. Start with a small, straightforward extract-and-load process (even a scheduled job that copies a handful of production tables into a separate database or a managed cloud warehouse on a nightly cadence) and let analysts query those copies directly, denormalized or lightly modeled, for exactly the reports people already run. Introduce actual dimensional modeling (declared grain, a real date dimension, slowly-changing-dimension handling) only once a second or third report reveals that ungoverned ad-hoc structure is producing inconsistent answers or is too slow to maintain by hand, which is the point at which the methodology and system-design questions the rest of this topic covers actually become relevant.

Worked example

A single unindexed analytical query scanning a 10-million-row production orders table for a monthly report can hold a lock or consume enough I/O bandwidth to add hundreds of milliseconds to unrelated transactional queries hitting the same table concurrently; at a company processing customer-facing checkout requests against that same table, a delay large enough for customers to notice during checkout is the concrete, business-visible cost of skipping a warehouse, not an abstract inefficiency. That single observation, "a report degraded checkout latency," is usually the moment a 20-person startup's engineering leadership actually approves the investment, well before any of the modeling-methodology questions in this topic become the operative concern.

Trade-offs and pitfalls

The most common mistake at this stage is over-building: reaching for a full Kimball-style bus architecture, multiple conformed dimensions, and Type 2 slowly changing dimension (SCD) history tracking before there is more than one or two reports that need any of it wastes engineering effort the startup does not have to spare, and most of that early investment will be redesigned anyway once real reporting needs are better understood. The opposite mistake, waiting until the production database is visibly struggling before doing anything, is also common and more expensive to unwind, since by then inconsistent numbers have usually already reached several audiences and eroded trust in whichever spreadsheet or dashboard people were relying on.

Try to safely cast the cleaned string and flag whatever doesn't survive the cast, rather than trying to enumerate every possible malformed pattern up front.

Approach

SELECT id, amount_text,
  CASE
    WHEN REGEXP_MATCHES(amount_text, '^\s*\(.*\)\s*$')
      THEN -1 * TRY_CAST(REGEXP_REPLACE(amount_text, '[^0-9.]', '', 'g') AS DECIMAL(12,2))
    ELSE TRY_CAST(REGEXP_REPLACE(amount_text, '[^0-9.\-]', '', 'g') AS DECIMAL(12,2))
  END AS amount_clean,
  (CASE
    WHEN REGEXP_MATCHES(amount_text, '^\s*\(.*\)\s*$')
      THEN -1 * TRY_CAST(REGEXP_REPLACE(amount_text, '[^0-9.]', '', 'g') AS DECIMAL(12,2))
    ELSE TRY_CAST(REGEXP_REPLACE(amount_text, '[^0-9.\-]', '', 'g') AS DECIMAL(12,2))
  END) IS NULL AS amount_error
FROM sales_raw;

Parenthesized amounts like '(500.00)' are detected up front by matching the whole string against a leading '(' and trailing ')', and are negated after stripping the parens themselves; anything else falls through to the original strip-and-cast path. Stripping everything except digits, the decimal point, and a minus sign handles common currency-symbol and thousands-separator noise ('$1,234.56' becomes '1234.56'); TRY_CAST (or SAFE_CAST, depending on dialect) returns NULL instead of erroring on anything that still isn't a valid number after cleaning, which is exactly the signal you want to flag as amount_error.

Worked example

"$1,234.56" cleans to "1234.56" and casts successfully; "(500.00)", the parenthesized-negative accounting convention, is detected by the leading/trailing-paren check and casts to -500.00 rather than silently losing its sign (verified: without this check the same string casts to a wrong, positive 500.00 with no error flag raised); "1.234,56" (a European locale convention) cleans to "1.23456" under this simple stripping rule and casts to the WRONG number rather than failing outright, which is the real danger of a purely mechanical strip-and-cast approach: it can silently produce a plausible-looking but wrong value instead of a clean failure.

Trade-offs and pitfalls

Because of that European-locale trap, a single global stripping rule is only safe once you've confirmed all your data uses ONE locale convention; if multiple locale conventions are genuinely mixed in the same column, you need to detect which convention a row is using (for example, by checking whether a comma or a period appears last) before choosing how to strip it, rather than applying one rule blindly to everything. The parenthesized-negative case is the same class of risk: any format convention the strip-and-cast approach doesn't explicitly detect will silently produce a plausible but wrong number rather than a clean failure, so before shipping this kind of check, enumerate every convention actually observed in a sample of the real data (currency symbols, thousands separators, parens, locale decimal/thousands swaps) and add an explicit branch for each one rather than assuming the stripping regex alone covers them. The same casting-and-flagging pattern reused here for currency strings applies unchanged to a column that changed storage type over time (VARCHAR with noise in old partitions, clean NUMERIC in new ones): identify the bad old-partition rows with the same TRY_CAST-is-NULL check before planning the backfill.

Direct answer. Placing a filter on the right-hand table's column in WHERE, rather than in the join's ON clause, forces that column to be non-NULL to pass the filter, which throws away every left row that had no match (and therefore had NULL there), functionally converting your LEFT JOIN back into an INNER JOIN.

Structured elaboration. A LEFT JOIN's whole purpose is to preserve unmatched left rows by filling the right side with NULLs. WHERE runs AFTER the join has already produced its rows (including the NULL-padded ones), so a condition like WHERE o.status = 'SHIPPED' evaluates to unknown for any row where o.status is NULL, and WHERE discards anything that isn't TRUE. The fix is to move the condition into the ON clause instead, so it only constrains WHICH right-side rows are allowed to match in the first place, rather than filtering the joined result afterward; unmatched left rows still make it through with NULLs, exactly as a LEFT JOIN should behave.

Worked example. customers(1),(2),(3). orders(order_id=100, customer_id=1, status='SHIPPED'), (order_id=101, customer_id=2, status='PENDING'). Customer 3 has no orders at all.

-- buggy: WHERE demotes this to an INNER JOIN
SELECT c.customer_id, o.order_id
FROM customers c
LEFT JOIN orders o ON c.customer_id = o.customer_id
WHERE o.status = 'SHIPPED';
-- returns only (1, 100). Customers 2 and 3 vanish, even though the LEFT JOIN
-- was written specifically to keep customers with no matching order.

-- fixed: move the condition into ON
SELECT c.customer_id, o.order_id
FROM customers c
LEFT JOIN orders o ON c.customer_id = o.customer_id AND o.status = 'SHIPPED';
-- returns (1, 100), (2, NULL), (3, NULL). Every customer still appears.

Trade-offs and pitfalls. A quick check for this class of bug: run the query with and without the suspect WHERE clause and compare row counts against COUNT(*) on the left table; if the "LEFT JOIN" version returns fewer rows than the left table has, something downstream demoted it. The same bug hides in more places than a literal comparison: o.status IS NOT NULL, o.amount > 0, even a seemingly harmless AND o.region = 'US' in WHERE all have the identical effect. As a review habit, any filter that mentions a LEFT-JOINed table's column should be treated as suspicious until you've confirmed it's meant to belong in ON rather than WHERE.

Direct answer

Use event-driven capture where the source can emit change events, and fall back to polling only where it cannot, since event-driven capture keeps the catalog current with far less wasted work; for a warehouse, subscribe to schema-change and table-creation events where the platform supports them; for a streaming platform, consume its own schema-registry change stream; for flat files in object storage, subscribe to the storage layer's object-created notifications rather than repeatedly listing the bucket. When a source is temporarily unavailable, mark its metadata as stale rather than deleting or blanking it, so the catalog's view degrades gracefully instead of corrupting.

Structured elaboration

Poll versus event-driven, per source type:

• Warehouse: most modern warehouses expose either native change events or queryable system tables for schema and object changes; event-driven capture (subscribing to those events) avoids the catalog re-scanning the entire warehouse's metadata on a fixed interval just to notice one new table.
• Streaming platform: schema changes are typically registered through a schema registry that itself can emit change notifications; consuming that stream keeps the catalog current within moments of a schema evolving, rather than discovering it on the next scheduled poll.
• Flat files in object storage: object storage systems commonly support event notifications on object creation, deletion, or modification; subscribing to those events avoids the alternative of periodically listing an entire bucket (expensive and slow at scale) just to detect new files.

Polling is still the right fallback where a source genuinely offers no event mechanism, or as a periodic reconciliation pass even alongside event-driven capture, to catch any event that was silently dropped (event delivery is rarely perfectly guaranteed, so an infrequent reconciliation poll is the backstop, not the primary mechanism).

Handling a source that's temporarily unavailable without corrupting the catalog's view:

• Never let a failed harvest overwrite existing metadata with an empty or partial result; a harvester that cannot reach its source should skip the update entirely and leave the last-known-good metadata in place.
• Explicitly mark the affected source's catalog entries as stale as of <timestamp> when the last successful harvest is older than the source's expected freshness window, so a consumer sees a clear staleness signal rather than data quietly going out of date with no indication.
• On recovery, reconcile rather than blindly overwrite, compare the newly-harvested state against the last-known state and apply the diff, which also surfaces exactly what changed while the source was unreachable (useful context, not just a return to normal operation).

Worked example

The object-storage source for a raw_events dataset goes unreachable for six hours due to a permissions misconfiguration on the storage bucket. The event-driven harvester, unable to connect, does not clear or overwrite the dataset's existing catalog entry; instead, after the source's expected freshness window (say, new files are expected hourly) passes without a successful harvest, the catalog entry is automatically flagged stale, last confirmed 3 hours ago. An analyst searching the catalog during this window sees the staleness flag alongside the (still-present, still-usable) prior metadata, rather than either a missing entry or metadata that silently looks current when it is not. Once the permissions issue is fixed, the harvester reconnects, event notifications for the files that arrived during the outage are replayed (most object-storage event systems retain a window of undelivered notifications), and the catalog reconciles to the current state, clearing the staleness flag.

Trade-offs & pitfalls

Event-driven capture is not free of gaps, event delivery systems can drop or delay a notification, so relying on it exclusively, with no periodic reconciliation poll at all, risks a catalog that silently drifts without ever showing a staleness flag, because the harvester technically "succeeded" on a smaller set of events than actually occurred. A reconciliation poll set too infrequently defeats its own purpose as a backstop; set too frequently, it reintroduces the cost that event-driven capture was meant to avoid. Marking sources stale rather than removing them protects against corruption but shifts the burden onto consumers to notice and respect the staleness flag, if the search and filtering interface does not surface staleness prominently, an analyst can still act on out-of-date metadata despite the flag technically being present.

Hybrid architecture:

• Ingest: use a streaming layer (Kafka) with lightweight producers.
• Low-latency store: materialize near-real-time aggregates in a fast OLTP/NoSQL store (Redis, RocksDB-backed service, or ClickHouse with TTL) for second-level metrics and drilldowns.
• Analytical store: periodically batch/stream into a columnar warehouse (Snowflake, BigQuery, ClickHouse) for complex ad-hoc queries.
  Schema choices:
• For real-time layer: schema-normalized key-value with pre-aggregated counters and event windows; support fine-grained keys for drilldowns.
• For warehouse: denormalized wide tables or fact+dimension model optimized for columnar scans.
  Syncing: use CDC/streaming transforms (Kafka Streams or Flink) to update real-time store and write compressed events to warehouse. Maintain materialized views and rollups in both stores.
  Query routing: route interactive dashboard queries to real-time store for recent windows (seconds) and to warehouse for deep historical analysis with fallbacks.
  Trade-offs: real-time store optimized for latency and limited ad-hoc flexibility; warehouse for flexible analytics. Use consistent aggregation semantics and eventual convergence strategies.

Direct answer

Model retention and revenue at the PLAN level, not the customer level: a customer who downgrades or pauses is still "retained" as a customer even though their plan-level revenue changed, so the cohort curve needs separate logic for whether the CUSTOMER stayed versus whether the ORIGINAL PLAN'S revenue was retained, expanded, or contracted, with proration and refunds explicitly reconciled rather than assumed away.

Structured elaboration

• Why customer-level and plan-level definitions diverge: a customer who downgrades from a $100/mo plan to a $60/mo plan has NOT churned as a customer (they're still paying, still active), but has produced $40/mo of REVENUE churn on their original cohort's plan-level curve. Conflating the two produces a retention curve that's technically not wrong for either definition alone, but answers neither question cleanly if the two are mixed.
• Renewals, expansions, downgrades, and churn each need their own bucket: renewal (same price continues), expansion (price increases, e.g. an upsell or add-on), downgrade (price decreases but customer remains), and full churn (customer stops paying entirely). A revenue-retention curve typically reports NET revenue retention, which nets expansions against downgrades and churn within the same cohort.
• Proration: a plan change partway through a billing period produces a partial-period charge at each of the old and new price; decide whether the cohort curve attributes that month's revenue at the blended prorated amount or simplifies to whichever plan was active longer that month, and state the choice.
• Multiple overlapping subscriptions per customer (a customer with two products, or a trial running alongside an existing paid plan) require deciding whether the cohort curve is computed per subscription or aggregated per customer; these produce different, both legitimate, answers.
• Validating the accuracy of the resulting curves: reconcile the SUM of a cohort's monthly plan-level revenue at each offset against an independent GAAP (Generally Accepted Accounting Principles)-style recognized-revenue figure -- i.e., the revenue number finance already reports for accounting purposes -- for the same customers and period, since a validated cohort curve should tie out to the finance team's own numbers, not just internally to itself.

Worked example

A cohort of 100 customers who signed up in the same month, each starting on a $50/mo plan. Suppose by month 3: 70 are still on the original $50/mo plan (pure renewal), 10 upgraded to $80/mo (expansion, +$30/mo each), 10 downgraded to $30/mo (downgrade, -$20/mo each), and 10 churned entirely (-$50/mo each). The cohort's month-3 plan-level revenue is $70(50) + 10(80) + 10(30) + 10(0) = 3500+800+300+0 = 4600$, against an original month-0 revenue of $100 \times 50 = 5000$. Net revenue retention at month 3 is $4600/5000 = 92\%$: expansions ($300 extra from upgrades) partially offset downgrades and churn ($700 lost combined), landing at 92% even though 10 of the original 100 customers stopped paying entirely. A customer-level retention curve for the same cohort at month 3 would instead report 90 customers still active (excluding only the 10 who fully churned), a DIFFERENT number answering a different question; presenting only one of these two curves without naming which one it is invites a stakeholder to draw the wrong conclusion from it.

A representative query producing the plan-level cohort curve directly from the schema in the question:

WITH cohort AS (
  SELECT customer_id, DATE_TRUNC('month', start_date) AS cohort_month
  FROM subscriptions
),
monthly_revenue AS (
  SELECT customer_id, DATE_TRUNC('month', event_date) AS revenue_month, SUM(amount) AS revenue
  FROM events
  WHERE event_type = 'payment'
  GROUP BY customer_id, DATE_TRUNC('month', event_date)
)
SELECT c.cohort_month,
  DATEDIFF('month', c.cohort_month, mr.revenue_month) AS month_offset,
  SUM(mr.revenue) AS cohort_revenue,
  COUNT(DISTINCT mr.customer_id) AS paying_customers
FROM cohort c
JOIN monthly_revenue mr ON mr.customer_id = c.customer_id
GROUP BY c.cohort_month, month_offset
ORDER BY c.cohort_month, month_offset;

Run against the 100-customer cohort above, this returns month_offset = 0 with cohort_revenue = 5000 and paying_customers = 100, and month_offset = 3 with cohort_revenue = 4600 and paying_customers = 90, reproducing both the 92% net revenue retention and the 90% customer-level retention computed above directly from the query rather than by hand.

Trade-offs & pitfalls

• A single "retention rate" without specifying customer-level vs. plan-level revenue is ambiguous and easy to misinterpret: always label which definition a chart is using, since the two can diverge substantially (as in the worked example, 90% customer retention vs. 92% net revenue retention are both true simultaneously and mean different things).
• Proration decisions materially change the numbers in a plan-change-heavy business and should be documented as an explicit modeling choice, not left as an implicit side effect of however the query happened to be written.
• Multiple subscriptions per customer is a common real-world wrinkle that a simplified one-subscription-per-customer model will get wrong; decide and document the unit of analysis (subscription vs. customer) up front.

Direct answer

When a WHERE clause mixes AND and OR conditions, correctness depends on how the boolean expression is grouped, not on the order you physically write the conditions in. SQL evaluates the whole boolean expression by its precedence rules (AND binds tighter than OR), and a NULL-aware condition (IS NULL / IS NOT NULL) has to be used deliberately, since NULL never satisfies = or <>. Get the grouping or the NULL handling wrong and the query silently returns the wrong rows with no error.

Structured elaboration

Three pieces make a mixed WHERE clause correct:

1. Equality flag: is_paid = TRUE is a plain boolean check. It works as expected on TRUE and FALSE rows (a NULL is_paid row is excluded too, since NULL = TRUE evaluates to unknown, not true).
2. Date range: prefer a half-open interval, order_date >= start AND order_date < end, over BETWEEN. BETWEEN is inclusive on both ends, which can double-count a row that lands exactly on the boundary between two adjacent ranges.
3. NULL-aware condition: any column that can be NULL (a cancellation timestamp, a discount code) needs IS NULL / IS NOT NULL, not = NULL or <> value. column <> value evaluates to NULL (not TRUE) for a NULL row, so that row is silently dropped even when the intent was to include it.

Where correctness actually gets lost is precedence: AND binds tighter than OR. WHERE a AND b AND c OR d parses as (a AND b AND c) OR d, not a AND b AND (c OR d). Whenever AND and OR sit in the same WHERE clause, wrap the OR branch in explicit parentheses. That grouping is the "structure" that matters. Physical execution order does not: the optimizer is free to reorder predicates as long as the logical result is preserved.

Worked example

Sample data (orders), tested in DuckDB 1.5:

Goal: paid orders placed in Q1 2024 that are either not cancelled, or cancelled but refunded.

Correct (OR branch parenthesized):

SELECT order_id
FROM orders
WHERE is_paid = TRUE
  AND order_date >= DATE '2024-01-01'
  AND order_date <  DATE '2024-04-01'
  AND (cancelled_at IS NULL OR refunded = TRUE)
ORDER BY order_id;

Result: order_id = 1, 2 (2 rows). Order 3 is correctly excluded (cancelled and not refunded), order 4 is excluded (too early), order 5 is excluded (unpaid), order 6 is excluded (too late).

Buggy (same conditions, parentheses removed around the OR):

SELECT order_id
FROM orders
WHERE is_paid = TRUE
  AND order_date >= DATE '2024-01-01'
  AND order_date <  DATE '2024-04-01'
  AND cancelled_at IS NULL OR refunded = TRUE
ORDER BY order_id;

Result: order_id = 1, 2, 5 (3 rows). Order 5 leaks in even though it is unpaid and from March, because the clause is now (is_paid AND date_range AND cancelled_at IS NULL) OR refunded = TRUE, and refunded = TRUE alone satisfies the second half of the OR.

Trade-offs & pitfalls

• Always parenthesize an OR branch that sits inside a longer AND chain, even when the grouping "looks" unambiguous. The next person editing the query may not preserve your intended precedence.
• Prefer half-open date ranges (>= start AND < end) over BETWEEN for timestamp columns, to avoid double-counting or off-by-one-day gaps caused by boundary values.
• column <> value silently excludes NULL rows. If NULLs should be included, add OR column IS NULL explicitly, do not assume <> covers it.
• Some engines will not warn on a missing parenthesis. The query just runs and returns a wrong but plausible-looking row count, which is why this bug class tends to survive casual review.

Direct answer

Migrating from Lambda to Kappa is fundamentally a data-parity project before it's an infrastructure project: the risk isn't standing up a streaming job, it's proving the new single codepath reproduces what the old batch layer was producing, for both live and replayed data, before you retire the batch layer that's been your correctness backstop.

Structured elaboration

Code refactoring: consolidate the batch and speed layer's business logic into one streaming-native implementation. In practice this usually means porting whatever the batch layer does (often SQL or a Spark job) into the stream processor's semantics (windowing, watermarks, state), which is where subtle behavior changes creep in, since "the same aggregation" can mean something different once you have to define how long to wait for late data.

State migration: any state the batch layer implicitly held (via full recompute each run) has to become explicit, durable, checkpointed state in the streaming job. Size this carefully; a batch job that scanned petabytes of history each run was never a good model for what needs to live in a stream processor's state store going forward, so decide what recent window actually needs to be live state versus what can be queried from cold storage.

Reprocessing/backfill plan: this is Kappa's whole value proposition, so prove it works before cutover. Pick a historical window, replay it through the new streaming job from the durable log, and confirm the output matches the old Lambda system's recorded output for that same window, not just that it runs without errors.

Parity tests: run the new Kappa pipeline in shadow, consuming the same live traffic as the existing Lambda system, for long enough to see your full range of edge cases (month boundaries, late data, any known-tricky periods). Diff outputs field by field, not just at an aggregate level, since a matching daily total can hide compensating errors underneath.

Rollback strategy: keep the old Lambda pipeline (or at minimum its batch layer) running and untouched until the new system has passed parity for an agreed bake period, and keep the cutover reversible, meaning downstream consumers should be able to be pointed back at the old output source without a schema change.

Worked example

Suppose the Lambda system's batch layer computes 30-day rolling revenue by user cohort, and the new Kappa job needs to replicate that with event-time windowing and a watermark policy. The parity test replays the last 90 days through the new job and compares its 30-day rolling revenue against the historical batch output, cohort by cohort. If the new job matches on 995 of 1,000 sampled cohort-days and the 5 mismatches all trace to one edge case (users who churned and re-signed up within the window), that's a concrete, fixable code bug (not a shrug-and-ship situation) found before cutover instead of in production.

Trade-offs and pitfalls

The main risk across all of this is declaring victory on aggregate-level parity ("the daily total matches") without checking finer granularity, since two wrong numbers can cancel out. The second risk is under-provisioning replay capacity: if replaying a full backfill window takes days instead of hours because the new streaming engine's state store or checkpointing wasn't sized for a large backfill, you've quietly recreated a batch job's latency inside a system that was supposed to make backfills fast. The third, org-level risk is retiring the old batch layer too early under pressure to "finish the migration"; keep it as a fallback until the new system has survived at least one real edge case (a late-arriving correction, a schema change) in production, not just in the parity test.

Direct answer

Compressing a three-month ramp into three weeks means deliberately not becoming broadly competent and instead becoming narrowly reliable on exactly what the project needs, while being explicit about what I'm skipping and how the resulting gap gets covered, whether that's a reviewer, a narrower scope, or stated uncertainty on anything I can't fully back. I would never let three weeks of learning quietly pass as equivalent to three months; the compression only works if everyone downstream knows what they're actually getting.

What compression actually means

Triage by what the project needs, not by the team's usual onboarding order. A normal three-month ramp typically builds broad familiarity before depth. With three weeks, I invert that: identify the two or three things this specific project actually requires me to be right about, and go deep only there, accepting shallow or absent knowledge everywhere else. If the timeline compressed further, to a single day, the triage gets sharper still: I would ask what one piece of context, if I got it wrong, would sink the project, and spend almost all the time there, explicitly skipping everything else rather than spreading thin.

Name the quality bars I refuse to drop even under compression. Compression is about learning less, not about shipping unverified work. I would still hold the same review and testing standards for anything I produce, even if the compressed ramp buys speed on learning but never on care.

Lean on other people's time, and be honest about the cost. The fastest lever available is borrowing a domain expert's attention instead of self-teaching everything from scratch, but that time is not free. I would be specific with the team about how much of someone's time I'm asking for and for how long, rather than letting it show up later as their own work quietly slipping.

Cover the gap with structure, not bravado. Where I know I'm still shallow, I build in a mandatory review step, narrow the scope of what I own until I catch up, or explicitly flag deliverables as carrying more uncertainty than the team's usual standard, rather than letting a compressed ramp quietly lower the bar without anyone deciding that on purpose.

Worked example

Joining a project three weeks before a launch, with the team's usual ramp closer to three months, I asked the lead directly what single area, if I got it wrong, would actually hurt the launch. The answer was one integration point with a partner system, so I deliberately left everything else about the surrounding codebase thin. I spent roughly half of the three weeks almost entirely on that integration, pairing daily with the engineer who owned it, which meant asking for about six hours a week of her time, made explicit up front rather than assumed. For the parts I stayed shallow on, I did not pretend otherwise: I flagged two areas in my own handoff notes as reviewed by me but not independently verified, and asked for an extra reviewer on anything touching them until I had more time. The launch shipped on schedule; the cost was that a change I made in one of the flagged areas weeks later took noticeably longer because I was still building real familiarity with it, a cost I had knowingly deferred rather than avoided.

Trade-offs and pitfalls

The core trade-off is depth for speed: three weeks buys narrow reliability, not the broad judgment three months would have given, and pretending otherwise is the real risk, not the compression itself. The most common pitfall is letting the compressed timeline quietly lower quality bars along with breadth, when only breadth should be sacrificed. A second pitfall is treating borrowed expert time as free; if it isn't planned and bounded, the person you leaned on absorbs the cost you didn't.