Amazon Staff Data Engineer Interview Preparation Guide

Direct answer

When priorities shift out from under a strategy, the discipline is re-scoring the backlog against the new constraint explicitly, not silently reprioritizing by gut feel, and being willing to say which committed work is paused, not just which new work is added.

Structured elaboration

• Get the new hard constraint stated precisely, in writing, from whoever owns it (legal, a regulator, an acquirer). "Compliance" and "explainability" mean different specific things to different people, and building against a vague version of the requirement wastes the pivot.
• Re-score every active and planned initiative against a small, explicit, written set of criteria weighted toward the new constraint (regulatory impact, reach, effort, and confidence in the estimate, adapted to the moment). The same test works whether the forcing constraint is a regulator, an acquirer, or a hard performance target like sustaining 10x throughput under a fixed budget. Writing the score down is what makes the "what got abandoned" conversation defensible later, rather than looking political.
• The same re-scoring discipline applies even when the shift is internal rather than external: three stakeholders each request a different initiative for the same cycle and there is capacity for only one; an explicit-criteria comparison is what makes that choice defensible instead of a popularity contest.
• Sequence the response across time horizons instead of one big-bang change: an immediate 0-3 month slice covering the non-negotiable, hard-deadline work; a 3-12 month slice building the durable capability (real data lineage and access control, not a one-off report); and a 12-36 month slice for restoring the paused strategic work once the mandatory floor is met.
• Get explicit sign-off from whoever controls funding and headcount (finance, a CTO, or equivalent) on the reprioritization itself, not just on the technical plan. Reallocating people away from committed roadmap items is a resourcing decision, not only a technical one.
• Keep a visible "paused, not cancelled" list. Initiatives dropped for the mandatory work need an owner and a resume trigger, or they quietly become permanently cancelled without anyone deciding that on purpose.

Worked example

A company gets acquired and a new regulator-driven requirement (data retention and explainability of reporting) becomes non-negotiable, on top of an existing roadmap built entirely around growth-facing dashboards. Rather than absorbing the new work as one more backlog item, you convene the people who actually own the requirement, translate it into concrete deliverables (an auditable reporting view, retention automation, documented data lineage), and re-score the backlog against a deadline-driven weighting, pausing lower-urgency growth experiments explicitly rather than letting them slip silently. You phase the response: the auditable view and retention work ship first because they carry the hard deadline; lineage and access-control work that makes future audits cheaper follows once the deadline is met; the paused growth analytics work gets a resume date once the mandatory floor is in place, coming back with a smaller footprint that reuses automation built for compliance rather than restarting from the original scope. Getting explicit funding sign-off matters here specifically because analysts had to move off committed growth work, and that is a resourcing call that needs to be made on purpose, not absorbed silently.

Trade-offs and pitfalls

The main failure is treating a hard external deadline as just another high-priority ticket instead of restructuring the whole plan around it, which produces a roadmap that stays busy but still misses the deadline. The second is never explicitly un-pausing the deferred work: once a strategic initiative is quietly shelved for a mandatory one, it tends to stay shelved unless someone owns bringing it back. The third is picking the requirement's minimum literal interpretation to move fast and then having to redo the work once the fuller requirement is clarified; it is usually cheaper to over-clarify scope with the requirement's owner up front than to guess and rebuild.

Direct answer

A shallow copy (copy.copy) duplicates only the outer container; every element inside it is still the same object shared with the original, so mutating a nested, mutable element (like a list inside a list of lists) is visible through both the original and the shallow copy. A deep copy (copy.deepcopy) recursively duplicates every nested object too, so the original and the deep copy share nothing and mutating one never affects the other.

Structured elaboration

• copy.copy(x) builds a new container of the same type as x and inserts references to x's existing elements; it does not look inside those elements at all. This is a copy of exactly one level.
• copy.deepcopy(x) walks the entire object graph starting from x, copying every nested mutable object it finds (keeping a memo dict internally to handle objects referenced more than once, or referencing themselves, without infinite recursion or duplicating shared objects unnecessarily).
• The distinction only matters for containers holding mutable elements. A list of numbers or strings has nothing for a deep copy to do differently, since numbers and strings are themselves immutable; shallow and deep copies behave identically there. The difference shows up specifically with nested lists, dicts, sets, or custom mutable objects.

Worked example

import copy

orig = [[1, 2], [3, 4]]
sh = copy.copy(orig)
dp = copy.deepcopy(orig)

orig[0].append(99)     # mutate a nested list in place
orig.append([5, 6])    # mutate the outer list

print("orig:", orig)
print("shallow:", sh)
print("deep:", dp)
print("shallow shares inner lists:", sh[0] is orig[0])
print("deep does not:", dp[0] is orig[0])

Output:

orig: [[1, 2, 99], [3, 4], [5, 6]]
shallow: [[1, 2, 99], [3, 4]]
deep: [[1, 2], [3, 4]]
shallow shares inner lists: True
deep does not: False

Two things happened to orig and only one shows up in sh: appending 99 to orig[0] (an in-place mutation of a shared inner list) is visible in sh, because sh[0] and orig[0] are literally the same list object; appending a whole new element [5, 6] to orig itself is not visible in sh, because that mutated the outer list, and sh is already a distinct outer list object from orig (that is what copy.copy gave it). dp reflects neither mutation, since every level was duplicated independently.

Trade-offs & pitfalls

• Reach for copy.copy (or, for a list specifically, the equivalent and more common list(x) or x[:]) when you only need a new top-level container and either the elements are immutable, or you deliberately want mutations to shared nested elements to be visible everywhere (an intentional sharing pattern, not an accident).
• Reach for copy.deepcopy when you need full isolation: a function that takes a nested structure, wants to experiment with modifying it, and must not have those changes leak back into the caller's original data.
• deepcopy costs time and memory proportional to the entire object graph size, not just the outer container, so calling it on a large or deeply nested structure (or one containing a large array-like object) can be far more expensive than a shallow copy; if only isolation of a specific nested piece is needed, copying just that piece is usually cheaper than deep-copying the whole structure around it.
• A very common real bug is assuming list(x) (or slicing x[:]) gives full independence from x when x is a list of lists (or list of dicts): it does not, since both are shallow-copy operations at the outer level only, which is exactly the surprise this question is testing for.

Direct answer

OpenLineage/Marquez, DataHub, and Apache Atlas differ most on maturity and operational weight: OpenLineage/Marquez is a lightweight, standards-based approach focused specifically on lineage capture with a simple metadata model, DataHub is a fuller-featured metadata platform (lineage plus catalog, discovery, and governance) with a more complex deployment footprint, and Apache Atlas is the most operationally heavy, historically tied closely to the Hadoop ecosystem, with a rich but more rigid metadata/classification model.

Structured elaboration

• Metadata and lineage model: OpenLineage defines a standard, portable EVENT format (job started, job completed, with input/output datasets) that many tools can emit natively, and Marquez is the reference implementation storing and serving that data as a graph; DataHub has a broader entity model covering datasets, dashboards, pipelines, and people, with lineage as one of several first-class relationship types; Atlas has a rich, extensible TYPE SYSTEM for classification and governance tagging, with lineage captured as part of that broader classification model, which is powerful but requires more upfront modeling effort.
• Ease of instrumentation: OpenLineage's growing ecosystem of native integrations (Spark, Airflow, dbt) means many jobs can emit lineage with configuration alone, no custom code; DataHub similarly has a metadata-ingestion framework with many connectors, though standing up the metadata GRAPH and getting teams to actively use its catalog/discovery features (not just lineage) is a bigger adoption lift; Atlas typically requires more deliberate integration work and is most natural if you're already deep in a Hadoop/Hive-centric stack.
• Operational maturity/scalability: Marquez (OpenLineage's reference server) is lightweight and easy to run for lineage alone, but you may outgrow its feature set if you want a full catalog/discovery experience later; DataHub is more operationally involved to run (more moving parts: a metadata service, search index, graph store) but delivers a fuller platform in one system; Atlas has the longest operational track record in large, established Hadoop-ecosystem deployments but is a heavier system to stand up and maintain than either of the other two, and is less actively evolving relative to OpenLineage's momentum as an emerging cross-tool standard.

Worked example

For a mid-size, fast-growing analytics company, OpenLineage plus Marquez is generally the better starting recommendation: the growing native-integration ecosystem (Spark, Airflow, dbt all have OpenLineage support) means lineage capture can be stood up incrementally, tool by tool, without a large upfront platform investment, and the lighter operational footprint matches a growing company's constrained platform-engineering headcount. If, later, the organization also wants full data-catalog and discovery features (search, business glossary, access-request workflows) beyond pure lineage, migrating to or layering DataHub becomes the natural next step, and OpenLineage's standard event format is portable enough that lineage captured for Marquez isn't fully wasted work if that migration happens, since DataHub also consumes OpenLineage-format events natively.

Trade-offs and pitfalls

The recommendation hinges specifically on "fast-growing, mid-size" in the prompt, a large, already-Hadoop-centric enterprise with existing Atlas investment and dedicated platform staff might reasonably stick with Atlas rather than migrating; the "right" choice is genuinely dependent on existing stack and team size, not a universal ranking. The pitfall in this kind of comparison is treating "which tool" as the main decision when the harder, more consequential choice is often GRANULARITY (table-level versus column-level lineage) and CAPTURE METHOD (automatic instrumentation versus manual annotation), decisions that matter more for whether the lineage graph is actually trustworthy than which specific tool implements the graph.

Direct answer

Spot is the cheapest option on paper, but the deadline matters as much as the price. Given the historical retry overhead is only about 10%, the interruption-adjusted spot cost still comes in well below on-demand or reserved, so spot is the right primary choice for this fault-tolerant batch job, with a fallback to on-demand if progress tracking shows the job is at risk of missing the 48-hour window.

Structured elaboration

How to compute each option correctly

• On-demand and reserved are straightforward: hourly rate times hours needed.
• Spot needs an adjustment: the quoted spot rate applies to the hours actually run, but interruptions mean you re-run some work, so the effective hours are higher than the raw 1,000-hour requirement. The retry overhead has to be added to the hour count before multiplying by the spot rate, not applied as a discount on top of a naive calculation.

$\text{Monthly}_{\text{spot}} = \text{Hours} \times (1 + \text{RetryOverhead}) \times \text{SpotRate}$

The actual decision rule isn't "pick the cheapest", it's "pick the cheapest option that still comfortably meets the deadline with acceptable risk." That means checking whether the interruption-adjusted runtime still fits inside 48 hours with margin, not just comparing the three monthly dollar figures.

When you'd override the cheap answer:

• The job isn't actually resumable (no checkpointing), so an interruption means restarting from scratch rather than continuing, which would blow well past the modeled 10% overhead.
• The deadline is tight relative to how variable interruptions have historically been, leaving little margin if a bad week of interruptions happens to land during this run.
• A downstream process depends on a guaranteed completion time rather than a "usually finishes on time" completion time.

Worked example

Comparisons:

$\$60.00 - \$22.00 = \$38.00, \quad \frac{38}{60} \approx 63.3\% \text{ cheaper than reserved}$

$\$100.00 - \$22.00 = \$78.00, \quad 78\% \text{ cheaper than on-demand}$

$\$100.00 - \$60.00 = \$40.00, \quad 40\% \text{ cheaper than on-demand}$

Recommendation: use spot as the primary option, since $22/month with a comfortable 10% historical overhead still leaves plenty of margin inside a 48-hour window for a fault-tolerant, presumably parallelizable batch job. Pair it with a simple fallback policy: monitor remaining work against time-to-deadline, and if the trend suggests the job won't finish in time, fall back to on-demand (or the already-purchased reserved capacity, if any) to guarantee completion rather than risk missing the deadline to save the last few dollars.

Trade-offs and pitfalls

• Comparing sticker prices ($0.02 vs. $0.10) without the retry adjustment understates spot's real cost. The correct comparison already accounts for it above, but it's a common shortcut that overstates the savings.
• Choosing spot for a job that isn't actually fault-tolerant is the biggest risk. If the job can't checkpoint or resume cleanly, an interruption means restarting from zero, and the historical 10% overhead figure, which presumably reflects jobs that can resume, would badly understate the real cost and risk for this one.
• Committing to a 1-year reserved instance for an occasional batch job locks in cost for a usage pattern that doesn't need it. Reserved capacity is the right tool for steady-state, predictable, always-on usage, not for a job that runs occasionally and can tolerate flexible scheduling.

The four Golden Signals are the metrics Google's SRE practice recommends watching first to know whether a user-facing service is healthy: latency (how long requests take), traffic (how much demand the service is under), errors (the rate of requests that fail), and saturation (how full the service's most constrained resource is). The idea is that if you can only instrument four things for a new service, these four give you the fastest read on user-visible health.

The four signals, one metric each, and why each is insufficient alone

Why they're used together

Each signal answers a different question (is it slow, is it busy, is it failing, is it running out of room), and the same underlying incident often shows up differently in each: a memory leak first appears as rising saturation, then rising latency as garbage collection pressure builds, and only later as errors once requests start timing out. Watching all four catches the problem earlier than waiting for the error signal alone.

Worked example

A checkout service starts a slow memory leak after a deploy. Saturation (heap usage) climbs steadily from 40% to 85% over six hours with no visible impact on error rate. Latency's p99 begins rising around the 70% saturation mark as garbage-collection pauses lengthen, while errors stay near zero the whole time. An on-call engineer watching only the error-rate dashboard would see nothing wrong until the service hits an out-of-memory crash; watching saturation and latency together would have surfaced the leak roughly five hours earlier.

Trade-offs and pitfalls

The Golden Signals tell you that something is wrong and roughly where (which of four broad categories), but they do not replace deeper instrumentation (distributed tracing, structured logs) needed to find why. A common pitfall is picking a saturation metric that isn't actually the service's bottleneck (watching CPU when the real constraint is a database connection pool), which gives a false "all healthy" reading right up until the real resource runs out. For batch or streaming systems, the same four ideas apply but the concrete metrics differ: "traffic" becomes throughput (rows or events per minute), and "latency" becomes end-to-end freshness (how stale the data is), not per-request response time.

Direct answer
A repeatable way to shape a real story into a principle-mapped interview answer: start from the story, not the principle; identify which one or two principles it most naturally demonstrates; structure the telling so the actions carry the evidence rather than announcing the principle by name; close with a concrete, ideally measurable result; and only state the principle's name explicitly if the interview format specifically calls for it.

Structured elaboration

1. Inventory first. Write down six to ten real situations spanning different flavors of experience (a technical trade-off, a disagreement, a mistake, a moment of leading without formal authority, a customer-facing choice).
2. Map second. For each story, ask what your actions actually demonstrated, rather than starting from which principle you want to show. Mapping from story to principle, not the reverse, keeps the story honest.
3. Structure with situation, task, action, result, and put roughly 60 to 70 percent of the telling time in the action section, since that is where the principle actually shows up.
4. Quantify the result where you honestly can. Where you can't, describe a concrete, verifiable change instead of a vague feeling of success.
5. Name the principle explicitly only if the format calls for it. Some interviewers want you to state it directly, in which case one closing sentence is enough; narrating the principle's name throughout reads as reciting rather than demonstrating.

Worked example
Consider a story about restoring a degraded service faster than the standard escalation path would have. Situation: a service degraded during a high-traffic period. Task: the candidate was the person on point. Action: rather than escalating immediately and waiting, they spent the first several minutes gathering the most likely signals, formed a hypothesis, tested it with a small, reversible change, and escalated only once they had evidence rather than a guess. Result: the issue was resolved well inside the window that would have triggered a customer-facing incident, and the candidate wrote up the diagnostic path afterward so the next person facing the same symptom could skip the initial investigation. If the interviewer's principle is framed around ownership or thorough investigation, it is the methodical hypothesis-testing and the follow-up write-up, not a sentence claiming the principle, that demonstrate it.

Trade-offs and pitfalls
Repeating the principle's name throughout a story ("this shows my ownership, which is also ownership because...") reads as reciting rather than demonstrating; state it once, if at all. Choosing a story because it sounds impressive rather than because it honestly demonstrates the specific actions a principle cares about is a common mismatch that a practiced interviewer will probe past. Time-boxing also matters: a detailed answer that never reaches a result is a frequent failure mode, so keep the action section rich but always land on a result.

Below is a practical leveling and development framework you can apply to a 50-person data engineering org with platform and product-aligned teams. Levels are parallel for ICs and managers so people can advance without switching tracks.

IC TRACK (Individual Contributor)

• Data Engineer I (Junior)
  • Expectations: implement and test ETL tasks, follow established patterns, fix bugs, document work.
  • Development moves: master the codebase, unit testing, basic SQL/Spark; pair-program with senior; own small pipeline.
• Data Engineer II (Mid)
  • Expectations: deliver end-to-end pipelines, optimize performance, handle monitoring/alerting, triage data quality incidents.
  • Development moves: lead a small project, write runbooks, improve observability, mentor interns.
• Senior Data Engineer (Senior)
  • Expectations: design robust data models, own cross-service ETL architecture, influence platform choices, guide best practices.
  • Development moves: lead cross-team projects, present RFCs, drive performance/ cost improvements, coach others.
• Staff Data Engineer (Principal)
  • Expectations: set platform standards, design systems for scale, define team roadmaps, unblock multiple teams, technical evangelist.
  • Development moves: publish design docs, shape hiring and architecture, influence product strategy, lead large migrations.
• Distinguished / Architect (Expert)
  • Expectations: organization-wide technical leader, long-term strategy, auditability/governance standards, represent org externally.
  • Development moves: lead org-level initiatives, research and introduce new paradigms, mentor other senior leaders.

MANAGER TRACK

• Engineering Lead / First-line Manager
  • Expectations: manage 1-3 engineers, run 1:1s, manage delivery, remove blockers.
  • Development moves: learn hiring basics, feedback conversations, sprint planning ownership.
• Engineering Manager
  • Expectations: own one team (~5-8), career development for reports, roadmap delivery, cross-functional partnership with PMs/data science.
  • Development moves: run performance reviews, set OKRs, coach technical leads, handle escalations.
• Senior Engineering Manager
  • Expectations: manage multiple teams or larger team (~10+), influence platform/product alignment, resource planning.
  • Development moves: manage managers, strategic planning, budget and hiring forecasts, stakeholder influence.
• Director of Data Engineering
  • Expectations: oversee platform vs product org splits, drive hiring plan, define operational SLAs and governance.
  • Development moves: partner with CTO/Head of Data, set multi-quarter strategy, represent team to execs.
• VP / Head (if applicable)
  • Expectations: own entire data engineering org strategy, org design, cross-org initiatives, P&L influence.
  • Development moves: business strategy, board-level communications, external partnerships.

RECOMMENDED CROSS-TRACK DEVELOPMENT (applies to both)

• Technical breadth: cloud services, Spark optimizations, data modeling, infra-as-code.
• Leadership skills: communication, stakeholder management, roadmap prioritization.
• Visibility: present postmortems, RFCs, brown-bags; publish team metrics (latency, cost, data quality).
• Mentorship & hiring: interview, onboard, and mentor; run projects that demonstrate impact (e.g., cost reduction, pipeline reliability).
• Metrics orientation: measure product impact (time-to-insight, SLA adherence, MTTR) and tie work to business outcomes.

Implementation tips:

• Make leveling rubric explicit with examples of signals of readiness.
• Encourage 1:1 career plans and a mix of stretch projects + formal training.
• Allow lateral moves between platform and product teams for breadth.

Direct answer. A managed relational database service (AWS RDS, Azure Database, Cloud SQL) handles patching, backups, and failover for you at the cost of some configuration flexibility and a service markup; a self-managed relational database on compute instances gives full control over configuration and licensing at the cost of your team owning every operational task. For most migrations, the managed service is the safer default, and self-managing should require a specific, well-justified reason.

Structured elaboration.

Worked example. For a 50TB OLTP migration, propose a recommendation framework based on three questions: does the workload need database-engine customization the managed service does not expose (a specific extension, a non-standard configuration), does the team have dedicated database-operations expertise to run self-managed infrastructure reliably at this scale, and does the organization's compliance posture require infrastructure control the managed service cannot provide? If the answer to all three is no, which is the common case, recommend the managed service: at 50TB, the operational risk of a self-managed HA and backup strategy failing during a real incident usually outweighs the cost savings, and the managed service's built-in point-in-time recovery and multi-AZ failover directly address the two riskiest parts of running a database at that scale. If the team already runs a mature self-managed database practice and needs a specific customization the managed service restricts, self-managing may be the right call, but that should be an explicit, justified exception rather than a default.

Trade-offs and pitfalls. A common mistake is comparing only the hourly instance-cost line item, which almost always favors self-managing, while ignoring the engineering time spent on patching, backup testing, and incident response, which almost always favors the managed service once properly costed. The opposite mistake is defaulting to managed without checking whether the specific engine configuration the application needs is actually supported, only to discover the restriction after migration has already started.

Situation: A junior engineer on my team struggles to write automated tests for ETL jobs — tests are missing, flaky, or only cover happy paths. I designed a three-month mentoring plan combining feedback, hands-on practice, SMART goals, and objective measurement.

Structure of feedback & learning sessions:

• Weekly 60-min touchpoints: 10-min recap by mentee, 30-min focused lesson (testing patterns, fixtures, mocking I/O), 20-min pairing on code or test reviews.
• Bi-weekly 45-min async code-review with annotated feedback using a checklist (naming, isolation, determinism, speed).
• Monthly demo: mentee presents one tested ETL component and lessons learned.

Hands-on exercises (progressive):

• Week 1–2: Write unit tests for individual transform functions (pure functions, edge cases).
• Week 3–4: Add tests that mock data sources/sinks (use local fixtures, pytest + moto/localstack for AWS).
• Month 2: Integration tests for small pipeline runs against sample datasets (Spark local mode or Dockerized DB).
• Month 3: End-to-end regression test around a full pipeline, plus flaky-test debugging and CI integration.

SMART goals (3-month):

• Specific: Increase test coverage for assigned ETL job from current X% to ≥75%.
• Measurable: Add ≥12 meaningful tests (unit/integration), reduce flaky tests to 0.
• Achievable: Incremental tasks and pairing support.
• Relevant: Tests validate critical transformations, schema changes, and failure modes.
• Time-bound: Reach targets within 12 weeks.

Objective measurement:

• Quantitative: test coverage %, number of tests added, CI pass rate, average test runtime.
• Qualitative: code-review checklist scores (start vs. end), reduction in post-deploy incidents tied to covered ETL.
• Weekly logs: track progress per exercise and blockers.

Example metrics to hit by week 12:

• Coverage ≥75% on target job, 12+ tests (6 unit, 4 integration, 2 e2e), CI green rate 98%, zero production data-quality incidents from that pipeline.

Why this works:

• Frequent, structured feedback builds habits; progressive exercises scaffold skill; SMART goals give clarity; objective metrics prove improvement and inform next steps.

Goal: support median order value per customer approximate and exact on demand. Strategies:

1. Approximate: per-customer sketches. Maintain streaming quantile sketches (e.g., t-digest or q-digest) per customer updated as orders arrive. Store serialized sketch in a customer_agg table; querying median = merge sketches and extract 0.5 quantile — very fast and low storage.
   Schema:

• customer_metrics(customer_id PK, orders_count INT, total_amount NUMERIC, tdigest BYTEA, last_updated)
  Writes: update tdigest in place or via append-and-merge in background for concurrency.

2. Exact on-demand: maintain recent order list or sampled materialized table. Options:

• Precompute exact medians for high-activity customers by maintaining an ordered data structure (e.g., skiplist) in an external store (Redis sorted set) to compute exact median in O(log n).
• For on-demand exact compute: run a database query using percentile_cont over partitioned data or use external MapReduce job; limit cost by computing only for requested customers and caching results.
  Hybrid approach:
• Use t-digest for interactive dashboards and alerts.
• For ad-hoc exact needs, run exact computation asynchronously and cache result; for large customers keep exact structure.
  Trade-offs:
• t-digest: fast, low memory, small error (configurable), merges easily—good for production dashboards.
• Exact computation: expensive for large histories; use only for small/high-value customers or infrequent reports.
  Implementation notes:
• Provide TTL and compaction for sketches to bound storage; ensure deterministic merging and versioning of sketch library.
• Evaluate approximation error empirically and surface confidence intervals to consumers.