Senior Full-Stack Developer Interview Preparation Guide - Spotify
Spotify's interview process for Senior Full-Stack Developer roles typically follows a structured funnel: an initial recruiter screening call, followed by 1-2 phone technical screens, and then 5-7 onsite rounds covering coding, system design, authorization domain expertise, API design, behavioral assessment, and cross-functional collaboration. The process emphasizes both backend systems thinking and full-stack versatility, with special attention to authorization/IAM systems (based on the User Platform team focus), REST API design patterns, and real-world scalability challenges.
Interview Rounds
Recruiter Screening
What to Expect
Initial conversation with Spotify recruiter to assess basic fit, background, career motivations, and openness to the role. This round confirms your interest in the authorization/IAM domain, validates that you have 5+ years of relevant experience, and discusses compensation and logistics (particularly relevant for the Toronto location). The recruiter will gauge your communication style and cultural alignment with Spotify's values around inclusion and collaboration.
Tips & Advice
Be authentic about your experience and career goals. Emphasize any experience with identity management, authorization systems, or working on platform infrastructure. Highlight previous roles where you worked across multiple layers of a system. Ask thoughtful questions about the team structure, the role's impact, and career growth opportunities. Mention if you have familiarity with Spotify's product and values.
Focus Topics
Motivation & Fit
Articulate why Spotify appeals to you, what excites you about platform infrastructure work, and how this role aligns with your career goals.
Practice Interview
Study Questions
Authorization/IAM Domain Interest
Explain your understanding of or interest in authorization systems, customer identity management, or access control. If you lack direct experience, discuss how your backend systems background translates.
Practice Interview
Study Questions
Full-Stack Experience Overview
Summarize 2-3 key projects where you worked across frontend, backend, and database layers. Explain what you learned about client-server complexity.
Practice Interview
Study Questions
Professional Background & Experience
Clearly articulate your 5+ years of experience, highlighting transitions between roles and growth trajectory. Emphasize full-stack projects where you owned end-to-end features.
Practice Interview
Study Questions
Technical Phone Screen - REST API & Backend Systems
What to Expect
First technical screening call (typically 60 minutes) where you'll discuss REST API design, backend architecture, and real-world implementation challenges. The interviewer will present scenario-based questions about designing APIs with complex requirements, handling authorization flows, pagination strategies, caching decisions, and database optimization. You may be asked to sketch architectural decisions on a whiteboard or shared document. This round evaluates your ability to reason about scalability, consistency, and practical trade-offs.
Tips & Advice
Ask clarifying questions before diving into solutions—clarify scale requirements, consistency needs, and whether this is internal or public-facing. Use the STAR method to discuss previous API design work. Be prepared to code simple REST endpoint implementations. Discuss caching strategies (Redis, browser caches), pagination approaches (cursor-based vs offset), and error handling patterns. Show familiarity with OpenAPI/Swagger. For authorization APIs, discuss secure token handling and session management. Reference Spotify's scale (700M+ users) to show you're thinking about real-world constraints.
Focus Topics
API Documentation & Error Handling
Explain how you'd document APIs (OpenAPI/Swagger), define error codes, and handle edge cases. Show you can write error responses that help clients debug issues.
Practice Interview
Study Questions
Caching Strategies (Redis, HTTP Caching, Database)
Know when and how to cache. Understand HTTP cache headers, Redis usage patterns, cache invalidation strategies, and cache coherence. Discuss trade-offs between freshness and performance.
Practice Interview
Study Questions
Backend Performance Optimization & Database Indexing
Discuss how you've optimized slow APIs (adding indexes, query rewrites, denormalization decisions). Know how to analyze query performance and make scaling decisions.
Practice Interview
Study Questions
Complex Filtering & Search API Design
Design APIs that support multi-parameter filtering (e.g., product search with price ranges, categories, sorting). Understand faceted search, cursor-based pagination, and efficient query execution at scale.
Practice Interview
Study Questions
Authorization & Security in APIs
Discuss JWT/OAuth patterns, token refresh flows, API key management, rate limiting, and how to design APIs that enforce authorization boundaries securely without sacrificing performance.
Practice Interview
Study Questions
REST API Design Principles & Patterns
Understand RESTful architecture principles (client-server separation, stateless, cacheable, layered system, uniform interface). Be able to design API endpoints that follow REST conventions and scale to millions of requests.
Practice Interview
Study Questions
Technical Phone Screen - Full-Stack Implementation & Code
What to Expect
Second technical phone screen (typically 60 minutes) focused on hands-on coding and full-stack implementation. You'll likely complete 1-2 coding problems that require both backend logic and frontend integration considerations. Problems may involve implementing user authentication flows, building features that touch database-to-UI layers, or solving problems where you need to consider both server and client performance. Expect real-time coding on a shared IDE or whiteboard, with emphasis on clean code, testing, and discussing trade-offs.
Tips & Advice
Write clean, production-ready code. Think out loud about design decisions: data structures, error handling, edge cases. For authorization contexts, consider security implications (don't hardcode secrets, validate inputs). Discuss testing strategy—how would you test this? Show familiarity with both backend languages (Java preferred per job posting) and frontend frameworks (React, TypeScript, Next.js). When working on full-stack problems, explicitly discuss the client-server boundary and state management concerns. Ask for hints if stuck—interviewers value problem-solving approach over perfect solutions.
Focus Topics
Code Quality, Testing & Production Readiness
Write code with error handling, logging, and unit test consideration. Discuss how you'd make code production-ready (validation, security, observability).
Practice Interview
Study Questions
React/TypeScript Frontend Development
Understand React component design, state management (props, hooks, context), TypeScript typing. Be able to build forms, handle async operations, and manage user interactions.
Practice Interview
Study Questions
Client-Server Complexity & State Management
Understand the boundary between client and server concerns. Discuss when logic belongs on backend vs frontend, how to handle network failures, optimistic updates, and data synchronization.
Practice Interview
Study Questions
Java Backend Development & SQL Queries
Demonstrate proficiency in Java for backend systems. Write clean, idiomatic Java code. Understand SQL query optimization, joins, indexing strategies, and how to debug slow queries.
Practice Interview
Study Questions
Full-Stack Feature Implementation (Database to UI)
Demonstrate ability to own features end-to-end: design database schema, write backend API, implement frontend UI, handle error states, consider performance across layers.
Practice Interview
Study Questions
Onsite Technical Interview - System Design
What to Expect
In-person or virtual system design interview (typically 60-75 minutes) where you'll architect a large-scale system related to authorization, identity, or access management. You might be asked to design: a user authentication system for Spotify's 700M+ users, an authorization policy evaluation engine, an OAuth provider, or a credential management system. Expect to make architectural decisions on databases (SQL vs NoSQL), caching layers, service boundaries, API contracts, and scaling considerations. Interviewers value your ability to ask clarifying questions, identify constraints, make trade-offs explicitly, and consider operational concerns (monitoring, debugging, capacity planning).
Tips & Advice
Start with questions: What's the scale? What are consistency requirements? Is this internal or public-facing? Sketch your architecture incrementally, explaining each decision. For authorization systems, discuss how to handle permission checks at sub-100ms latencies at global scale. Discuss data consistency models—is eventual consistency acceptable? How would you handle cache invalidation? Talk about monitoring and debugging strategies. Be prepared to pivot when asked 'what if we had 10x more users' or 'what if this needs to be real-time?' Show you understand distributed systems trade-offs (CAP theorem, idempotency, retries).
Focus Topics
API Design for System Components
Design clean, scalable APIs between services in your architecture. Consider versioning, backward compatibility, and error handling.
Practice Interview
Study Questions
Database Selection & Query Optimization
Choose appropriate databases (PostgreSQL, DynamoDB, etc.) for different use cases. Discuss indexing strategies, query optimization, and data modeling for scale.
Practice Interview
Study Questions
Caching Architecture at Scale
Design multi-layer caching strategies (CDN, application cache, query cache). Discuss cache invalidation patterns, thundering herd problems, and measuring cache effectiveness.
Practice Interview
Study Questions
Operational Concerns & Reliability
Discuss monitoring strategy, alerting thresholds, debugging tools, capacity planning, graceful degradation, and how you'd handle failures in authorization systems.
Practice Interview
Study Questions
Distributed System Fundamentals & Scalability
Understand service decomposition, load balancing, replication, sharding, consistency models (strong vs eventual), and how to make systems scale to billions of operations.
Practice Interview
Study Questions
Authorization & Identity System Architecture
Design scalable authorization systems that handle permission checks, role-based access control (RBAC), attribute-based access control (ABAC), or policy evaluation at scale. Consider latency-critical paths and cache invalidation.
Practice Interview
Study Questions
Onsite Technical Interview - Backend Deep Dive
What to Expect
Technical interview (60-75 minutes) diving deep into backend system design, architecture patterns, and operational complexity. You might design a highly-available service, discuss trade-offs in data consistency, microservices architecture, event-driven systems, or production challenges you've solved. This round evaluates your understanding of backend concerns: request routing, connection pooling, transaction handling, error recovery, observability, and operational debugging. Expect detailed questions about how you'd build reliable systems that handle failures gracefully.
Tips & Advice
Draw from real production challenges you've faced. Use specific examples of systems you've built, scaled, or debugged. Discuss the SOLID principles and how they apply to backend architecture. Be prepared to discuss service-oriented vs monolithic approaches and trade-offs. Talk about observability (logging, metrics, tracing) as a first-class concern, not an afterthought. Discuss failure modes and how you'd make systems resilient (timeouts, retries, circuit breakers, bulkheads). Show you've thought about operational burden and how to reduce on-call pain.
Focus Topics
Scaling Backend Systems
Discuss vertical vs horizontal scaling, load balancing, database sharding, and capacity planning. How would you identify scaling bottlenecks?
Practice Interview
Study Questions
Observability, Monitoring & Debugging
Discuss logging strategy, metrics collection, distributed tracing, and how you'd debug production issues in complex systems. What would you monitor for an authorization service?
Practice Interview
Study Questions
Resiliency Patterns & Error Handling
Design systems that fail gracefully using timeouts, retries, circuit breakers, bulkheads, and fallback strategies. Discuss how to test failure scenarios.
Practice Interview
Study Questions
Performance Optimization & Latency Reduction
Strategies for reducing latency: request batching, async processing, caching, database optimization, connection pooling, and profiling techniques.
Practice Interview
Study Questions
Data Consistency & Transaction Management
Discuss ACID properties, eventual consistency, distributed transactions, saga patterns, and how to choose consistency models for different use cases.
Practice Interview
Study Questions
Service-Oriented Architecture & Microservices
Design backend systems using microservices patterns. Discuss service boundaries, communication protocols (REST vs gRPC vs messaging), and managing complexity in distributed systems.
Practice Interview
Study Questions
Onsite Technical Interview - Full-Stack Problem Solving
What to Expect
Technical interview (60-75 minutes) focused on end-to-end feature implementation across multiple layers. You'll work on a realistic problem that spans frontend, backend, and database concerns. Example: implement a feature where users manage their connected apps and permissions, requiring both UI components and backend API work. You'll write code (backend and frontend), discuss design decisions, and explain how layers interact. Interviewers assess whether you can own complete features, make good architectural choices across layers, and deliver production-quality solutions.
Tips & Advice
Break the problem into frontend, backend, and database components. Write clean, idiomatic code. For authorization contexts, think about security: validate inputs on backend, never expose secrets, use appropriate auth mechanisms. Discuss error handling: what if the backend call fails? How do you show errors to users? Think about edge cases and how your frontend handles them. Discuss testing: what would you test? At this level, interviewers expect you to consider trade-offs holistically, not just write code.
Focus Topics
Testing Strategy Across Layers
Discuss how you'd test the feature: unit tests for components, backend API tests, integration tests, and end-to-end tests. What's your priority?
Practice Interview
Study Questions
Next.js & TypeScript for Full-Stack Development
Show familiarity with Next.js (fullstack framework) and TypeScript typing. Understand server-side rendering, API routes, and how Next.js simplifies full-stack development.
Practice Interview
Study Questions
Frontend State Management & Data Fetching
Manage component state, handle async data fetching, show loading/error states, and discuss state management tools (hooks, context, or state management libraries).
Practice Interview
Study Questions
End-to-End Feature Ownership (Database to UI)
Demonstrate ability to own a complete feature: schema design, backend API, frontend UI, error handling, and testing. Show you can navigate complexity across all layers.
Practice Interview
Study Questions
Authorization & Permissions in Feature Implementation
Implement features with proper authorization checks. Understand where authorization logic belongs (frontend vs backend), how to prevent privilege escalation, and secure API design.
Practice Interview
Study Questions
Onsite Behavioral & Impact Interview
What to Expect
Behavioral interview (45-60 minutes) exploring your experience, decision-making, teamwork, and impact. You'll discuss past projects using the STAR method, how you've handled conflicts, made technical trade-offs, contributed to culture, and grown as an engineer. Questions may focus on: leading cross-functional initiatives, mentoring junior engineers, making architectural decisions, navigating ambiguity, and delivering impact at scale. Interviewers assess whether you're a senior engineer who can influence direction, not just execute tasks.
Tips & Advice
Prepare 4-5 stories demonstrating senior-level impact: owning a large project, mentoring teammates, making a key architectural decision, navigating ambiguity, and recovering from failure. Use the STAR method (Situation, Task, Action, Result). Focus on your leadership and influence, not just technical skills. Discuss how you enabled others, drove decisions, and delivered measurable impact. Reference Spotify's values (inclusion, collaboration, innovation). Prepare thoughtful questions about the team, their challenges, and culture. Be authentic—interviewers want to understand who you are, not just your resume.
Focus Topics
Learning from Failure & Resilience
Share a project or initiative that didn't go as planned. What did you learn? How did you respond? How did you prevent similar issues in future?
Practice Interview
Study Questions
AI Fluency & Modern Development Practices
The job posting mentions 'AI Fluency.' Discuss how you've used AI tools (LLMs, GitHub Copilot, etc.) to improve productivity. Are you experimenting with AI in your work?
Practice Interview
Study Questions
Navigating Technical Trade-Offs & Decisions
Tell a story about making a significant technical decision (e.g., choosing a database, refactoring legacy code, or architectural redesign). How did you gather input? How did you communicate the decision?
Practice Interview
Study Questions
Mentorship & Growing Others
Discuss how you've mentored junior engineers, helped teammates grow, or contributed to hiring. What's your philosophy on mentorship? Give specific examples of impact.
Practice Interview
Study Questions
Cross-Functional Collaboration & Influence
Share examples of working with product managers, designers, data analysts, or other teams. How did you align on goals? How did you influence technical decisions across team boundaries?
Practice Interview
Study Questions
Project Ownership & Delivery at Scale
Discuss a significant project you led end-to-end. What was the scope? How did you break it into phases? What challenges did you face and how did you overcome them? What was the impact?
Practice Interview
Study Questions
Onsite Culture & Leadership Interview
What to Expect
Final onsite interview (45-60 minutes) with a senior engineer or manager focused on cultural fit, values alignment, and long-term potential. You'll discuss your career aspirations, what kind of environment you thrive in, how you handle ambiguity and change, and your perspective on engineering excellence. The interviewer assesses whether you align with Spotify's values (inclusion, collaboration, creativity, passion for music/audio), can navigate fast-changing environments, and see yourself growing within the company. This is also your opportunity to ask deep questions about the team, culture, and career trajectory.
Tips & Advice
Be genuine and reflective. Discuss what you value in engineering: quality, speed, learning, impact, or culture? How do you balance trade-offs? Talk about your career aspirations—are you interested in team leadership, technical depth, or both? Show you've thought about why Spotify specifically appeals to you beyond compensation. Research Spotify's mission (unlock creative potential, give artists opportunity to live off art, give fans access to creators) and discuss how it resonates. Ask thoughtful questions about team structure, technical challenges, and career paths. This round is mutual evaluation—you're assessing if Spotify is right for you.
Focus Topics
Navigating Ambiguity & Fast-Moving Environments
Tell a story about working in an ambiguous situation with unclear requirements or rapidly changing priorities. How did you approach it? What did you learn?
Practice Interview
Study Questions
Engineering Excellence & Quality Standards
What does 'engineering excellence' mean to you? How do you balance speed with quality? What practices or standards do you advocate for in your teams?
Practice Interview
Study Questions
Growth Mindset & Learning Orientation
Discuss your approach to learning. How do you stay current with technology? What recent skills have you acquired? How do you help teams learn and grow?
Practice Interview
Study Questions
Collaborative Problem-Solving & Teamwork
Share examples of how you foster collaboration, communicate across differences, build psychological safety, or create environments where people do their best work.
Practice Interview
Study Questions
Spotify Mission & Values Alignment
Demonstrate understanding of Spotify's mission to unlock creative potential for artists and give fans access to creators. Discuss how this mission resonates with you and shapes how you approach engineering.
Practice Interview
Study Questions
Frequently Asked Full-Stack Developer Interview Questions
For a payment flow that integrates with a third-party gateway, evaluate the trade-offs of three approaches: (A) end-to-end tests running against the gateway's sandbox, (B) integration tests that mock the gateway's responses, and (C) contract tests verifying the request and response schemas between your service and the gateway. Explain where contract tests sit relative to integration and end-to-end tests and what problem they solve that the other two do not, then recommend which of the three you would run on every pull request versus nightly, and justify your choice by risk and cost.
Sample Answer
Contract tests exist to solve a problem neither end-to-end nor mocked-integration tests solve well: proving your service and the gateway agree on the SHAPE of their interaction, without needing the real gateway running and without silently drifting out of sync with what the gateway actually does.
Where contract tests sit, and what they uniquely solve
A contract test sits at the seam between integration and end-to-end: like an integration test, it runs fast and needs no live external dependency; like an end-to-end test, it is checking something about the REAL interface, not a hand-written assumption about it. Concretely, a contract test verifies that a captured or agreed-upon schema (which fields exist, their types, allowed values) for the gateway's request and response still holds, and critically, this contract can be verified independently on each side: your service checks it against the contract, and (in a full consumer-driven setup) the gateway's own team checks their real implementation against that same contract, so drift is caught the moment either side changes, at unit-test speed, without needing both systems running together in the same test.
Why (B), a hand-mocked integration test, is not equivalent
An integration test that mocks the gateway's responses is only as good as the assumptions baked into the mock: if the team writes the mock once and the gateway later changes its real response shape, the mocked test keeps passing forever, having silently drifted from reality. This is exactly the gap a contract test closes, since a contract test is verified against an actual agreed specification (or a captured real response), not an assumption the test author wrote down once and never revisited.
Evaluating the three approaches for a payment flow
- (A) End-to-end against the sandbox: highest realism (a real request genuinely reaches something resembling the gateway), but slowest and most fragile, since it depends on the sandbox's availability, network conditions, and test-account state, none of which your team controls.
- (B) Mocked integration tests: fast and reliable to run, but only as trustworthy as the mock's freshness, with the drift risk described above.
- (C) Contract tests: fast like (B), but without the drift risk, since the contract itself is the source of truth both sides verify against, rather than an assumption one side wrote down.
A worked, executable contract test
CONSUMER_CONTRACT = {
"required_fields": {"transaction_id": str, "status": str, "amount_cents": int},
"allowed_status_values": {"succeeded", "declined", "pending"},
}
def verify_contract(response_body, contract):
violations = []
for field, expected_type in contract["required_fields"].items():
if field not in response_body:
violations.append(f"missing required field: {field}")
elif not isinstance(response_body[field], expected_type):
violations.append(f"field '{field}' has wrong type")
if "status" in response_body and response_body["status"] not in contract["allowed_status_values"]:
violations.append(f"unexpected status value: {response_body['status']!r}")
return violations
Run against the CURRENT provider shape ({"transaction_id": "txn_abc123", "status": "succeeded", "amount_cents": 4999}), this correctly returns zero violations. To prove the check is real and not a no-op, it was also run against two realistic breaking changes: a provider release that renames amount_cents to amount (returned violation: missing required field: amount_cents), and a provider that introduces a new status value the contract never declared, such as "requires_action" (returned violation: unexpected status value: 'requires_action'). Both breaking changes were caught, at zero network cost and in well under a millisecond, exactly the class of drift a stale hand-written mock would miss silently.
Recommendation: what runs on every pull request versus nightly
Run the contract test (C) on every pull request: it's fast, deterministic, and catches the highest-value class of bug (a broken assumption about the gateway's real shape) at the lowest cost. Run a smaller number of mocked integration tests (B) on every pull request too, for the request/response HANDLING logic the contract test doesn't cover (how your code reacts to a decline, a timeout, a malformed amount). Reserve the sandbox end-to-end test (A) for nightly, since it is the slowest and least reliable of the three, and its unique value, proving the real network path and real gateway behavior work together, does not need to be re-proven on every single commit.
Trade-offs and pitfalls
A contract test is only as good as how the contract itself stays current: without a process (ideally automated, via a shared contract broker - a service, such as a Pact Broker, where consumer teams publish the contracts they depend on and provider teams look up every contract they need to satisfy) for the gateway team to verify their real implementation against the same contract your consumer test uses, a "contract" test degrades back into the same drift risk as a hand-mocked test, just with extra ceremony. The value of contract testing comes specifically from BOTH sides verifying against a shared source of truth, not from the format of the test itself.
Design a 30-60-90 day onboarding plan for a new hire joining your team. What do you prioritize in each phase, and how do you know they're on track?
Sample Answer
Direct answer
A good 30-60-90 plan moves someone from learning the environment, to contributing under supervision, to owning outcomes independently, with the phase boundaries defined by demonstrated behavior (what they can do unsupervised) rather than by the calendar alone. Track it with a small number of concrete, visible outputs per phase so "on track" is something you can point to, not just a feeling.
The three phases, by what changes
- Days 1-30 (learn and observe): environment setup, codebase or domain orientation, shadowing, and one small real contribution rather than a toy task, so the first change is real but low-risk.
- Days 31-60 (contribute under guidance): own a medium-sized piece of work end to end with a mentor available for review and unblocking, not doing it alongside them line by line.
- Days 61-90 (own outcomes): lead something (a project, an on-call rotation, a smaller onboarding task for the next hire) with the mentor as a backstop, not a co-pilot.
How you know they're on track
- Define the signal per phase in advance, not retroactively: for phase 1, did they reproduce the environment and ship one small real change without major help; for phase 2, is their review feedback shrinking in volume and severity over successive changes; for phase 3, can they make a reasonable decision alone and only escalate the genuinely hard calls.
- Check in on cadence (weekly early on, less frequent later) rather than waiting for day 30, 60, or 90 to find out something drifted three weeks ago.
Adjusting the plan for real constraints
- Limited training resources: when there's no dedicated ramp-up bandwidth (no spare mentor hours, no formal training material), lean harder on asynchronous artifacts: written runbooks, recorded walkthroughs, a curated list of the most representative recent changes, and a lighter-touch weekly sync instead of daily pairing. The phases stay the same; what changes is how much is self-serve versus live.
- Cross-skill ramp: if someone hired primarily for one skill set is expected to also ship in an adjacent one by day 90 (for example, a backend-focused hire expected to ship frontend work), that adjacent skill needs its own explicit milestone inside the plan, not an assumption it'll happen by osmosis. Concretely: days 1-30 stays focused on their strong area to build early confidence and trust; days 31-60 introduces the adjacent skill on a small, well-scoped, low-risk piece with close review; days 61-90 has them own something end to end in the new area, even if smaller in scope than their core-skill ownership.
Worked example
For a new hire joining an established codebase with a small team and no dedicated onboarding budget (the limited-resources case), the 30-60-90 looked like: days 1-30, self-serve environment setup using a written runbook plus a single half-day pairing session, culminating in one small, real bug fix; days 31-60, ownership of one medium feature with async review as the main touchpoint, and a short weekly 15-minute sync instead of daily check-ins; days 61-90, the new hire wrote the onboarding runbook update for the next person, which served double duty as both a real deliverable and a check on whether they actually understood the system well enough to explain it. Being on track was tracked by a short checklist per phase (environment reproducible, first fix merged with normal review effort, feature shipped with review comments trending down) rather than a single blanket "how's it going" check-in.
Trade-offs and pitfalls
- Treating the day boundaries as fixed calendar dates rather than behavioral milestones creates false confidence; someone can hit day 60 without actually being ready for phase-3 ownership, and pushing them into it anyway sets them up to fail.
- Under-supporting the adjacent-skill ramp (assuming a backend engineer will "pick up" frontend without an explicit milestone) is a common way cross-skill onboarding quietly fails; it needs the same structure as the primary skill, just smaller in scope.
- Compressing the plan under limited training resources by cutting phase 1 short (rushing into real ownership before the environment and codebase are understood) trades a faster-looking ramp for more review overhead and rework later.
Describe the core observability signals you would add to a Java microservice to support production debugging and performance monitoring: metrics, structured logs, and distributed traces. For each signal give three concrete examples (metric name, log field, or span annotation), list Java libraries you'd use, and explain strategies to avoid high-cardinality metrics.
Sample Answer
Core observability signals for a Java microservice
Metrics (library: Micrometer/Prometheus):
- http_requests_total{method, status}
- http_request_duration_seconds (histogram)
- db_query_duration_seconds{query}
Structured logs (library: Logback + Logstash encoder / SLF4J): include fields:
- service, level, timestamp
- request_id / trace_id
- user_id (masked) and error_code
Distributed traces (library: OpenTelemetry/Jaeger): span annotations:
- db.query (query name + duration)
- http.client_request (url, method)
- cache.hit (key, hit=true/false)
Avoid high-cardinality metrics by not tagging with user_id or raw IDs; use coarse buckets or sample rare tags. Use logs for high-cardinality identifiers, and apply scrubbing/PII redaction. Export tracing only for sampled requests to limit volume.
A request to your frontend A shows end-to-end latency of 800ms but instrumented spans for every RPC and local CPU time show only ~30-50ms each (A CPU 20ms, A->B network 20ms, B total 30ms, C total 30ms). What are plausible hidden causes for the large wall-clock latency that spans do not show, and describe three experiments or tooling probes you would run to surface the missing delay (e.g., tracing sampling, kernel run-queue metrics, strace for futex waits).
Sample Answer
Plausible hidden causes
Instrumented spans only measure the time a library explicitly wrapped. Roughly 700ms is unaccounted for, which points at delay happening outside what any span is timing:
- Queuing before the traced work even starts. Time spent in a listen/accept queue, a load balancer queue, or a thread-pool/executor queue before a handler begins running is invisible to a span that only starts its clock once the handler executes.
- CPU scheduling delay (off-CPU time). A thread can be runnable but not actually running because the CPU is busy with something else, another tenant, garbage collection, or kernel work. A span reports wall-clock duration, but if the underlying instrumentation doesn't distinguish "doing work" from "waiting to be scheduled," a big gap between a span's reported CPU time and its wall-clock time is exactly this.
- Blocking calls that happen outside the traced boundary. DNS resolution, a TLS handshake, or waiting to check out a connection from a pool typically happen before an RPC client library's span even starts, so they're silently absorbed into the gap.
- Garbage collection or other stop-the-world pauses on any of the hosts involved, which stall the whole process regardless of which span happens to be "active" at that moment.
- Retries not reflected proportionally. A span might report only the final, successful attempt's duration, hiding one or two earlier failed or slow attempts that also consumed wall-clock time.
Three experiments/probes to surface the missing delay
- Instrument the queueing boundary explicitly. Add a timestamp at the earliest possible point (e.g., when the load balancer or reverse proxy accepts the connection, or when a request enters the process) and compare it against when the handler's own span actually starts. This directly measures accept-to-dispatch queuing delay that current spans skip entirely.
- Kernel-level off-CPU analysis. Use
strace -T -fon the process to see per-syscall wait time (a lot of time infutexcalls means lock contention; time inconnect/readmeans I/O waits), or an eBPF-based tool (eBPF lets you attach lightweight tracing/profiling programs to running kernel events without changing kernel code) (like an off-CPU profiler) to directly measure how long threads sat runnable but not scheduled on a core. Checkingperf schedor the kernel's scheduler run-queue stats tells you whether the host itself is CPU-oversubscribed. - Instrument the client library's connection lifecycle. Turn on debug metrics or logging for "time spent acquiring a connection from the pool," and separately time DNS resolution and the TLS handshake. A cold client, or one whose pooled connections keep expiring, can pay these costs on every request without any of it showing up as a traced RPC span.
Running all three against the same time window narrows which bucket (pre-dispatch queuing, CPU scheduling, or connection-lifecycle overhead) actually accounts for most of the missing ~700ms, rather than guessing.
A database is projected to grow 5x in storage and IOPS over the next 12 months. Walk through how you would build a capacity plan for it: how you would model the growth, what safety margins you would build in, how you would project cost, and what automation or alerting you would put in place so capacity never becomes an unplanned outage.
Sample Answer
Direct answer
Capacity planning is forecasting a real growth curve from actual trend data, provisioning ahead of it with a safety margin sized to how long it takes you to actually add capacity, and wiring alerts so the response is scheduled work rather than a page. For a projected 5x in 12 months, the model you pick matters more than the headline number: a smooth compound curve behaves very differently from a step change tied to a known launch, and your alert thresholds should be set from your own provisioning lead time, not a round number borrowed from a blog post.
How to think about it
Model the growth. Compound growth is the safe default without better information: solve for the monthly rate that gets you from current usage to 5x over 12 months.
(1+r)12=5so r=51/12−1. If you instead know about a specific step (a big customer signing, a feature launch), layer that as a discrete jump on top of the smooth trend rather than smoothing it away.
Safety margin sized to lead time. WARN and PAGE thresholds should not be arbitrary round numbers. WARN needs to fire early enough that the provisioning action (a resize request, an approval, the resize itself) finishes before usage reaches PAGE. If it cannot, lower WARN, do not accept the risk.
Cost projection. Project the same curve onto the billing metric, storage dollars per GB-month, provisioned input/output operations per second (IOPS, how many read or write operations the storage layer can do each second) dollars per IOPS-month, so finance sees the number rising before the invoice does, not after.
Forecasting inputs people forget. Compute and storage do not always grow at the same rate and should be modeled separately. Scheduled scaling events (a known launch, a marketing campaign) should be layered on top of the organic trend, not baked into it. Write-ahead log (WAL, the durability log a database writes before committing a change) volume and retention grow faster than data size on write-heavy workloads, and if WAL is retained for a replication window or point-in-time recovery, that retention is its own growing volume. Backup and archival storage across your retention policy is the item most often missed entirely: a 5x growth in live data usually means 5x-or-more growth in backups, especially under a multi-year retention requirement, since old backups do not shrink just because live data changed shape.
Automation and alerting. Alert on the trend, not just the level; a sudden change in growth rate is often the earliest real signal something changed, before any threshold is crossed. Tie WARN to a ticket and runbook, PAGE to on-call, and where the platform allows it, wire the resize itself to fire automatically once a threshold is crossed and a change window is open, instead of requiring a human to notice a graph.
Scaling triggers as a companion to alerting. A trigger is a pre-agreed threshold that fires a specific scaling ACTION, not just a notification: "at 1,000,000 queries per second (QPS) sustained for 15 minutes, add a read replica" is a trigger. "Storage at 85%, page a human who decides what to do" is an alert. A complete capacity plan has both.
Worked example
Pinned inputs, reproducible on re-run:
current_storage_gb = 800
current_iops = 6000
growth_multiple = 5.0
horizon_months = 12
alert_warn_pct = 0.70
alert_page_pct = 0.85
lead_time_months = 1.0
r = growth_multiple ** (1 / horizon_months) - 1
Real output from running it:
implied monthly compound growth rate: 14.3530%
month storage_gb iops
0 800 6000
6 1789 13416
12 4000 30000
provision AT LEAST 4706 GB / 35294 IOPS so month 12's actual usage
(4000 GB / 30000 IOPS) lands at the 85% page threshold, not above it
alert thresholds: WARN at 3294 GB (70%), PAGE at 4000 GB (85%)
at this growth rate, WARN fires roughly 1.4 months before PAGE would,
against a 1-month provisioning lead time (sufficient)
The 1.4-month runway against a 1-month lead time is the check that matters: if that number had come out below the lead time, the fix is to lower WARN, not hope the resize finishes in time.
A second, QPS-based variant shows the scaling-triggers framing on a different metric: 100,000 QPS growing to 1,000,000 QPS over 18 months instead of 12. Same method, different multiple and horizon: r=101/18−1≈13.65% a month, giving 100,000 at month 0, about 215,000 at month 6, about 464,000 at month 12, and 1,000,000 at month 18 (each value is 100000×(1+r)m, computed directly from the pinned rate above). The companion trigger here would be concrete and actionable, for example "add a read replica once sustained QPS crosses 500,000 for 15 minutes," not just a dashboard threshold.
The same method scales to a longer horizon too, for example a 10x-over-3-years warehouse case (multiple 10, horizon 36 months): the growth-rate formula is unchanged, but at that horizon the WAL retention and backup and archival line items above stop being a footnote and start dominating, since backup windows and cross-region replication bandwidth become material costs at multi-terabyte, multi-year scale in a way they are not at 12 months.
Trade-offs and pitfalls
Setting WARN too close to PAGE leaves no time to act, exactly what the runway check above exists to catch. Alerting on absolute usage only, and missing rate-of-change, means the earliest real signal gets ignored until a static threshold trips. Treating compute and storage as one number hides real divergence: a read-heavy service can 5x its IOPS while barely growing on disk, and a write-heavy audit log can do the reverse. The single most common gap is forgetting backups and WAL retention, teams size the live database carefully and then get surprised when the backup bill, or the backup window and recovery time objective, breaks first.
In Go or Python, write a compact function that validates a token's 'scope' claim contains the required scope for an endpoint and that the token 'sub' is the resource owner or the token bearer has an 'admin' role. Assume cryptographic verification of the token is already performed; focus on the claim logic and potential pitfalls like scope format variations.
Sample Answer
Direct answer
Assuming the token's signature, expiration, audience, and issuer have already been verified upstream, what's left is pure claim logic: does the token carry the specific scope this endpoint requires, and is the caller either the resource's owner or someone holding an admin role. The tricky part in practice isn't the boolean logic, it's that different identity providers emit the scope (and role) claim in different shapes, so the check has to normalize before it compares, or it will silently reject valid tokens from a provider configured slightly differently than the one you tested against.
Structured elaboration (approach)
- Normalize the scope claim. The RFC 6749 / RFC 8693 standard shape is a single space-delimited string, for example
"read:orders write:orders". Some non-standard identity provider configurations instead emit a JSON array,["read:orders", "write:orders"]. Handle both by converting either shape into a set before checking membership; treat a missing claim as an empty set rather than raising, since "no scopes granted" is a legitimate, if maximally restrictive, token state. - Required-scope check. The normalized set must contain the exact scope string this endpoint requires.
- Resource-ownership check. The token's
subclaim must equal the id of the resource owner passed in by the caller. - Admin override. The same string-or-array pitfall applies to a roles claim; normalize it the same way, and check for
"admin"in the result. - Combine the last two with an OR, not an AND: either condition alone is sufficient to grant access, given the required scope is already present.
Worked example (executed)
def _normalize_scopes(scope_claim) -> set:
# Handle the two shapes IdPs actually emit for a scope claim:
# - RFC 8693 / RFC 6749 standard: a single space-delimited string
# - Non-standard but real in the wild: a JSON array of strings
# A missing claim normalizes to an empty set rather than raising.
if scope_claim is None:
return set()
if isinstance(scope_claim, str):
return set(scope_claim.split())
if isinstance(scope_claim, (list, tuple, set)):
return set(scope_claim)
raise TypeError(f"unsupported scope claim type: {type(scope_claim)!r}")
def has_required_access(claims: dict, required_scope: str, resource_owner_id: str) -> bool:
# claims: the ALREADY-VERIFIED token claims (signature/exp/aud/iss checked upstream)
# required_scope: the single scope string this endpoint requires
# resource_owner_id: the id of the user who owns the resource being accessed
scopes = _normalize_scopes(claims.get("scope"))
if required_scope not in scopes:
return False
roles = _normalize_scopes(claims.get("roles"))
is_owner = claims.get("sub") == resource_owner_id
is_admin = "admin" in roles
return is_owner or is_admin
def run_demo():
cases = []
claims = {"sub": "user-42", "scope": "read:orders write:orders", "roles": "member"}
cases.append(("space-delimited scope + owner match -> allow",
has_required_access(claims, "write:orders", "user-42") is True))
claims = {"sub": "user-99", "scope": ["read:orders"], "roles": ["member", "admin"]}
cases.append(("array-shaped scope + admin role, not owner -> allow",
has_required_access(claims, "read:orders", "user-42") is True))
claims = {"sub": "user-42", "scope": "read:orders", "roles": "member"}
cases.append(("missing required scope, owner -> deny",
has_required_access(claims, "write:orders", "user-42") is False))
claims = {"sub": "user-7", "scope": "write:orders", "roles": "member"}
cases.append(("scope present, not owner, not admin -> deny",
has_required_access(claims, "write:orders", "user-42") is False))
claims = {"sub": "user-42", "roles": "admin"}
cases.append(("absent scope claim -> deny regardless of role",
has_required_access(claims, "write:orders", "user-42") is False))
all_pass = True
for name, passed in cases:
print(f"[{'PASS' if passed else 'FAIL'}] {name}")
if not passed:
all_pass = False
print(f"\nALL_PASS={all_pass}")
if __name__ == "__main__":
run_demo()
Output, from an actual run (python3 scope_claim_check.py):
[PASS] space-delimited scope + owner match -> allow
[PASS] array-shaped scope + admin role, not owner -> allow
[PASS] missing required scope, owner -> deny
[PASS] scope present, not owner, not admin -> deny
[PASS] absent scope claim -> deny regardless of role
ALL_PASS=True
Complexity and edge cases
Normalizing a claim with k space-delimited or listed scope entries is O(k); every membership test afterward is O(1) average against a Python set. The five cases above cover the named pitfall directly (a space-delimited string and an array both parsed correctly to the same result) plus the boundary conditions: required scope absent, present scope but neither owner nor admin, and no scope claim at all.
Trade-offs and pitfalls
- Silently treating an unrecognized scope claim type (neither a string nor a list) as "no scopes" would fail open into looking like a permissive default in some code paths, depending on how the caller handles a
Falsereturn versus an exception. This implementation raisesTypeErrorfor anything unexpected instead, so an integration bug in a new client fails loudly during testing rather than quietly granting or denying access based on an accidental default. - Keep the ownership check and the admin check as an OR, never fold in additional business rules like "and the account must be active" directly into this function; if that's a real requirement, it belongs as its own explicit check, so this function's contract stays exactly "scope plus ownership-or-admin" and nothing more.
- Scope strings that look similar but aren't identical are a real, silent source of bugs:
"orders:write"versus"order:write"will never match, and no amount of normalization catches a plain typo in the scope name. That has to be caught by a shared constants module the token issuer and every consuming endpoint both import from, not by the claim-check logic itself.
Users can attach videos of several GB to a post. Walk me through the upload feature end to end: what the user sees, how the bytes reach storage, how the backend learns the upload finished, and how you clean up when users abandon or fail halfway.
Sample Answer
Direct answer
Three ideas carry this design, and each can be said in one breath. First, the file is cut into numbered parts that go straight from the browser to object storage, so one dropped connection costs one part and not the whole file. Second, every status change on the upload row is a conditional update (an UPDATE whose WHERE clause names the status it expects, so only one of several racing callers changes the row). Third, cleanup has more than one layer, because no single layer catches every abandoned upload.
The video never passes through your API servers. The backend creates an Amazon S3 multipart upload (the object store splits one big file into numbered parts that can be sent in parallel, retried individually, and stitched together at the end), signs a short-lived URL for each part, and the browser sends the parts straight to object storage. A signed (presigned) URL is an ordinary storage URL with a time-limited signature in its query string, produced by the backend with its own credentials. Whoever holds it can perform exactly the one operation it was signed for (for example, PUT part 3 of this upload) until it expires, which is why a browser with no storage credentials is allowed to write to a private bucket. The client then calls complete; the backend asks storage to assemble the object, marks the upload processing, and a worker scans it and transcodes it (re-encodes the video into formats and sizes that browsers and phones play). Cleanup is layered: an explicit cancel, a database reaper (a scheduled job that finds expired rows and cleans them up) for expired uploads, and a bucket lifecycle rule (a storage-side setting that deletes or aborts things after a set age) as a backstop, meaning a last line of defence for what the others missed. Every state change is a conditional update, so the repeated complete calls and the repeated storage events that will happen cannot do the work twice.
What the user sees
- Pick the file. The page checks size (our product cap, 10 GiB) and type, shows the name and a progress bar. A post can be written while the upload runs.
- Uploading. A bar driven by bytes that storage has confirmed, a speed and time-left estimate, and Pause and Cancel. Closing the tab triggers a "leave page?" prompt only while parts are in flight.
- After a failure or reload. The upload id and the finished parts are kept in the browser's IndexedDB (browser-side database). The user reselects the same file; the page accepts it only if name, size and last-modified time match, asks the server which parts exist, and sends only the missing ones.
- Processing. "Processing your video" until the status is
ready. The author can submit the post once the bytes are safely stored, but other people see it only when the video isready.
One upload, traced
Take a 6 GiB file (GiB = 1,073,741,824 bytes; MiB = 1,048,576 bytes; the sizes below use these binary units). The server picks a 16 MiB part size, so the file becomes 384 parts. The browser slices the file: part 1 is bytes 0 to 16 MiB minus 1, part 2 is the next 16 MiB, and so on. For part 1 it asks the API for a signed URL, sends the bytes with PUT, and storage answers with an ETag header, a short fingerprint of that part. The browser keeps (1, etag1). It repeats this for parts 2 to 384, four at a time. After the last part it calls complete with the list of 384 (part number, ETag) pairs; storage checks each pair, joins the parts into one object, and the row moves on to processing. If the connection dies at part 200, parts 1 to 199 are already stored, so only part 200 onward is sent again.
API contract
| Call | Purpose | Response |
|---|---|---|
POST /v1/uploads {filename, size_bytes, content_type} | start; server creates the multipart upload and the row | 201 {upload_id, part_size, part_count, expires_at} |
GET /v1/uploads/{id}/parts?numbers=1,2,3,4 | sign the next few parts only (on demand, so a 640-part upload does not presign 640 URLs up front) | 200 {"urls": {"1": "...", ...}, "uploaded": [1,2]} |
POST /v1/uploads/{id}/complete {parts: [{n, etag}]} | assemble the object | 202 {status: "completing"}; repeating it returns the current status |
GET /v1/uploads/{id} | status for polling (every 2 s, backing off to 10 s) | 200 {status, attempts} |
DELETE /v1/uploads/{id} | user cancels | 200 {status: "abandoned"} |
How bytes reach storage: the browser sends each part with PUT to its signed URL. The signature is limited by the permissions of whoever signed it, and the URL expires, so a leaked URL is useless soon. The browser must record each part's ETag (a version tag the store returns per part) because the complete request needs the part number and ETag of every part. JavaScript can read that response header only if the bucket's CORS configuration lists it under ExposeHeaders. (CORS, cross-origin resource sharing, is the browser rule that stops a page on one site from reading responses from another origin unless that origin opts in; here the bucket is the other origin, and ExposeHeaders is its opt-in for specific response headers such as ETag.)
Storage limits that shape the plan, as documented for Amazon S3: at most 10,000 parts per upload, each part 5 MiB to 5 GiB (the last may be smaller), and a 48.8 TiB maximum object. ListParts (the storage call that lists which parts of an upload have already arrived) returns at most 1,000 parts per call, so resuming a large upload pages through it. Choosing the part size: the 10,000-part limit means a part must be at least size / 10,000, so the function rounds that up to a whole MiB, and never goes below 16 MiB so that a normal file is not split into thousands of tiny requests. Math.max picks the larger of the two, Math.ceil rounds up. With a 10 GiB cap the 16 MiB floor always wins (10 GiB / 10,000 is about 1 MiB); the size-based term only matters for files above 156.25 GiB.
const MiB = 1024 ** 2, GiB = 1024 ** 3, MAX_PARTS = 10000, MIN_PART = 5 * MiB;
// choose a part size: at least 16 MiB, and large enough that the file fits in 10,000 parts
export function partPlan(sizeBytes) {
const part = Math.max(16 * MiB, Math.ceil(sizeBytes / MAX_PARTS / MiB) * MiB);
return { partMiB: part / MiB, parts: Math.ceil(sizeBytes / part), lastPartMiB: (sizeBytes - (Math.ceil(sizeBytes / part) - 1) * part) / MiB };
}
for (const g of [0.5, 6, 10]) console.log(`${g} GiB ->`, JSON.stringify(partPlan(g * GiB)));
console.log('largest file 16 MiB parts allow:', (16 * MiB * MAX_PARTS) / GiB, 'GiB');
// time to re-send if the connection dies: whole file vs one part, at 10 Mbit/s upstream
const bytesPerSec = 10e6 / 8;
console.log('re-send whole 6 GiB at 10 Mbit/s (minutes):', (6 * GiB / bytesPerSec / 60).toFixed(1));
console.log('re-send one 16 MiB part (seconds):', (16 * MiB / bytesPerSec).toFixed(1));
Run in a node:22 container, it prints:
0.5 GiB -> {"partMiB":16,"parts":32,"lastPartMiB":16}
6 GiB -> {"partMiB":16,"parts":384,"lastPartMiB":16}
10 GiB -> {"partMiB":16,"parts":640,"lastPartMiB":16}
largest file 16 MiB parts allow: 156.25 GiB
re-send whole 6 GiB at 10 Mbit/s (minutes): 85.9
re-send one 16 MiB part (seconds): 13.4
That last pair is the reason for parts: a dropped connection costs one 16 MiB part (about 13 seconds at 10 Mbit/s) instead of the whole 6 GiB file (about 86 minutes). The client sends 4 parts at once and retries each part up to 3 times with exponential backoff and jitter (random extra delay so many clients do not retry in lockstep). It pauses on the browser's offline event.
Finish: how the backend learns the upload ended
Two paths lead to the same transition, and both are safe to repeat:
- The client calls
complete. The server moves the rowpendingtocompletingwith a conditional update, calls storage's complete operation with the part list, then moves it toprocessingand queues the job. Before assembling it can compare the part list withListParts, which is meant for verification. After assembly it checks the object size equals the declared size, and fails the upload if not. - A storage event. The bucket can notify a queue when an object is created. Amazon S3 documents event notifications as delivered at least once (a message may arrive twice, but is not silently dropped by design), so duplicates are normal. The handler uses the same conditional update, so a second delivery changes zero rows and does nothing. This path rescues the case where the server crashed after storage finished assembling but before the row was updated.
The state table and its guards
CREATE TABLE uploads (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
user_id bigint NOT NULL,
s3_upload_id text NOT NULL,
object_key text NOT NULL,
size_bytes bigint NOT NULL CHECK (size_bytes BETWEEN 1 AND 10 * 1024^3),
status text NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending','completing','processing','ready','failed','abandoned')),
attempts integer NOT NULL DEFAULT 0,
expires_at timestamptz NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX uploads_open ON uploads (expires_at) WHERE status = 'pending';
INSERT INTO uploads (id, user_id, s3_upload_id, object_key, size_bytes, expires_at) VALUES
('00000000-0000-0000-0000-00000000000a', 7, 'mpu-a', 'raw/a', 6442450944, now() + interval '24 hours'),
('00000000-0000-0000-0000-00000000000b', 7, 'mpu-b', 'raw/b', 1000000, now() - interval '1 hour'),
('00000000-0000-0000-0000-00000000000c', 8, 'mpu-c', 'raw/c', 1000000, now() - interval '2 hours');
UPDATE uploads SET status = 'completing' WHERE id = '00000000-0000-0000-0000-00000000000a' AND status = 'pending' RETURNING id, status;
UPDATE uploads SET status = 'completing' WHERE id = '00000000-0000-0000-0000-00000000000a' AND status = 'pending' RETURNING id, status;
UPDATE uploads SET status = 'processing' WHERE id = '00000000-0000-0000-0000-00000000000a';
UPDATE uploads SET attempts = attempts + 1, status = CASE WHEN attempts + 1 >= 3 THEN 'failed' ELSE 'processing' END
WHERE id = '00000000-0000-0000-0000-00000000000a' RETURNING attempts, status;
UPDATE uploads SET attempts = attempts + 1, status = CASE WHEN attempts + 1 >= 3 THEN 'failed' ELSE 'processing' END
WHERE id = '00000000-0000-0000-0000-00000000000a' RETURNING attempts, status;
UPDATE uploads SET attempts = attempts + 1, status = CASE WHEN attempts + 1 >= 3 THEN 'failed' ELSE 'processing' END
WHERE id = '00000000-0000-0000-0000-00000000000a' RETURNING attempts, status;
WITH due AS (
SELECT id FROM uploads WHERE status = 'pending' AND expires_at < now()
ORDER BY expires_at LIMIT 100 FOR UPDATE SKIP LOCKED)
UPDATE uploads u SET status = 'abandoned' FROM due WHERE u.id = due.id
RETURNING u.id, u.s3_upload_id, u.status;
Reading the statements in order, with the table as set up by the INSERT: upload a is a live 6 GiB upload, b and c are pending uploads whose expires_at is already in the past.
- First guarded
UPDATE ... WHERE status = 'pending'ona: the condition holds, so one row changes tocompletingandRETURNINGprints it. - The identical statement again:
ais nowcompleting, theWHEREmatches nothing, and zero rows come back. This is a secondcompletecall, or a second copy of the storage event. - The unguarded
UPDATEtoprocessingprints nothing because it has noRETURNING; it stands for the assemble step finishing. - The three retry statements are the same text.
attempts = attempts + 1bumps the counter, andCASE WHEN attempts + 1 >= 3 THEN 'failed' ELSE 'processing' ENDsays: if this is the third failure, give up; otherwise stay inprocessingfor another try. In anUPDATE,attemptson the right side is the value before the update, henceattempts + 1. - The
WITH due AS (...)query is the reaper. It selects up to 100 pending rows past their expiry, oldest first, andUPDATE ... FROM duemarks themabandoned.
Run in a postgres:16 container, the guarded update returns one row the first time (completing) and (0 rows) the second, so only one caller proceeds. The three failure updates print 1 | processing, 2 | processing, 3 | failed: a processing job is retried until three attempts, then marked failed and the user is told. The reaper returns the two expired rows (mpu-c and mpu-b) as abandoned and leaves the live upload alone. FOR UPDATE SKIP LOCKED locks the rows the query selects and skips any row another transaction has already locked, so two reaper instances can run at once without claiming the same row.
Server-side processing at feature altitude
A worker takes the job from a queue (a list of pending jobs that workers pull from one at a time) and (1) scans the object for malware, (2) inspects the container to confirm it really is a video and within limits, (3) writes a thumbnail and web-friendly renditions (copies of the video at several resolutions and bitrates) into a separate processed/ prefix, and (4) sets ready. Retries use the attempt counter above with growing delays; a job that fails three times goes to failed. The raw bucket stays private. Web clients get the processed files through a CDN (content delivery network, a cache close to the viewer) with long cache lifetimes, since each rendition has a new key.
Cleaning up abandoned and failed uploads
Storage keeps uploaded parts, and bills for them, until the upload is completed or stopped. There are three layers:
| Layer | Trigger | Action |
|---|---|---|
| Cancel | user presses Cancel | DELETE marks abandoned and aborts the multipart upload |
| Reaper | every 15 minutes, rows still pending past expires_at (24 hours) | mark abandoned, call abort for each |
| Lifecycle rule | bucket-level rule AbortIncompleteMultipartUpload, whose DaysAfterInitiation setting is the number of days after an upload starts before storage aborts it (the AWS example uses 7) | storage stops uploads not completed in that time and deletes their parts |
Part uploads that were in flight when an abort arrived can still land afterwards, which is why the lifecycle rule exists as a sweep for anything the reaper missed. Failed processing keeps the raw object for a few days for investigation, then a retention rule deletes it.
Test, roll out, observe
Tests: the conditional-update SQL above, a Playwright test that kills the connection mid-part and asserts only that part is resent, a test that calls complete twice, and one that delivers the storage event twice. Roll out to staff first with a low size cap. Observe the share of uploads ending abandoned or failed, the age of the oldest pending row, queue depth (how many jobs are waiting for a worker; a rising number means workers cannot keep up), and bytes in incomplete multipart uploads (the leak the lifecycle rule guards).
Trade-offs
Direct-to-storage upload removes video bytes from your servers and gives resumability, and costs a more complex client and a second path to learn completion. For files under about 100 MB a single signed PUT is simpler; multipart pays off as sizes grow and networks worsen.
What is the circuit breaker pattern? Walk through its states, closed, open, and half-open, what triggers each transition, and how you'd choose the failure threshold and time window for a real dependency.
Sample Answer
The circuit breaker pattern stops calling a failing dependency once it's clearly unhealthy, so callers fail fast instead of piling up waiting on a dependency that isn't going to answer, and the dependency gets breathing room to recover instead of being hit with an ever-growing retry storm on top of whatever's already wrong with it.
The three states
stateDiagram-v2
[*] --> Closed
Closed --> Open: failure threshold crossed
Open --> HalfOpen: cooldown elapses
HalfOpen --> Closed: probe requests succeed
HalfOpen --> Open: probe requests fail
| State | Behavior | What triggers the next transition |
|---|---|---|
| Closed | Calls pass through normally | Error rate or consecutive failures cross a defined threshold within the tracking window |
| Open | Calls fail immediately (or return a fallback); the dependency isn't called at all | A fixed cooldown period elapses |
| Half-open | A small number of probe requests are allowed through to test recovery | Probes succeed (close the breaker) or fail (reopen it, usually with a longer cooldown) |
Choosing the threshold and window for a real dependency
Base the threshold on the dependency's own historical baseline, not a round number picked by feel: if a dependency's normal error rate is 1-2%, a threshold like "error rate exceeds 50% over a 1-minute window" is a real signal of degradation, not noise. Combine multiple signals rather than trusting one: an error-rate threshold alone can be fooled by a burst of retriable timeouts, so pairing it with a consecutive-failure count and a latency percentile (for example, p99 exceeding a set ceiling) catches degradation that shows up as slowness before it shows up as outright errors.
Worked example: why the half-open probe count matters
Say the breaker opens, waits out its cooldown, and moves to half-open, sending 5 probe requests before deciding whether to close. If the dependency is still genuinely degraded, with a true underlying failure rate of p=0.3 (30% of calls failing), the probability that all 5 probes happen to succeed by chance despite that is:
P(all 5 probes succeed)=(1−p)5=(0.7)5≈0.168(16.8%)That's not a rare fluke, it's roughly a 1-in-6 chance of prematurely closing the breaker on a dependency that's still 30% broken, which then immediately re-floods it with full traffic and likely reopens the breaker on the very next window. This is the concrete argument for either using more probes (the same calculation with 10 probes drops the false-close probability to 0.710≈0.028, about 2.8%) or ramping traffic gradually after a half-open success instead of jumping straight from 5 probes to 100% traffic.
Trade-offs and pitfalls
Setting the threshold too sensitive (a low error-rate bar or a short window) causes flapping: the breaker opens on transient noise, degrades the user experience with unnecessary fallbacks, and can itself become a source of alerts nobody trusts. Setting it too lax delays protection long enough for the caller's own retries and connection-pool exhaustion to cascade into a second incident on top of the first. The half-open probe-count math above is the same trade-off in miniature: too few probes risk a premature, false-positive close; too many probes delay recovery and keep failing extra requests during the test window. In practice this is tuned with production data and game-day testing rather than picked once and left alone, and the same three-state logic applies regardless of what's on the other side of the call, an AI inference endpoint that starts throwing GPU-OOM errors under load trips the same breaker, on the same threshold logic, as a slow downstream REST dependency; only the specific error signal being watched changes.
Evaluate the multi-tenant database-partitioning strategies available to a SaaS product: a shared schema for all tenants, a separate schema per tenant, and a fully isolated database per tenant. Discuss tenant isolation, operational overhead, migration complexity, mitigating a hot tenant, and how each approach affects backup/restore and compliance obligations.
Sample Answer
Direct answer
Shared schema, schema-per-tenant, and database-per-tenant trade isolation against operational overhead in a fairly direct line: shared schema is cheapest to run and hardest to isolate, database-per-tenant is the strongest isolation and the most expensive to operate at scale, and schema-per-tenant sits in between. The right default for most SaaS products is shared schema with a tenant-ID column and strict row-level access enforcement, escalating specific tenants to their own schema or database only when isolation or compliance requirements demand it.
Structured elaboration
| Dimension | Shared schema | Schema per tenant | Database per tenant |
|---|---|---|---|
| Tenant isolation | Weakest: logical only, enforced by application code or row-level security on every query | Stronger: tenants can't accidentally query across schemas, but share the underlying database engine, connection pool, and physical resources | Strongest: separate connection pool, separate resource limits, separate blast radius per tenant |
| Operational overhead | Lowest: one schema to migrate, monitor, and back up | Moderate: N schemas to migrate (usually via one script run N times), shared instance to monitor | Highest: N databases to provision, monitor, patch, and back up independently |
| Migration complexity | Simplest: one migration, applies to all tenants at once | Each tenant's schema must be migrated, typically in a loop; a failed migration on one tenant does not have to block others | Same as schema-per-tenant, but now includes coordinating N separate database instances, which is slower and has more infrastructure surface to fail |
| Hot-tenant mitigation | Hardest: a hot tenant's query load is shared with every other tenant's connection pool and cache; requires application-level rate limiting or query governance per tenant | Better: a hot tenant's queries are isolated to its own schema, but still share instance-level resources like buffer cache and disk I/O, input/output operations per second | Best: a hot tenant's dedicated instance can be scaled, resource-limited, or moved without affecting any other tenant |
| Backup, restore, compliance | Hardest to satisfy tenant-specific requirements: restoring one tenant's data means restoring (or filtering out of) a backup that contains every tenant's data, complicating any customer-specific compliance obligation to isolate or delete their data on request | Better: a schema can be backed up and restored somewhat independently, though within a shared instance's backup mechanics | Best: a tenant's data has a clean backup, restore, and deletion boundary that maps directly to compliance requirements like data residency or a right-to-deletion request |
Why shared schema is usually still the right starting default. Most SaaS products have far more small tenants than large ones. Database-per-tenant at 1,000 tenants means 1,000 databases to patch, monitor, and pay for, most of them nearly idle; that operational and cost overhead is rarely justified until a tenant's size, regulatory requirement, or noisy-neighbor impact specifically demands it. Shared schema, with a tenant_id on every row and either application-enforced or database-native row-level access control, keeps operational cost flat as tenant count grows.
Mitigating a hot tenant without changing the whole architecture. Before escalating a hot tenant to its own schema or database, first try per-tenant rate limiting or query budgets within the shared schema, and confirm indexes are structured so the tenant_id is always the leading column (so a hot tenant's queries don't force a wider scan that also slows every other tenant sharing the table). Escalate to schema or database isolation only when a tenant's load genuinely cannot be governed down to a fair share, or when the tenant has a hard compliance requirement (data residency, contractual data isolation) that shared infrastructure cannot satisfy regardless of load.
Worked example
A concrete escalation path: start every tenant on shared schema. When a specific tenant's query volume or data size grows large enough that its queries are measurably degrading other tenants' P95 latency (95th-percentile latency) even after rate limiting and index tuning, migrate that one tenant to its own schema within the same database instance, which isolates its query plans and connection usage without a full new database to operate. If that tenant later has a compliance requirement (for example, a contractual obligation for physically isolated storage) that schema-level isolation cannot satisfy, migrate that tenant again, this time to its own dedicated database. This keeps the operational overhead proportional to the number of tenants that actually need the stronger isolation, rather than paying database-per-tenant overhead for every tenant up front.
As a concrete illustration of that escalation path (illustrative planning numbers, not measured): assume a shared-schema deployment normally holds P95 query latency at 40ms for all tenants, and one tenant, "Tenant A," grows to 80 million rows (versus a typical tenant's 500,000 rows) and 1,200 queries per second at peak, roughly 100x a typical tenant's 12 QPS. Once Tenant A's queries push shared-schema P95 latency past a 100ms threshold for the other tenants sharing the instance, even after rate limiting and index tuning, that crossing is the concrete trigger to migrate Tenant A to its own schema. Suppose Tenant A later signs a contract requiring physically isolated storage, a data-residency clause naming a specific region and no shared infrastructure: schema-per-tenant cannot satisfy a physical-isolation clause like that, since the underlying database instance and its disks are still shared with other tenants, so that specific, named contractual requirement is what triggers the second migration, to database-per-tenant.
Trade-offs & pitfalls
The most common mistake is picking database-per-tenant early because it feels safest, then discovering the operational cost (patching, monitoring, connection-pool sizing) scales linearly with tenant count and becomes the team's dominant maintenance burden long before any tenant actually needed that level of isolation. The opposite mistake is staying on shared schema past the point where a specific tenant's compliance requirement (data residency, contractual isolation, a right-to-deletion request that must not risk touching other tenants' data) genuinely cannot be satisfied by row-level controls; that is not primarily a performance decision and should not be deferred purely to avoid operational overhead. The schema-per-tenant middle tier is often underused: it gives real isolation gains for hot or sensitive tenants without the full cost of a separate database instance, and is worth defaulting to as the escalation step before database-per-tenant rather than jumping straight there.
Why do keys matter when rendering lists in React? Given the following code, explain potential bugs and show an improved approach: {items.map((item, idx) => <ListItem key={idx} data={item} />)}. Describe how React's reconciliation uses keys and why using stable unique ids is important when items are added, removed, or reordered.
Sample Answer
Direct answer
key is how React matches an item in this render's list to the same item in the previous render. A component instance is one live copy of a component that React is keeping on screen, with its own state and DOM nodes; two rows rendered from the same Row function are two instances. To mount an instance is to create it for the first time, and to unmount it is to destroy it and discard its state. A render is different: it is React calling the component function again on an existing instance. With key={idx} the identity is the item's position, so when items are added, removed or reordered, React hands each component instance the data of whatever item now sits at its old position. Props update, but anything that lives in the instance (state, uncontrolled input text, focus, an in-flight effect) stays at the position and ends up attached to the wrong item. The fix is a key derived from the data: key={item.id}, a stable unique value that travels with the item.
How reconciliation uses the key
Reconciliation is React's comparison of the previous element tree with the new one to decide what to update. For a list of siblings, React pairs old and new children by key (falling back to position when no key is given). A pair with the same key and type keeps its component instance and receives new props. A new key is mounted; a missing key is unmounted. React's documentation puts it this way: "Keys tell React which array item each component corresponds to, so that it can match them up later," and "Index as a key often leads to subtle and confusing bugs." It also notes that omitting the key makes React use the index anyway.
Reproducing the bug
The question's snippet is items.map((item, idx) => <ListItem key={idx} data={item} />). The program below renders rows of three items, each with two kinds of per-row state: a "star" toggle held in useState and a note typed into an uncontrolled <input> (one whose text lives in the DOM node, not in React state). It stars item A and types a note on it, then prepends a new item Z, and runs the scenario under three key strategies. Row is wrapped in memo (React.memo, which skips calling the function again when every prop is unchanged) so the render counts show real work (a function that React ran again).
import { JSDOM } from 'jsdom';
import assert from 'node:assert/strict';
const dom = new JSDOM('<!doctype html><body></body>', { url: 'http://localhost' });
globalThis.window = dom.window; globalThis.document = dom.window.document;
globalThis.IS_REACT_ACT_ENVIRONMENT = true;
const React = (await import('react')).default;
const { render, act, cleanup, fireEvent } = await import('@testing-library/react');
const { useState, useEffect, memo } = React;
const h = React.createElement;
let stats;
const Row = memo(function Row({ item }) {
stats.renders++;
const [starred, setStarred] = useState(false); // state kept by React, tied to the row's identity
useEffect(() => { stats.mounts++; }, []);
return h('li', { 'data-name': item.name },
item.name, ' ',
h('button', { onClick: () => setStarred((s) => !s) }, starred ? 'starred' : 'star'),
h('input', { 'aria-label': 'note ' + item.name })); // uncontrolled: the DOM node holds the typed text
});
let randomCounter = 0;
const keyFor = { index: (item, i) => i, id: (item) => item.id, random: () => 'k' + randomCounter++ };
if (process.env.BREAK) keyFor.id = (item, i) => i; // BREAK=1 removes the technique: ids no longer used
function List({ items, mode }) {
return h('ul', null, items.map((item, i) => h(Row, { key: keyFor[mode](item, i), item })));
}
function scenario(mode) {
stats = { renders: 0, mounts: 0 };
const A = { id: 'a', name: 'A' }, B = { id: 'b', name: 'B' }, C = { id: 'c', name: 'C' }, Z = { id: 'z', name: 'Z' };
const { container, rerender } = render(h(List, { items: [A, B, C], mode }));
const li = (n) => container.querySelector(`li[data-name="${n}"]`);
fireEvent.click(li('A').querySelector('button')); // star A
fireEvent.change(li('A').querySelector('input'), { target: { value: 'note for A' } });
stats.renders = 0; stats.mounts = 0;
rerender(h(List, { items: [Z, A, B, C], mode })); // prepend Z
const rows = [...container.querySelectorAll('li')].map((l) => ({
name: l.dataset.name, star: l.querySelector('button').textContent, note: l.querySelector('input').value }));
cleanup();
return { rows, renders: stats.renders, mounts: stats.mounts };
}
for (const mode of ['index', 'id', 'random']) {
const r = scenario(mode);
console.log(`key=${mode}: renders after prepend=${r.renders}, mounts after prepend=${r.mounts}`);
for (const row of r.rows) console.log(` ${row.name} ${row.star} note="${row.note}"`);
if (mode === 'index') {
assert.deepEqual(r.rows[0], { name: 'Z', star: 'starred', note: 'note for A' }); // A's state landed on Z
assert.equal(r.rows[1].star, 'star');
assert.equal(r.mounts, 1); assert.equal(r.renders, 4);
} else if (mode === 'id') {
assert.deepEqual(r.rows.find((x) => x.name === 'A'), { name: 'A', star: 'starred', note: 'note for A' });
assert.deepEqual(r.rows[0], { name: 'Z', star: 'star', note: '' });
assert.equal(r.mounts, 1); assert.equal(r.renders, 1);
} else {
assert.equal(r.mounts, 4); assert.equal(r.rows.find((x) => x.name === 'A').note, '');
}
}
console.log('all assertions passed');
Run in a node:22 container it prints:
key=index: renders after prepend=4, mounts after prepend=1
Z starred note="note for A"
A star note=""
B star note=""
C star note=""
key=id: renders after prepend=1, mounts after prepend=1
Z star note=""
A starred note="note for A"
B star note=""
C star note=""
key=random: renders after prepend=4, mounts after prepend=4
Z star note=""
A star note=""
B star note=""
C star note=""
all assertions passed
Reading the three runs:
key=index: the star and the note typed for A are now shown on Z, and A shows as unstarred with an empty note. Position 0 kept its instance (with its state) while the data changed under it. This is the bug, and it is worse for editable rows than for static text. All 4 rows rendered, because every position received a different item, and only 1 mounted (the new last position).key=id: A keeps its star and note, Z is a fresh row, and only 1 row rendered at all, because the three existing rows received equal props andmemoskipped them. Stable keys are also what make the cheap path possible.key=random(the "generate keys on the fly" mistake): all 4 rows were destroyed and recreated, A lost both pieces of state, and the work is the same as a full remount of the list. React's documentation warns that this "will cause keys to never match up between renders, leading to all your components and DOM being recreated every time."
With BREAK=1 the id strategy is replaced by the index. Run in the same container, the id run then repeats the index result and the first id assertion fails, so the demo proves the key is what carries the state:
key=id: renders after prepend=4, mounts after prepend=1
Z starred note="note for A"
A star note=""
B star note=""
C star note=""
AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected
{
name: 'A',
+ note: '',
+ star: 'star'
- note: 'note for A',
- star: 'starred'
}
The A row shows star with an empty note instead of starred with its note.
Choosing a good key
- Use a stable, unique id from the data: a database primary key, or an id assigned when the item is created (for example a counter or
crypto.randomUUID()called when the item is added to state). Never assign it during render, since a value computed in render changes on every render, which is therandomcase above. - Keys need to be unique only among siblings, not globally.
- Index keys are acceptable only for a list that is static and never reorders, filters or inserts, such as the lines of a poem (the same React page makes this exception).
- If you filter or sort, the same id still identifies the same item, so inputs, focus and animations follow the item.
- Changing a key deliberately uses the same mechanism the
randomrun showed: a new key means a new instance with fresh state, so<Form key={userId} />starts from empty state wheneveruserIdchanges.
Trade-offs and pitfalls
- The index bug is silent when rows hold no state: a read-only list "works" with index keys, which is why it survives code review. It appears once a row gains an input, a checkbox, focus, a CSS animation or a fetch in an effect.
- An id is only a good key if it is truly unique and stable. Using a field like
namethat users can edit or that can collide produces the same class of bug. - This run is a single scenario (one prepend on three items). It shows the mechanism, not performance numbers.
Running the code
npm i react@18.3.1 react-dom@18.3.1 @testing-library/react@16.0.1 @testing-library/dom@10.4.0 jsdom@25.0.1
node keys.mjs # prints the three runs and "all assertions passed"
BREAK=1 node keys.mjs # id strategy replaced by index: assertion fails
Save the listing as keys.mjs beside a package.json containing { "type": "module" }. Tested on node:22 (v22.23).
Want to create your own tailored preparation guide using our deep research?
Get Started for FreeInterview-Ready Courses
Visual-first, interactive, structured learning paths
Browse Full-Stack Developer jobs
AI-enriched listings across hundreds of company career pages
Explore Jobs