Company Research and Business Understanding Questions

1. DAU/MAU (Engagement ratio)

• Why: Measures stickiness — how often monthly users become daily users; strong proxy for product–market fit and ad/monetization potential.
• Threshold/alert: DAU/MAU drops >15% week-over-week or ratio <10% (for a social app) triggers investigation.
• Engineering impact: Prioritize features/bugs that increase habitual use (notifications, feed relevancy), A/B test experiments, and optimize onboarding flow.

2. 7- and 30-day Retention (cohort retention)

• Why: Directly maps to long-term revenue and LTV; acquiring users is costly so retention multiplies ROI.
• Threshold/alert: 7-day retention decline >10% vs. baseline cohort or sudden cohort decay -> alert.
• Engineering impact: Focus on onboarding, personalization, feature parity fixes; implement instrumentation to trace drop-off points.

3. Conversion Rate (key funnel: install→signup→pay/goal)

• Why: Ties product changes to revenue or core KPI (subscriptions, purchases).
• Threshold/alert: Conversion falls >20% for any funnel step relative to rolling baseline.
• Engineering impact: Fix UX regressions, reduce friction, run experiments; prioritize payment reliability and analytics accuracy.

4. Error Rate / SRE Metrics (5xx, JS exceptions, API failure rate)

• Why: Directly affects user experience and retention; high error rates reduce trust and conversions.
• Threshold/alert: 5xx >0.5% of requests or spike x3 baseline => immediate pager.
• Engineering impact: Triage outages, rollbacks, root-cause fixes; invest in observability, retries, circuit breakers.

5. Core Web Vitals / Performance (LCP, TTFB, FCP)

• Why: Performance correlates with engagement, SEO, conversion and ad revenue.
• Threshold/alert: LCP >2.5s or 10% of users experience >4s → alert.
• Engineering impact: Prioritize critical-path optimization, CDN, asset bundling, lazy-loading, and performance budgets in PR checks.

For each metric, ensure instrumentation, alerting tied to ownership and runbooks, and that data feeds experimentation and roadmap prioritization.

Week 1 — Orientation & context

• Goals: environment bootstrapped (local build/tests pass), read high-level architecture, run the app end-to-end.
• Meet: manager (expectations), onboarding buddy, tech lead, product manager.
• Docs: README, architecture diagram, deployment runbook, coding standards, backlog board.
• Checkpoint (end week): demo a successful local run and a short write-up of system flow.

Week 2 — Codebase dive & tests

• Goals: read 2–3 core modules, understand data models/APIs, add/execute unit tests.
• Meet: pair with owner of core module, QA engineer.
• Docs: API specs, data schema, test strategy.
• Checkpoint: submit one small PR that adds a unit/integration test; get first review.

Week 3 — Low-risk first project

• Goals: deliver a small enhancement or bugfix touching one service; include tests and docs.
• Project: add a configurable feature flag, fix UI text bug, or add logging/metrics in a component.
• Meet: reviewer, DevOps for deployment checklist.
• Checkpoint: merged PR and deployment to staging; automated tests pass.

Week 4 — Ownership & ramping scope

• Goals: take ownership of a small ticket queue, participate in design discussion, propose minor refactor.
• Meet: cross-functional stakeholders for next sprint planning.
• Docs: contribute to or update relevant docs.
• Checkpoint: own 2–3 tickets, lead one code review.

Measurable indicators of productivity

• Local build + tests green within first 7 days.
• First merged PR by day 10–18.
• Number of accepted tickets (2–5) and reviews completed by day 30.
• Can explain end-to-end flow and deployment steps to team.
• Increasing independence in debugging and design conversations.

Mentoring & feedback cadence

• Daily standups + 3×/week short sync with buddy first 2 weeks, weekly 1:1 with manager. Midpoint (day 15) and 30-day review with actionable goals for next 60 days.

Immediate data to collect (first 0–15 minutes)

• Preserve state: take process snapshots / core dumps, export current heap/threads (jstack/jmap for Java; pprof for Go), collect container/pod YAML, and save current service logs to immutable storage.
• High-resolution telemetry: enable or increase sampling for distributed tracing (trace all errors and p95 requests), CPU/mem/GC metrics at 1s granularity, thread counts, blocking/lock metrics, event-loop lag (Node), and per-request latency histograms.
• Infra metrics: network I/O, socket counts, FD usage, disk I/O, context switches, and host/container CPU steal.
• DB and downstream: enable slow-query logging, connection pool usage, active transactions, lock/wait statistics, queue lengths.
• App-level: request paths, payload sizes, response codes, backend call latencies, feature flags—correlate p95 spikes with request types.

Key hypotheses to test (prioritized)

1. GC / stop-the-world pauses causing tail latency (check GC pause histograms, safepoint logs).
2. DB contention/locks or slow downstream calls (DB locks, long-running queries).
3. Connection pool exhaustion or timeouts (connection pool active=limit, queueing).
4. Resource starved hosts (CPU steal, I/O wait, noisy neighbor).
5. JIT or code path that runs under certain inputs (specific endpoint causing heavy work).
6. Network issues (packet loss causing retries).
7. Deploy/config change or cron job causing periodic load.

Experiments & investigative steps

• Correlate timestamps: join trace spans, logs, infra metrics to see which component spikes with p95.
• Increase trace sampling for p95 requests and capture full span stacks; add request-id propagation if missing.
• Run targeted flamegraphs / async-profiler during spike windows to find hot functions.
• Enable DB slow query and EXPLAIN problematic queries; instrument latency per SQL.
• Simulate increased concurrency locally/on staging with same request patterns to reproduce.
• Temporarily bump connection pool size on a single instance to see if latency reduces (A/B test).
• Toggle GC tuning (e.g., shorter pause collectors) on canary instance to validate GC hypothesis.
• Add short lived synthetic probes to downstream services to detect intermittent slowness.

Safe mitigations to stabilize latency (while preserving evidence)

• Scale horizontally: add extra instances behind LB to reduce queueing without restarting existing processes.
• Throttle or shed load at edge: enable rate limiting / degrade non-critical features (e.g., disable expensive analytics) with feature flags.
• Increase timeouts/queue depth cautiously for critical downstreams to avoid cascading failures; prefer circuit breakers.
• Add request prioritization: route critical checkout flows to warmed-up nodes or a separate pool.
• Avoid full redeploys/rolling restarts until snapshots/diagnostic captures are preserved; prefer canary changes.
• If restarting helps, snapshot logs and generate core dumps first; restart one instance to validate improvement.
• Keep elevated telemetry retention and turn on debug logging scoped to problematic endpoints to collect evidence for postmortem.

What success looks like

• Short-term: p95 spikes mitigated to acceptable SLO with minimal feature impact and preserved diagnostics.
• Root cause: reproducible trace/flamegraph/DB trace that identifies bottleneck and a targeted fix (query optimization, GC tuning, pool sizing, or code path refactor).
• Follow-up: permanent observability (end-to-end traces, latency budgets, alerting on queueing/GC pauses) and runbook updates.

Requirements & constraints:

• Maintain weekly releases while enabling independent deploys for multiple product teams within 12 months
• Minimize customer-visible risk, allow incremental rollback, keep CI fast

Phased strategy (quarterly milestones):

1. Phase 0 — Prep (0–2 months)

• Inventory modules, identify bounded contexts and seams (data ownership, traffic patterns)
• Add runtime feature flags and API gateway routing stubs
• Baseline metrics and SLAs

2. Phase 1 — Strangling & Read-only Extraction (2–5 months)

• Introduce adapter layer: route calls to monolith by default but allow per-team toggles
• Extract read-only services (e.g., reporting) behind stable APIs
• Build independent build pipelines (CI) per extracted component; keep single CD until validated

3. Phase 2 — Stateful Service Extraction & Independent CD (5–9 months)

• Extract stateful services with clear data ownership; use change-data-capture (CDC) or eventing for sync
• Implement separate CD pipelines: automated canary deploys, health checks, and fast rollback
• Move teams to own pipelines and deploy cadence

4. Phase 3 — Full Independence & Lifecycle Ownership (9–12 months)

• Decommission routes to monolith, finalize API contracts, enforce SLAs
• Cross-team governance for shared infra (service mesh, observability)

CI/CD changes:

• Modular CI: per-service pipelines with unit, integration, and contract-test stages
• Shared artifact registry, immutable builds, semantic versioning
• CD supports canary + automated metrics-based promotion, and feature-flag-driven rollouts
• Enforce pipeline as code and gated merges

Contract testing approach:

• Define explicit API contracts (OpenAPI/Protobuf) stored in a contract registry
• Consumer-driven contract tests (Pact or similar): consumer CI publishes expectations; provider CI verifies before deployment
• Add contract verification in CI and run nightly end-to-end smoke against staging

Validation & correctness steps:

• Start with local and CI unit/integration tests, then contract verification
• Canary deploys with synthetic and real traffic; compare key metrics (latency, errors, throughput) against baseline
• Deploy shadowing: duplicate traffic to new service without affecting responses
• Automated rollback triggers on SLA regressions, and manual runbooks for complex failures
• Gradual traffic shifting (10% → 50% → 100%) with health gates
• Post-mortem and telemetry review after each extraction

Risk mitigations & trade-offs:

• Use feature flags and gateway routing to reduce blast radius; adds operational complexity
• CDC/eventing avoids big DB migrations but requires eventual consistency handling
• Prioritize low-risk services first to build confidence and automation

This approach enables incremental migration, preserves weekly release stability, and gradually transfers ownership to product teams with automated safety nets.

Situation: We need to change an orders table in production (add columns, change types) with zero downtime.

Plan overview (expand-then-contract, non-blocking, backfill with checkpoints):

1. Prep: create feature flags and deploy app code able to read/write both old and new schema (dual-write mode).
2. Expand (non-blocking DDL):

• Add new columns with NULL/defaults in small transactions:

ALTER TABLE orders ADD COLUMN new_status VARCHAR(32);
ALTER TABLE orders ADD COLUMN new_total_cents BIGINT;

• If changing type (e.g. decimal -> bigint), add new column instead of ALTER TYPE.

3. Dual-write deployment: App writes both old and new columns. Read path continues using legacy columns while you backfill.

4. Batched backfill with checkpointing to avoid long locks:

• Use id ranges or updated_at cursor; process N rows per batch (e.g. 10k).
• Example backfill loop (pseudocode):

-- mark checkpoint = last_id
UPDATE orders SET new_total_cents = CAST(total*100 AS BIGINT)
WHERE id > :last_id AND id <= :last_id + :batch_size AND new_total_cents IS NULL;
-- record last_id after success

• Use transactions per batch, small timeout, retry on deadlocks. Persist checkpoint in migration table.

5. Verification queries after each batch/period:

-- sampling
SELECT COUNT(*) FROM orders WHERE new_total_cents IS NULL;
SELECT SUM(CASE WHEN new_total_cents != CAST(total*100 AS BIGINT) THEN 1 ELSE 0 END) AS mismatches;

6. Monitoring during migration:

• Track replication lag, per-batch duration, failure rate, DB CPU/IO, table bloat.
• Alert if replication lag > threshold, error rate spikes, or batch time grows.

7. Cutover (contract):

• When backfill complete and verification passes, switch reads to new columns via feature flag.
• Deploy app reading new fields.
• Wait a short stabilization window while monitoring.

8. Cleanup:

• Drop old columns in a final non-blocking ALTER TABLE DROP COLUMN (some DBs require rewrite; schedule during low traffic or use logical deletion flag first).

Rollback strategy:

• While in dual-write: disable feature flag to revert to legacy reads; stop writing new columns (keep old writes). Use checkpoint table to resume or abort backfill.
• If catastrophic DB issue: stop migration jobs, revert app to single-write legacy version, restore from recent snapshot if needed.

Trade-offs & notes:

• Avoid in-place type changes to prevent table rewrite.
• Use online schema tools (gh-ost, pt-online-schema-change) for MySQL if ALTER rewrites.
• Test on staging with production-like volume.