Software Development Lifecycle and Tradeoffs Questions

Covers fundamental software development lifecycle concepts and the technical tradeoffs made during product development. Topics include the lifecycle phases of requirements gathering and analysis, design, implementation and development, testing, deployment, and maintenance, and awareness of different lifecycle models such as waterfall, agile and scrum, and iterative development. Also covers practical engineering techniques and tradeoffs such as feature flags, split testing, blue green deployments, canary releases, technical debt, and how decisions affect velocity quality and maintainability. Emphasis is on understanding how individual engineering work fits into the broader process and how to reason about tradeoffs between speed cost scalability and code quality.

HardSystem Design

110 practiced

Describe a zero-downtime database migration strategy for adding a new column and backfilling its values across a large table used by multiple services. Include expansion and contraction phases, backfill orchestration, safe ordering of deploys across services, how to use feature flags to enable new code paths, and how to handle rollback if backfill or migration fails at scale.

Sample Answer

Requirements & constraints:- Zero downtime, support concurrent readers/writers across services, large table (millions–billions rows), ability to rollback safely, backfill must be resumable and non-blocking.

High-level plan:1. Expansion phase (make schema compatible)- Add new column as NULLABLE with a default of NULL (avoid costly table rewrite). If DB requires default rewrite, use ADD COLUMN without default + then ALTER to set default later.- Deploy schema migration first, before application code changes.

2. Backfill orchestration- Use an idempotent, chunked backfill job (e.g., worker pool, Airflow, custom service). Backfill by primary key ranges or time windows to avoid long transactions.- Make each chunk: SELECT for range -> compute value -> UPDATE rows in batches (e.g., 1k–10k) with retries and exponential backoff.- Use optimistic concurrency (WHERE column IS NULL) to avoid overwriting newer values; include version/timestamp if needed.- Rate-limit and throttle based on DB load; run during off-peak or with dynamic throttling tied to CPU/IO metrics.- Ensure resume: persist progress checkpoint (last processed id/range). Support parallel workers with non-overlapping ranges.

3. Safe deploy ordering across services- Deploy in this order: a) Schema migration (nullable column). b) Deploy consumers/writers that can write to both old and new columns but still read from old primary source — use dual-writes if feasible (write to new column while continuing to write existing path). c) Start backfill. d) Switch readers to prefer new column behind a feature flag (read-new fallback-old). e) Once reads validated, flip full read to new column and stop using old column. f) Contraction: make column NOT NULL (set default if needed), remove old column and dual-write code in a later deploy.

4. Feature flags usage- Wrap new read/write paths behind flags with progressive rollout: - Start with internal traffic or a small % of requests. - Flags control: enable-write-to-new, enable-read-prefer-new.- Use flags to quickly disable new code without rollback.

5. Rollback and failure handling- If backfill causes errors or overload: stop/pause backfill jobs, revert feature flags to read from old column, and revert writers to stop dual-writing if necessary.- Because column was added nullable and reads default to old column until switched, rollback is safe.- For logical data corruption: run compensating backfill or use backups/snapshots; ensure backfill is idempotent so re-running fixes partial failures.- If schema change must be reverted: remove column only after ensuring no code depends on it; dropping is destructive — prefer leaving column and disabling use, then drop in maintenance window if needed.

6. Monitoring & safety- Monitor latency, DB CPU/IO, replication lag, error rates, backfill progress, and feature-flag metrics.- Add alarms to auto-pause backfill on threshold breaches.- Smoke tests and canary checks after each rollout step.

Trade-offs:- Dual-writes add complexity and eventual consistency risk; avoids long blocking migrations.- Fully rewriting column in-place is faster but risky for downtime — avoided at scale.

This plan ensures zero-downtime behavior by adding schema compatibility first, backfilling safely and resumably, progressively switching services behind feature flags, and providing clear rollback paths.

EasyTechnical

70 practiced

Compare trunk-based development and git-flow branching models. For a team practicing continuous delivery with multiple daily deployments, recommend a branching strategy and explain tradeoffs around merge conflicts, release isolation, CI load, and developer experience. Describe how feature flags influence your recommendation.

Sample Answer

Trunk-based development (TBD) and git-flow are two common branching models with different tradeoffs.

Definitions:- Trunk-based: short-lived branches or direct commits to main/trunk; integrate frequently (multiple times per day).- Git-flow: long-lived feature branches, develop/release branches, and maintenance branches; releases are prepared on dedicated release branches.

Recommendation:For a team doing continuous delivery with multiple daily deployments, I recommend trunk-based development plus disciplined feature flags.

Why (tradeoffs):- Merge conflicts: TBD reduces large, painful conflicts because integrations are small and frequent. Short-lived branches (hours) keep diffs small. Git-flow’s long-lived branches increase conflict likelihood and costly merges.- Release isolation: Git-flow gives explicit release branches, which can simplify final QA. But with feature flags, TBD achieves release isolation by gating unfinished work without branching—features merged to trunk but toggled off.- CI load: Git-flow spawns many long-running CI pipelines for each branch and release; TBD focuses CI on trunk and short branches, leading to predictable, optimized pipelines. However, high-frequency merges require fast, reliable CI to prevent noise.- Developer experience: TBD encourages fast feedback, simpler workflow, and less context switching. Git-flow can be more comfortable when teams aren’t ready for CD, but adds process overhead.

Role of feature flags:Feature flags are essential for TBD in CD: they allow incomplete features to be merged safely, enable gradual rollouts, and support hotfixes without branching. Use flag lifecycle (create, rollout, cleanup) and strict toggle ownership to avoid technical debt.

Practical guardrails:- Enforce trunk builds green, small PR sizes, automated tests, and fast CI.- Use short-lived topic branches (<24–48h) if needed.- Automate flag cleanup and track flags in code/issue trackers.

Summary: For multiple daily deployments, trunk-based + feature flags offers faster feedback, fewer merge headaches, and better developer flow; invest in CI, testing, and flag hygiene.

MediumTechnical

64 practiced

Describe a safe rollback strategy for a coordinated deployment that touched three interdependent microservices A -> B -> C. After rollout, service B shows increased errors. Explain how you would diagnose whether to roll back only B, roll back all services, toggle behavior with flags, or take other mitigations. Include considerations for database migrations, schema compatibility, and minimizing customer impact.

Sample Answer

Situation: We just completed a coordinated rollout for A → B → C and monitoring shows B’s error rate spiked after deployment.

Approach (diagnose then decide):1. Immediate triage (fast, low-impact)- Roll forward monitoring to pinpoint error type, endpoints, and time window (logs, traces, metrics).- Check distributed traces to see whether errors originate inside B’s new code, or from inputs from A or downstream C.- Verify deployment metadata: which versions of A/B/C are live and when.- Look for correlated infra issues (resource spikes, network errors, config/secret changes).

2. Decision criteria to rollback B only vs all:- Roll back only B if: - Traces show exceptions thrown within B’s new code paths, independent of changes in A/C. - B’s new binary or config introduced a regression (e.g., null pointer, bad parsing). - No incompatible schema or irreversible DB changes were applied by B.- Roll back all (A,B,C) if: - Errors stem from interface contract mismatches (A sends payloads expecting new B behavior) and quick single-service rollback would leave A or C in incompatible states. - A or C rely on new behavior that B no longer provides when partially rolled back.- Use feature flags/toggles when: - The risky change is feature-scoped and behind a flag that can be turned off quickly across services. - You can disable the feature in B without code rollback, and A/C gracefully handle the feature-off behavior.

3. Database & schema considerations- Never roll back to code that expects older schema if forward migration was destructive (dropped columns, transformed data).- Prefer backward-compatible DB migrations (additive changes, tolerant reads). If B performed a non-reversible migration, do not rollback B without plan to migrate data back — consider hotfix or feature-flag to bypass problematic code path.- If migration occurred only in C or A, ensure B’s rollback won’t reintroduce writes incompatible with migrated schema.

4. Minimizing customer impact- Gradual mitigation: toggle feature flags, reduce traffic to new B (traffic-splitting, circuit-breakers), or route a percentage to the previous version.- If errors affect only a subset of endpoints, disable those endpoints selectively.- Communicate externally/internal status and estimated remediation times.

5. Remediation steps- If quick fix exists, patch B, run smoke tests in staging, deploy canary, then ramp.- If rollback chosen: perform controlled rollback of B in one region/cluster first, validate, then global.- Post-incident: capture root cause, add more health checks, improve schema practices, and add a runbook for coordinated rollbacks.

This process prioritizes accurate diagnosis, avoids unnecessary full-system rollbacks, and treats DB migrations as a hard constraint when deciding safe rollback scope.

MediumTechnical

88 practiced

Compare the strategies of rebuilding container images separately for each environment versus promoting a single immutable image artifact from dev -> staging -> prod. Discuss traceability, reproducibility, vulnerability scanning, and operational simplicity. Recommend an approach and describe how to implement artifact promotion in CI/CD.

Sample Answer

Short answer: promoting a single immutable image artifact from dev → staging → prod is generally preferable to rebuilding per-environment because it maximizes traceability, reproducibility, consistent vulnerability posture, and simplifies operations. Below I compare both and give a recommended implementation for CI/CD artifact promotion.

Comparison- Traceability: Single immutable image gives one canonical SHA (image digest) you can trace back to a git commit and build metadata. Rebuilding per-environment produces different digests and makes it harder to prove "what ran where."- Reproducibility: Promoting the same artifact ensures the exact same binary executes across environments. Rebuilds risk non-determinism (tooling, base image updates, build args) unless you invest in fully hermetic builds.- Vulnerability scanning: Scanning one artifact once (and re-scanning on policy changes) is efficient and consistent. Rebuilding per-env may introduce new vulnerabilities (e.g., different base layers) and forces repeated scans.- Operational simplicity: Promotion simplifies CI/CD pipelines (build once, promote many). Rebuilds complicate rollback, auditing, and debugging because artifacts diverge.

Recommendation- Use immutable artifacts (container images tagged by digest and semantic metadata) and promote them through registries or a metadata store rather than rebuilding.

Implementation in CI/CD (high-level)1. Build stage: - Build image once from a specific git commit and build pipeline; tag with git SHA and version; push to registry; store metadata (commit, build logs, SBOM) in an artifact store.2. Security & QA: - Run automated unit/integration tests, produce SBOM, run SCA/vulnerability scanners, sign the image (Cosign/Notary).3. Promotion: - If tests & policies pass, mark the artifact as promoted (set a registry tag like staging/prod that references the same digest, or maintain a promotion record in an artifact catalog like Nexus/Artifactory or OCI repository with promotion labels). - Use immutable digests (pull by @sha256) in deployment manifests.4. Deployment: - Deploy the promoted digest to target environment via CD agent. For prod, require additional approvals and cryptographic signature verification.5. Rollback & auditing: - Rollback by redeploying a previous digest. Keep audit logs linking digest → commit → CI results → scans.

Best practices- Pull by digest, not mutable tag.- Store SBOMs and vulnerability scan results together with the artifact.- Sign images and verify signatures in CD.- Automate promotion with gates: automated tests, security policies, manual approvals where required.- Use immutable registries and retention policies to prevent digest deletion.

This approach reduces risk, simplifies debugging, and enforces consistent security posture across environments.

HardTechnical

80 practiced

Two teams manage the same Kubernetes namespace via GitOps and submit concurrent Git changes: one updates a Service and the other an Ingress that conflict, both merged and applied, causing an outage and cluster drift. Design policies, automation, and repo structure changes (e.g., repo-per-app, app-of-apps), admission controls, merge-queue behavior, and recovery automation to prevent and recover from such conflicts while preserving developer velocity.

Sample Answer

Approach: treat this as a governance + automation problem: prevent conflicting concurrent edits by making ownership explicit, shifting conflicts left into CI, enforcing runtime consistency with admission controls, and enabling fast automated recovery when drift occurs — all while keeping self-service for developers.

Design (repo & ownership)- Repo-per-app (or repo-per-team) with an app-of-apps umbrella repo for cluster-wide composition. Each app repo owns its k8s resources (Service, Deployment, ConfigMap). Ownership enforced by labels/annotations (team:, owner:) and a CODEOWNERS file in manifests.- Shared infra (Ingress, cluster-wide Services, DNS) lives in dedicated infra repos with clear ownership and approval gates.

Pre-merge automation (left-shift)- CI renders resources (helm/kustomize) and runs a “merge-simulation” that applies resources into a disposable test cluster or uses kubectl diff / server-side dry-run against a staging namespace to detect resource-level conflicts (name+kind) and semantic conflicts (port collisions, selector mismatches).- A Git merge-queue (e.g., Batcher or Mergify/custom service) serializes merges per resource ownership group: parallelism across unrelated resources, serialized for same resource keys. The queue runs pre-merge validations and rejects merges that introduce conflicts.- Add an automated “policy check” step using OPA/Gatekeeper/Conftest: enforce ownership, immutable fields, allowed hostnames, port ranges, and that Ingress host rules conform to team mapping.

Admission controls (runtime safety)- Cluster admission via OPA/Gatekeeper + validating webhooks to reject changes that violate policies (e.g., two different owners modifying same resource name in same namespace, Ingress host ownership mismatch, or Service->Ingress binding mismatch). Use a mutation webhook to inject ownership labels from cert-authority if missing.- Use namespace-level resource quota and network policies to reduce blast radius.

GitOps operator & sync behavior- Use ArgoCD/Flux in app-of-apps mode. Configure “sync waves” and health checks. Enable automated rollback on failed syncs and use “prevent drift” mode to alert on out-of-band changes.- Require GitOps operator to only sync manifests owned by the app (via path or label selector). Disallow overlapping paths across repos.

Conflict detection & recovery automation- Continuous drift detection: alerting when live != desired. On detected conflict/outage, automated remediation: 1. Block further merges via repository-wide merge freeze (API toggle from CI) and mark the merge-queue paused. 2. Auto-generate a revert PR for the offending merge(s) (CI inspects git history and creates revert commits). Run the same validation pipeline against the revert and auto-merge if checks pass and owner approvals present. 3. If rollback fails or needs immediate action, trigger ArgoCD rollbacks to last healthy commit and run postmortem automation to collect manifests, diffs, and health checks.- Implement a human-in-loop incident flow: Slack/PagerDuty alerts with one-click runbook actions (revert, rollback, pause-sync).

Preserving velocity- Fast feedback loops: run expensive checks (render + diff) in parallel and report results as PR status checks; use predictive merge-queue to avoid long blocking. Provide self-service tools (local kustomize/helm wrappers, pre-commit hooks) so developers catch conflicts before PR.- Fine-grained serialization: only serialize merges that touch same resource keys; unrelated PRs proceed concurrently.

Trade-offs and rationale- Repo-per-app reduces conflicts and clarifies ownership but increases number of repos — mitigated by app-of-apps composition.- Admission policies may initially block some workflows; provide exception workflows and fast appeals to maintain momentum.- Automated reverts speed recovery but require solid validation to avoid flapping.

Metrics & governance- Track time-to-detect, mean-time-to-recover, number of merge conflicts, number of rejected PRs by policy. Run quarterly audits of ownership and policy effectiveness.

This design prevents and detects conflicts early, enforces ownership at merge and runtime, automates safe recovery, and balances control with developer velocity via targeted serialization and fast CI feedback.

Unlock Full Question Bank

Get access to hundreds of Software Development Lifecycle and Tradeoffs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.