FAANG-Standard Interview Preparation Guide: Junior Level DevOps Engineer

Direct answer

HEAD normally points at a branch name (you're "on main"), which in turn points at a commit. Detached HEAD means HEAD points directly at a commit instead of at a branch name. Any new commits you make in that state aren't attached to any branch, and the moment you check out something else, they become unreachable and eventually eligible for garbage collection. It happens most often from checking out a specific commit hash, a tag, or a remote-tracking branch directly (e.g. origin/main) rather than a local branch.

Structured elaboration

How it occurs: git checkout <commit-hash>, git checkout <tag>, or git checkout origin/main (checking out a remote-tracking ref instead of your own local main) all put you in detached HEAD. Git warns you explicitly when this happens.

Two ways to recover:

1. If you haven't switched away yet: run git switch -c recovery-branch (or the older git checkout -b recovery-branch). This immediately creates a new branch pointing at HEAD's current commit, attaching whatever you committed to a real branch before it can be lost.

2. If you already switched away and the commits look gone: use git reflog, git's local record of every place HEAD has pointed. By default git prunes reflog entries after 90 days, but that window is shorter, 30 days by default, specifically for entries pointing at a commit that isn't reachable from any branch, exactly the detached-HEAD-then-switched-away case, so don't wait too long. Find the lost commit's hash in the reflog, then git branch recovery-branch <hash> to reattach it (or git checkout <hash> followed by git switch -c recovery-branch).

Worked example

Illustrative transcript, hashes are placeholders (yours will differ):

$ git checkout a1b2c3d
Note: switching to 'a1b2c3d'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
...
$ echo "quick fix" >> notes.txt
$ git add notes.txt
$ git commit -m "quick fix"
[detached HEAD 9f8e7d6] quick fix

At this point, running git switch -c rescue-fix attaches commit 9f8e7d6 to a new branch called rescue-fix, and it's safe. If instead the developer had run git switch main first without creating a branch, 9f8e7d6 would appear "lost," recoverable via git reflog, which would show an entry like 9f8e7d6 HEAD@{1}: commit: quick fix, giving the exact hash to reattach with git branch rescue-fix 9f8e7d6.

Trade-offs and pitfalls

Detached HEAD itself isn't dangerous, it's a normal, intentional state for briefly inspecting old code or a tag. The risk is entirely about committing while in it and then forgetting you're not on a branch. The reflog is a genuinely strong safety net, but it's local-only (it lives in your .git directory, not on any remote) and it does eventually expire unreachable entries, so the sooner you create a branch for work you want to keep, the safer you are.

Direct answer

Use service virtualization or mocks for dependencies you don't need real behavior from and want deterministic, fast responses; use a real staging-like cluster when the interaction between services (real timing, real data flow, real failure modes) is what the test exists to validate. When flakiness comes from inter-service timing rather than the tests themselves, the fix is almost always improving isolation and determinism in how services are wired together, not adding more retries to mask it.

Structured elaboration

• When to mock: a dependency that's genuinely external to what's under test, has well-known, stable behavior, or would introduce timing variance you don't want to test against right now (a third-party payment gateway's real latency profile, for instance).
• When to use real services: when the actual point of the test is the interaction itself (does service A correctly handle service B's real async event ordering, does a real network partition get handled gracefully); mocking that away would test nothing meaningful.
• Seeding data: for multi-service E2E tests, seed each service's data store independently but consistently (shared identifiers, consistent timestamps) so the services agree on the state of the world at test start, rather than relying on each service's own default/seed data lining up by coincidence.
• Reducing timing-driven flakiness: the actual fix for inter-service-timing flakiness is rarely "add a longer sleep" (which just slows tests down and often still fails occasionally); it's making the test wait on an explicit readiness signal (a health check, a specific event, a polling check with a bounded timeout) rather than a fixed delay, and reducing accidental coupling between services that don't need to interact for this specific test.
• The noisy-neighbor variant: when a shared test cluster serves many concurrent test runs, apparent "flakiness" is often contention between unrelated runs (shared rate limits, shared connection pools, resource starvation) rather than a timing bug in any individual test; moving to per-run isolated capacity (even lightweight namespace-level isolation) removes this class of failure entirely rather than trying to tune around it.

Worked example

A test verifying an order-placement flow across three services was intermittently failing because the third service's async event consumer sometimes hadn't processed the order-created event by the time the test asserted on it. Replacing a fixed 2-second sleep with a bounded poll-until-condition (check for the expected downstream state, with a timeout and clear failure message if it's never reached) eliminated the flakiness because the test now waits exactly as long as needed rather than guessing a fixed delay that was sometimes too short.

Trade-offs & pitfalls

Adding sleeps or blanket retries to tests exhibiting timing-related flakiness treats the symptom rather than the cause, and tends to make the test suite both slower and only slightly more reliable rather than genuinely fixed; the real fix is almost always an explicit readiness/completion signal the test can poll or wait on.

Direct answer

For a large microservices organization where a PR often touches multiple services, the pipeline needs to detect exactly which services a change actually affects, build and test only those, and give fast PR feedback (targeting roughly 10-15 minutes) while still coordinating safely across services that must deploy together or in a specific order.

Structured elaboration

Change-impact detection. The pipeline needs a mapping from changed files to affected services, either a simple path-based rule (a change under services/payments/ affects the payments service) or, for shared libraries, a real dependency graph so a change to a shared package correctly triggers every service that depends on it. Path-based rules are cheap to build and cover most cases; the gap is transitive dependencies through shared code, which needs either a maintained dependency graph or, as a safe fallback, treating a shared-library change as affecting everything until the graph is trustworthy enough to narrow it.

Fast PR feedback at scale. Only build and test the affected services identified above, run their fast unit/lint checks on every PR, and reserve slower cross-service integration tests for merge or scheduled runs where they don't block an individual developer. Caching (dependency and build-output) and parallelizing across the affected services (rather than serializing them) both compound with the impact-detection narrowing to keep the PR path fast even as the org and codebase grow.

Artifact and cache sharing. Services share build caches and, where they share dependencies, dependency caches, so building service B right after service A doesn't redundantly re-resolve identical shared dependencies.

Coordinating interdependent deployments. Most changes are independently deployable and should be: each service has its own pipeline and deploys on its own schedule without waiting on unrelated services. For the genuine minority of changes that require coordination (a breaking API change to a shared contract, a schema change multiple services depend on), the pipeline needs an explicit mechanism, not an implicit hope that timing works out: contract tests that fail loudly if a producer breaks a consumer's expectations, and, where strict ordering matters (a new field must exist before a consumer can read it), a deployment order expressed as pipeline metadata or dependency declarations, with backward-compatible rollout patterns (additive-first, remove-later) preferred over forcing simultaneous coordinated deploys wherever possible, since simultaneous coordination across many independent teams is inherently fragile.

Worked example

A monorepo with 30 services: a PR touching only the checkout service triggers checkout's own build, lint, and unit tests (targeting under 10 minutes), using cached dependencies shared across services. A PR touching a shared common-auth library triggers build and unit tests for every service that depends on it (identified via a maintained dependency manifest), and additionally runs consumer-driven contract tests against each dependent service's expectations. Cross-service integration tests run on merge to main, not on every PR, against a shared ephemeral environment. A breaking change to the checkout service's public API is rolled out additive-first (new field added and dual-written) so the checkout and inventory services don't need a simultaneous, coordinated deploy.

Trade-offs and pitfalls

The most common mistake at this scale is a change-impact system that's imprecise in the unsafe direction (missing a real dependency and under-testing a change) rather than the safe direction (over-triggering and testing more than strictly necessary); when in doubt, the fallback should widen the affected set, not narrow it. The second is under-investing in the explicit coordination mechanism for the genuinely cross-service changes, assuming teams will informally coordinate timing, which breaks down as team count grows; a small number of changes genuinely need real cross-service coordination tooling, even though the vast majority of changes should never need it.

Approach overview
I would treat this as a layered network + CI reliability problem: reproduce, collect evidence, then apply short-term mitigations (retries/caching) and long-term fixes (mirrors, network/runners).

Reproduce & gather data

• Re-run failing job with increased logging (apt/yum/pip verbose, curl --trace) and capture timestamps, HTTP status codes, and transient IPs.
• Correlate failures with runner location, time-of-day, specific package hosts.

Immediate mitigations

• Add exponential backoff retries around package installs (e.g., apt-get with --retries/apt-fast; pip with --retries and --timeout). Example pattern: retry 3 times with backoff 1s, 4s, 8s.
• Use transient-failure tolerant wrappers in Dockerfile (shell loop or small retry helper).

Caching & mirrors

• Enable CI-level Docker layer cache and package caches (squid/apt-cacher-ng or pip wheel cache) to avoid repeated network fetches.
• Configure reliable mirrors or CDN endpoints for package managers (apt sources, PyPI mirror/simple index, npm registries). Prefer closer-region mirrors or company internal mirror.
• Prebuild base images with dependencies and store in a registry.

DNS / proxy / networking debugging

• In failing runner, run dig/nslookup and curl to package hosts to check resolution and latency; capture traceroute to pinpoint where 502/timeouts occur.
• Check resolver config ( /etc/resolv.conf ), use stable resolvers (Cloudflare/Google) or local caching resolver.
• If behind proxy, validate proxy health, authentication, and connection pooling.

CI runner / infra considerations

• Ensure runners have adequate ephemeral port ranges, MTU settings, and NAT resources; watch for connection exhaustion.
• Use runners in multiple AZs/regions and stickiness to healthy ones, or autoscale to avoid overloaded hosts.
• Harden network egress: increase TCP/HTTP timeouts for CI, maintain HTTP keep-alive, and allow larger connection limits.

Long-term

• Instrument metrics and alerts on package fetch failures, cache hit ratio, and runner network errors.
• Maintain internal artifact registry/mirror, prebuilt images, and documentation to reduce external dependencies.

This combination gives quick reliability gains (retries, cache) and sustainable fixes (mirrors, runner config, observability).

The architecture is a pipeline of five stages, each trading some fidelity for cost, plus a reconstruction path at query time for the cases where someone genuinely needs the detail back. The stages: label reduction, aggregation and rollups, sampling, storage tiering, and retention.

Pipeline

flowchart LR
  A[Raw High-Cardinality Metrics] --> B[Label Reduction]
  B --> C[Aggregation and Rollups]
  C --> D[Sampling]
  D --> E[Storage Tiering]
  E --> F[Retention Enforcement]
  F --> G[Query Layer]
  G --> H[On-demand Rehydration: top-K exemplars]

• Label reduction: the single highest-leverage step. Cardinality is combinatorial across independent label dimensions: if endpoint has 80 distinct values, status has 6, pod has 800, and region has 4, the bounded series count is their product.

Adding one more label with effectively unbounded cardinality (like a raw request_id, close to one unique value per request) multiplies this by request volume instead of by a fixed factor, turning a bounded 1.5M-series metric into an unbounded one. Label reduction means identifying and removing or bucketing exactly those unbounded dimensions before ingestion, keeping the combinatorially-bounded ones.

• Aggregation and rollups: pre-compute coarser-grained series (per-service instead of per-pod, for example) at ingestion time so most dashboard and alert queries never touch the raw high-cardinality series at all.
• Sampling: for label values that are useful individually but too numerous to keep in full (e.g., per-customer-ID series for a B2B product with thousands of customers), keep exact series for the top-K by volume or spend, and sample or bucket the long tail.
• Storage tiering: recent raw data on fast storage, older data downsampled and moved to cheaper storage, as in a standard hot/warm/cold retention policy.
• Retention enforcement: hard expiry so the pipeline's cost stays bounded over time regardless of how ingestion volume trends.

Reconstructing detail on demand

The trick that makes this acceptable to users is that "finer detail" doesn't mean "keep everything forever," it means "keep enough breadcrumbs to go get the detail from a cheaper source when someone actually asks." Two mechanisms:

1. Exemplars: attach a sampled trace ID or raw event reference to an aggregated metric bucket, so a spike in the aggregate can be drilled into by fetching the handful of exemplar traces/logs that were kept in full, even though the metric itself was aggregated.
2. On-demand reprocessing: if raw pre-aggregation data still exists in a cheap cold tier (e.g., unindexed compressed blobs), a rare deep-dive query can trigger an offline reprocessing job rather than requiring the hot path to keep everything queryable in real time.

Sizing against a cost ceiling

Given an illustrative monthly storage budget of $5,000 and an illustrative unit cost of $0.02/GB-month (a labeled assumption for this worked example, not a live vendor quote):

Using the 1,536,000 bounded series from the label-reduction step above, at a 5-minute downsampled resolution with 8 bytes/point (consistent with the multi-aggregate downsampling estimate used for retention-tier design):

series = 1_536_000
bytes_per_point = 8
interval_s = 300
points_per_day = 86400 / interval_s
bytes_per_day = series * points_per_day * bytes_per_point   # 3.539 GB/day
max_days = (250_000 * 1e9) / bytes_per_day

That affords roughly 70,600 days of 5-minute-resolution history under the budget, which is obviously far beyond any real retention need, so the budget is not actually the constraint at this series count and resolution. Repeating the same calculation for raw 15-second resolution at 2 bytes/sample instead gives 17.69 GB/day and about 14,128 affordable days, roughly 5x shorter than the downsampled case (the 20x fewer points at 5-minute resolution is partly offset by needing 4x more bytes per point to store min/max/sum/count instead of a single value, netting exactly 20/4 = 5x). The concrete lesson: at this series count, the budget comfortably covers years of retention either way, so cost pressure at 1.5M series is not what forces sampling or tiering, it forces label reduction to happen so the series count never gets to the point where the arithmetic above breaks down (e.g., adding the unbounded request_id label would blow past 250 TB in a matter of hours).

Trade-offs and pitfalls

• Treating sampling as the first line of defense (instead of label reduction) is the most common mistake: sampling a metric whose cardinality is unbounded because of a labeling error still leaves an unbounded number of series, each just sampled less; the series count itself, not just the sample rate, is what needs to be bounded first.
• Aggregation destroys the ability to answer "which specific instance caused this" without exemplars; a design that aggregates without keeping any drill-down path trades away debuggability that's expensive to get back later.
• A fixed cost ceiling naturally reframes the problem: it's not "how do we store everything cheaper," it's "what data can we afford to keep at what fidelity," and that framing should drive which stage of the pipeline (label reduction vs. sampling vs. tiering) absorbs the cost pressure, since as shown above they don't interact linearly.
• Query-time reconstruction only works if the cheap cold tier is actually queryable, even slowly; if cold data is written in a format nothing can read without a bespoke recovery process, "reconstruct on demand" is really "data is gone" with extra steps.

Direct answer
Three concrete smoke tests for a service, each targeted at a different class of catastrophic failure, and each paired with the response that actually fits its severity and blast radius: a health-endpoint check (auto-rollback on failure, since it signals the process itself may be broken), an end-to-end transaction through a specific new/flagged code path (disable via feature flag on failure, since that's the fastest way to shed just the risky new logic without discarding the rest of the release), and a critical-dependency reachability check (alert or auto-rollback depending on whether the dependency is critical-path, since not every degraded dependency justifies an immediate rollback).

Structured elaboration

1. Health endpoint: GET /healthz returns 200. Verifies: the process started, is listening, and basic internal wiring didn't crash on boot. Success criteria: HTTP 200 within a short timeout (a few seconds). On failure: auto-rollback, immediately and without waiting for further evidence, since a process that isn't even up is the most severe and least ambiguous signal there is; there's no narrower fix available at this layer.
2. End-to-end core transaction through a flagged new code path: for a checkout service, place a test order through a sandboxed test account, specifically exercising a new pricing-calculation path that shipped behind a feature flag. Verifies: the ACTUAL new business logic works, not just that the process is alive. On failure: disable via feature flag first, not a full rollback, since the failure is isolated to logic that's already gated behind a flag; flipping the flag off falls back to the previous, known-good pricing path instantly across all instances without discarding the rest of the release. Escalate to a full auto-rollback only if disabling the flag doesn't resolve the failure (meaning the regression isn't actually confined to the flagged path).
3. Critical-dependency reachability: confirm the service reports its database and payment-processor connections as healthy (via the health endpoint's detailed response or a separate dependency-check endpoint). Verifies: the new version can actually reach what it needs. On failure: the response depends on which dependency: if the payment-processor or primary database is unreachable, auto-rollback, since the service will degrade further as traffic increases; if a non-critical, degradable dependency (a recommendation service, a non-blocking analytics sink) is unreachable, alert-only and let a human decide, since the core service can keep functioning in a degraded mode and an automatic rollback would be an overreaction to a non-blocking issue.

Worked example
Immediately after deploy: /healthz returns 200 (pass). The flagged new pricing path's test order returns an incorrect total (fail); the pipeline flips the pricing feature flag off, and a re-run of the same test order now returns the expected total, confirming the flag disable resolved it without a full rollback. Separately, the dependency check shows the payment-processor connection as unreachable (fail); because this is a critical-path dependency, this triggers an automatic rollback of the whole release regardless of the pricing-flag outcome, since a service that can't reach its payment processor will fail broadly once real traffic hits it.

Trade-offs and pitfalls
The common mistake is treating every smoke-test failure the same way (blanket auto-rollback for everything), which discards a release's healthy majority just to fix a narrow, flag-isolated regression, and is slower in practice since the whole release then has to be re-shipped and re-verified from scratch. The opposite mistake, alerting on everything and waiting for a human, is too slow for unambiguous, severe failures like a dead health endpoint. Matching the response to the test (full rollback only when the failure isn't narrowly containable, feature-flag disable when it is, alert-only when the dependency isn't on the critical path) gets the fastest safe recovery in each case rather than one blunt instrument for every failure.

Direct answer
TIME_WAIT is the state the side that sent the FINAL ACK of a connection close sits in for a fixed period (commonly twice the maximum expected segment lifetime, often around 60 seconds on Linux) before the connection's resources are fully released. It exists so a delayed, duplicate packet from an old connection can't be mistaken for part of a brand-new connection reusing the same address/port pair. A half-open connection is one where only one side still believes the connection is alive; the other side has already reset, crashed, or otherwise abandoned it without a clean FIN exchange.

Structured elaboration
TIME_WAIT exists to protect two things: (1) it guarantees the final ACK the closing side sent actually gets through, by giving time to retransmit it if the peer's FIN gets retransmitted (meaning the ACK was lost); (2) it prevents a stray, delayed packet from a previous incarnation of a connection (same 4-tuple: source IP, source port, destination IP, destination port) from being delivered into a brand new connection that happens to reuse the same 4-tuple before the old segments have had time to disappear from the network.

To detect a large number of sockets stuck in TIME_WAIT on Linux, ss -tan state time-wait | wc -l (or the older netstat -ant | grep TIME_WAIT | wc -l) gives a live count; watching this metric over time distinguishes a normal, self-draining backlog from a genuine problem.

Worked example
A server that closes millions of short-lived outbound connections per hour (for instance, a service making one HTTP call per request to an upstream) can exhaust its available ephemeral source ports if TIME_WAIT sockets accumulate faster than they expire, because each TIME_WAIT socket still holds its port reserved. The two standard mitigations are: raise the number of available client-side (ephemeral) ports and/or reuse connections via keepalive/connection pooling so fewer connections churn through TIME_WAIT in the first place; and, on the SERVER side specifically, enabling SO_REUSEADDR and, where safe, tcp_tw_reuse lets a new outgoing connection reuse a TIME_WAIT 4-tuple once TCP timestamps confirm it's safe to do so, rather than waiting out the full timer.

Trade-offs & pitfalls
Disabling or drastically shortening TIME_WAIT globally (rather than tuning port ranges or reuse settings) is the wrong fix: it reintroduces the exact correctness problem TIME_WAIT was designed to prevent, stray old packets landing in a new connection and corrupting it. The safe levers are reducing HOW MANY connections churn through the state (pooling, keepalive) and widening the ephemeral port range, not shrinking the safety window itself.

Direct answer

A promotion packet is a structured evidence file built over months, not written the week before the review. It has an impact summary, a scope and ownership section, a section that explicitly counts non-technical contributions, and a visibility trail showing others already recognized the work before the packet existed. The packet documents evidence, the persuasive conversation that uses it is a separate skill and belongs in the room, not on the page.

Structured elaboration

The non-technical ledger is the section most packets under-build. Mentoring and documentation are real promotion evidence and should be quantified in terms you can actually verify, not vague credit.

The visibility trail has two strong levers. Turning a side project into a visible internal product, something that started as personal initiative and is now relied on by others, and building an external technical brand, a talk, an open source contribution, or public writing, that puts your name on the work before the packet does.

Building visibility so the packet isn't a surprise means never letting it be the first time your manager, or their manager, hears about the work in it. Share progress in venues that already exist, be deliberate about which pieces of work you narrate publicly, and invest consistently in one or two visibility levers rather than spreading thin.

Worked example

"Over the year before my review cycle I kept a running log of contributions as they happened, not reconstructed at the end. When I built a small internal tool to solve a recurring problem on my team, I didn't stop at solving my own problem, I documented it, offered it to an adjacent team, and it ended up adopted as the default approach for that class of problem, a side project that became a small internal product. Separately I wrote up a design decision as an internal post, which a colleague on another team referenced months later solving a similar problem, so I had a quoted instance of influence beyond my own team. When I assembled the packet, the non-technical ledger included the two people I'd onboarded that year with specific notes from them on what helped, plus a piece of documentation that became the team's reference for a process I'd designed. None of this was manufactured for the packet, it was work I'd done anyway, tracked as I went."

Trade-offs & pitfalls

• Building the whole packet in the final month reads as reverse-engineered and gives peers no time to corroborate it.
• Treating mentoring and documentation as filler instead of first-class evidence under-sells real work that reviewers weight more than most candidates expect.
• Over-investing in external visibility at the expense of internal scope evidence. External brand supports the case, it doesn't replace it.
• Keep the packet as evidence and artifacts, save the persuasive framing and objection handling for the conversation itself, or the packet reads as a sales pitch instead of a record.

Direct answer

Decide what to delegate by looking at two things: where the task sits relative to the person's current skill level, and what happens if they get it wrong. Delegate work that stretches them but is reversible or cheap to fix. Keep for yourself work that needs context you can't hand off in time, decisions whose blast radius exceeds the trust you've built with this person so far, or one-off tasks where teaching would take longer than doing it. Treat each handoff as a deliberate intervention, not an offload: pick the task for the specific gap it targets, define what "done" looks like up front, and calibrate how much support comes with it.

Decision framework

Match difficulty to their zone of growth. Too easy and it's busywork with no development value. Too hard with no support and it's discouraging or risky. The sweet spot is a task just past what they've done independently before.

Weigh reversibility, not just difficulty. Prefer delegating decisions that are cheap to undo (a first draft, a component design, a low-stakes customer interaction) over ones that are hard to walk back (a commitment made externally, a change with security or compliance exposure). Trust for higher-stakes delegation gets built incrementally through the reversible tasks.

Compare time-to-teach against time-to-do. If explaining the task well would take meaningfully longer than doing it yourself, and it's a one-off with no repeat value, do it yourself. If it's a skill they'll use again, the teaching cost is an investment that pays back on the second and third time.

Define the support structure explicitly. Delegating isn't handing off and disappearing. Decide in advance: what checkpoints happen, what they can decide alone versus what needs a quick check-in, and what "stuck enough to escalate" looks like.

What you keep. Work that needs institutional context you can't transfer in the available time, early-relationship politically sensitive conversations, and anything where a mistake would damage a stakeholder's trust in the team broadly rather than just cost you some rework time.

Worked example

Say you're leading a project with three distinct pieces. One is well-scoped, reversible, and slightly above where this person has worked before: a strong candidate to delegate as a growth task, with a design check-in before they start building and a review before it ships. Another piece is customer-facing with real cost if it goes wrong: you either delegate it with heavy pairing so you catch problems before they land, or you keep it yourself this round and delegate the next similar piece once trust is established. The third is a one-off internal chore with no growth value: you delegate it purely for your own capacity, not as a coaching move, and you say so, because dressing up busywork as a growth opportunity erodes trust.

Trade-offs and pitfalls

Delegating only "safe" tasks because failure is expensive to you personally caps the person's growth. They never build judgment under real stakes if you only ever hand them things that can't go wrong.

Delegating and then vanishing looks like empowerment but is often abdication. The failure mode shows up late, when it's expensive to fix, because there was no checkpoint designed to catch it earlier.

Over-specifying the implementation defeats the purpose. If you hand someone a task but dictate every step, there's no room left for them to exercise judgment, which is the actual thing you're trying to develop.

The honest trade-off: delegating a stretch task usually costs you more short-term time, in reviewing and coaching, than doing it yourself would. That extra cost is the investment, and it's worth naming rather than pretending delegation is free.

Direct answer

A comprehensive Kubernetes cluster security strategy layers four kinds of control: what is allowed to run (admission control plus the current Pod Security Standards, which replaced the now-removed PodSecurityPolicy), what is allowed to run only if it is provably yours (image signing with sigstore/cosign), what is allowed to talk to what (NetworkPolicy segmentation), and who is allowed to do what (RBAC, role-based access control, least privilege, plus encrypted, rotated secrets). None substitutes for another; the design work is sequencing the rollout so each layer is validated in audit mode before it can reject a real deployment.

Structured elaboration

Admission control: OPA Gatekeeper plus Pod Security Standards

Use Gatekeeper for custom, org-specific rules (image provenance, required labels, naming conventions) and the built-in Pod Security Admission controller for the baseline hardening levels (privileged, baseline, restricted) defined by the Pod Security Standards. This split matters because PodSecurityPolicy, the older mechanism for this same job, was deprecated in Kubernetes 1.21 and fully removed in 1.25; a runbook or an older interview question that still describes writing a PodSecurityPolicy object is describing something that no longer exists in a supported cluster. The current mechanism is a namespace label:

kubectl label namespace payments pod-security.kubernetes.io/enforce=restricted

with audit and warn variants of the same label for a non-blocking rollout, mirroring Gatekeeper's own dry-run-then-enforce pattern.

Image signing and verification (sigstore/cosign)

CI signs each image at build time (cosign sign), ideally keyless and backed by the transparency log (Rekor) rather than a long-lived private key that itself has to be rotated and protected. Gatekeeper, or a purpose-built admission webhook, verifies the signature and, where available, checks provenance or SBOM (software bill of materials) attestations before allowing the image to run. Cache verification results at the admission layer; re-verifying every pull on every pod restart adds latency to something that should not change between restarts of the same image digest.

Network segmentation with NetworkPolicy

Default-deny per namespace, then explicit allow rules per service-to-service relationship:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: payments
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

This is enforced by the CNI (Container Network Interface) plugin, not the API server. A cluster on a CNI without NetworkPolicy support will accept this object but silently not enforce it, a common false sense of security worth checking explicitly: kubectl get networkpolicy succeeding proves the object exists, not that anything is actually blocking traffic.

RBAC hardening

Least privilege means a Role/RoleBinding scoped per namespace and per workload's own ServiceAccount, never a shared default ServiceAccount with broad permissions, and never a wildcard verb or resource in a Role used by application workloads. Periodic access reviews, who can actually reach cluster-admin, directly or through a chain of bindings, catch privilege creep that accumulates silently over time.

Secrets: encryption and rotation

Encrypt etcd at rest using the KMS (Key Management Service) encryption provider, specifically KMS v2, the recommended provider since it went stable in Kubernetes 1.29 (KMS v1 was deprecated in 1.28 and is disabled by default starting in 1.29). KMS v2 caches a data-encryption key locally after one call to the external KMS plugin, instead of calling out on every secret read, removing what used to be a real apiserver-latency cost of encryption at rest. Layer short-lived, automatically rotated credentials (a secrets manager issuing dynamic credentials, or a cloud provider's workload-identity federation mapping a ServiceAccount to a cloud role) on top, so long-lived static secrets are the exception rather than the default.

Multi-tenant blast-radius controls, the absorbed noisy-neighbor angle

At the Kubernetes object level, not the platform-availability layer, ResourceQuota and LimitRange are what actually stop one tenant's namespace from starving another's: ResourceQuota caps aggregate CPU, memory, and object counts per namespace, and LimitRange sets default and maximum per-container requests so a single unbounded pod cannot consume a quota's entire budget alone. This is deliberately narrow: quota and limit-range mechanics are this topic's ground; broader noisy-neighbor blast-radius design (autoscaling interaction, billing isolation) belongs to availability and disaster-recovery planning, not to a cluster security strategy.

Worked example (rollout sequencing, not a fabricated metric)

A four-phase rollout, each phase gated on the previous one running clean in audit mode for a full deploy cycle:

1. Baseline: enable audit logging, run Gatekeeper in dryrun, apply pod-security.kubernetes.io/warn=restricted everywhere (visible, non-blocking).
2. Enforce cheap wins: require resource limits and non-privileged containers cluster-wide (few legitimate exceptions), enable KMS v2 encryption for new secrets.
3. Enforce identity-sensitive controls: require image signature verification in production namespaces, flip Pod Security to enforce=restricted namespace by namespace as owners confirm compatibility.
4. Enforce network segmentation last, since it carries the highest blast radius for a mistake: roll out default-deny NetworkPolicies namespace by namespace, watching connection-refused rates as the signal to fix an allow-rule gap before moving to the next namespace.

Trade-offs and pitfalls

• PodSecurityPolicy no longer exists in any supported Kubernetes version; if an older reference or absorbed legacy question still frames this control that way, translate it to the Pod Security Standards labels above rather than reproducing a removed API object.
• Signature verification and default-deny NetworkPolicy are the two highest-friction controls for developers; sequence them last, after cheaper wins have built trust in the rollout process.
• KMS v1 is disabled by default in current Kubernetes; a cluster still relying on it is itself a finding, not a stable baseline to build on.