DoorDash Systems Administrator (Junior Level) - Interview Preparation Guide

Immediate goals: preserve evidence, gather diagnostic data, minimize downtime (failover/rollback), avoid further writes to damaged system.

Initial triage (both OSes)

• Verify reachability and alerts, note time of crash and affected services.
• Put host into maintenance mode (disable monitoring/automated remediation) and if possible shift traffic to standby.

If system still reachable (Linux)

• Capture uptime/last boot and kernel messages:

who -b
journalctl -k -b -1    # logs from previous boot
dmesg -T | tail -n 200

• Preserve crash dump / vmcore:

# check kdump
systemctl status kdump
ls -l /var/crash /var/lib/kdump
# copy vmcore off-box
scp /var/crash/<vmcore> analyst@collector:/data/

• Trigger safe sysrq (if hung and acceptable):

echo s > /proc/sysrq-trigger   # sync
echo u > /proc/sysrq-trigger   # remount ro
echo b > /proc/sysrq-trigger   # reboot (only if necessary)

• Collect config and state:

tar czf /tmp/sysinfo.tgz /etc /var/log
ss -tunap > /tmp/conns.txt
ps aux > /tmp/ps.txt
scp /tmp/sysinfo.tgz analyst@collector:/data/

• If memory dump missing, use makedumpfile/crash utilities to extract vmcore.

If system still reachable (Windows BSOD remote collection)

• Retrieve minidump and MEMORY.DMP:
  • Copy from \host\C$\Windows\Minidump and C:\Windows\MEMORY.DMP to safe storage.
• Query Windows event logs remotely:

Get-WinEvent -ComputerName host -FilterHashtable @{LogName='System';ID=1001} | Format-List
wevtutil qe System /q:"*[System[(EventID=1001)]]" /f:text > system_bsod.txt

• Check crash settings:

Get-WmiObject -Class Win32_ComputerSystem -ComputerName host

• If machine down, collect SAN snapshots or disk images rather than booting to avoid altering evidence.

Forensics & analysis

• Preserve timestamps and chain-of-custody (who copied which files and when).
• Analyze vmcore/minidump with crash, makedumpfile, WinDbg:
  • Linux: crash /usr/lib/debug/lib/modules/... /path/to/vmcore
  • Windows: WinDbg -z MEMORY.DMP !analyze -v

Restore service safely

• If quick rollback possible, failover to replica or boot last known-good image.
• If repair required, boot into rescue mode, apply kernel rollback or driver updates, test in staging, then bring back gradually.
• Postmortem: root cause, mitigations (kdump enabled, kernel/drivers pinned, monitoring alerts), and update runbooks.

Key priorities: copy crash dumps off-box, avoid writing to evidence disks, restore service via failover, and perform controlled analysis offline.

Situation & goal
With a constrained backup budget I prioritize business risk (RTO/RPO) by classifying data and applying tiered frequency, storage tiers, and retention to reduce cost while meeting recovery SLAs.

Policy by data class

• Transaction logs (DB WAL/SQL logs): Frequency: continuous or every 5–15 min. Storage tier: hot object storage (or nearline if compressible). Retention: 7–14 days on hot; archive older than 14 days to cold for 30–90 days. Trade-off: short-term frequent restores possible; longer retention on cold increases restore time.
• Databases (full backups): Frequency: daily full with differential/incremental every 4–12 hours. Tier: daily/full to standard block storage; older fulls to cold blob. Retention: 30 days for fulls, 6–12 months for monthly archives. Trade-off: incremental reduces storage and network; longer restores require roll-forward of logs.
• Large file shares (user files): Frequency: snapshots daily, incremental every 24 hours, weekly full. Tier: recent 30 days on hot; 30–365 days on cool/cold. Retention: 30–90 days for quick recovery, 1 year for compliance on cold. Trade-off: deduplication and snapshot chaining reduce cost but increase restore complexity.
• VM images: Frequency: weekly full, daily incremental (if supported). Tier: store on object storage with lifecycle to cold after 30 days. Retention: 3 months standard, 1 year archived. Trade-off: infrequent fulls save space; rebuilding VMs from older images takes longer.

Operational controls

• Use lifecycle policies to auto-transition tiers.
• Enable dedupe & compression; apply backup windows to off-peak times.
• Test restores for each class regularly and track cost vs SLA; adjust frequencies until risk tolerance met.

This balances cost by using frequency + incremental strategies for hot data and lifecycle transitions to cold for long-term retention, accepting longer RTO for lower-priority/archival data.

Direct answer

Progressive disclosure means showing the minimum someone needs to make their next decision first, then letting them opt into more detail only if they need it, rather than presenting every layer of a decision at once. For technical documentation, that means separating what we decided and why it matters from how it's actually implemented, and only showing the second layer to someone who asks for it.

Structured elaboration

Two concrete ways to build this into documentation:

1. A "decision, then detail" page structure. The top of the page states the decision and its business-relevant effect in one or two sentences. Directly below, an expandable or clearly linked section holds the reasoning (why this option over the alternatives, what constraint drove it), and a separate section holds the implementation (exact commands, config, code). A reader making a go/no-go call never has to scroll past architecture detail to find the decision.
2. Collapsed detail blocks inside a page that stays otherwise readable. Long code blocks, diagrams, or benchmark tables default to collapsed, with a label that tells the reader what's inside before they open it, not just "details." This keeps the page skimmable top to bottom for someone doing a first pass, while an engineer implementing the change can expand everything in order.
   Both work because they let the reader choose their own depth instead of the writer choosing it for them, and because the label on each layer, decision, why, how, tells the reader which layer they're in before they commit to reading it.

Worked example

A raw engineering note might read: "Switched to regional read replicas with async replication and connection pooling via PgBouncer to cut p95 read latency." Applied with progressive disclosure, the page becomes:
Top line (decision layer): "We added copies of the database closer to users in each region so read requests don't have to cross the country, which is what was making some pages feel slow for customers far from our main data center."
Expandable "why" layer: explains the latency problem was concentrated in specific regions and why a cache alone wasn't sufficient, still in plain language.
Expandable "implementation" layer, collapsed by default: regional read replicas (copies of the database kept near each user region), updated by async replication (the copy is written a short delay after the original, not instantly), and connection pooling via PgBouncer (a tool that reuses open database connections instead of opening a new one per request), plus config snippets and the failover procedure.
A reader deciding whether to approve the change never has to parse "PgBouncer" or "async replication" to get the decision; an engineer implementing it clicks straight through to exactly that.

Trade-offs and pitfalls

Progressive disclosure can misfire if the top layer is vague instead of just simple: "we improved performance" tells the reader nothing they can act on, while "reads are faster for users far from our main region" does. It also fails if the label on a collapsed section doesn't say what's inside; readers won't expand something called "details," so label it with what they'll actually get, for example "config and rollback steps." And it isn't free: every layer you maintain is another thing that can drift out of sync with the code, so it's worth it for docs people repeatedly return to, not a one-off internal note nobody will reread.

Direct answer

Use ss (or the older netstat) to see exactly what's listening, on which address and port, and confirm whether the bind is scoped to loopback only or to all interfaces, since that single distinction explains a large share of it's-listening-but-nothing-external-can-connect reports. Beyond the bind address and process ownership, a separate class of local blockers, the host firewall, SELinux, and network namespaces, can each independently prevent an otherwise healthy listener from being reached, and each needs its own specific check rather than being lumped in with the network.

Structured elaboration

1. List listening sockets with process ownership: ss -ltnp (listening, TCP, numeric, show process) lists every listening TCP socket along with the PID and process name holding it; this immediately answers whether anything is actually listening on this port, and whether it's the process you expect.
2. Read the local address field carefully: 0.0.0.0:8080 means the process is listening on all interfaces and is reachable from outside the host (firewall permitting); 127.0.0.1:8080 means it is bound only to loopback and is fundamentally unreachable from any other host, no matter what firewall rules say, because the OS never even considers external interfaces for that socket.
3. Check established connections and their state with ss -tn (without the -l) to see active connections; a large number stuck in SYN-RECV suggests the three-way handshake is not completing (possibly a firewall dropping the client's ACK, or a backlog queue issue on the server), while a large number in TIME_WAIT on a busy server is often benign churn rather than a problem, unless it is approaching ephemeral port exhaustion.
4. Check the host firewall explicitly, as its own distinct layer from anything upstream: on iptables-based hosts, iptables -L -n -v (or iptables -S) shows whether a rule is dropping or rejecting the port in question, and on firewalld-based hosts, firewall-cmd --list-all shows the active zone's allowed services and ports; a socket can be correctly listening on 0.0.0.0 and still be unreachable purely because the host's own firewall drops the inbound SYN before it ever reaches that socket.
5. Check SELinux (on systems that enforce it) as a separate, non-firewall local blocker: getenforce confirms whether SELinux is enforcing at all, and if it is, a service listening on a non-standard port that was never labeled for that port's SELinux port-type will be denied at the kernel security-module level even though the bind and firewall are both correct; ausearch -m avc -ts recent (or sealert on systems that have it) surfaces the specific denial, and semanage port -l | grep <port> shows what port-type is currently associated with that port, with semanage port -a -t <type> -p tcp <port> as the fix once the correct type is identified.
6. Check network namespaces on containerized or namespace-isolated hosts: a process can be listening perfectly well, but inside a different network namespace than the one you are inspecting from (for example inside a container's own namespace rather than the host's default namespace), so ss run in the wrong namespace will show nothing at all, not a loopback-only bind or a blocked port. ip netns list enumerates namespaces on the host, and ip netns exec <ns> ss -ltnp (or nsenter --net=<path> ss -ltnp) runs the same check inside the namespace that actually owns the socket; a 0.0.0.0 bind inside a container's namespace is still only reachable from outside according to whatever port-publishing or bridging rule (for example Docker's own iptables-based NAT rules) connects that namespace to the host and the network beyond it.
7. Distinguish not-listening-at-all from listening-but-blocked-further-out: if ss -ltnp shows nothing on the expected port in the correct namespace, the application itself never started or crashed, a fact entirely independent of firewalls, SELinux, or networking; if it does show a correct, externally-bound listener and external clients still cannot connect, the fault has moved to one of the local blockers above, or to something outside the host entirely.

Worked example

ss -ltnp shows LISTEN 0 128 127.0.0.1:8080 0.0.0.0:* users:(("myapp",pid=4521,fd=6)). The process is confirmed running and listening, but bound specifically to 127.0.0.1, loopback only; changing the bind address to 0.0.0.0 is the first fix, before any firewall or SELinux check is even relevant. Suppose instead ss -ltnp shows the same process correctly bound to 0.0.0.0:8080, and external clients still cannot connect. iptables -L -n -v on the host shows no DROP or REJECT rule referencing port 8080, ruling out the host firewall. getenforce reports Enforcing, and ausearch -m avc -ts recent shows an AVC denial for that process attempting to bind to port 8080, because the application was reconfigured to use a nonstandard port that was never added to SELinux's port-type list for that daemon; semanage port -a -t http_port_t -p tcp 8080 (or the appropriate type for the service) resolves it. In a third case, the process runs inside a container; ss -ltnp on the host shows nothing at all for port 8080, not because the process is not listening but because it is listening inside the container's own network namespace; docker exec <container> ss -ltnp (or ip netns exec <ns> ss -ltnp) confirms the process is listening correctly inside its namespace, and the actual question becomes whether the container's port-publishing rule correctly maps the host port to it.

Trade-offs & pitfalls

It is easy to see the process is listening in ss output and conclude the service is correctly exposed, without reading the bind address carefully enough to notice it is loopback-only, or without checking that you are even looking in the right network namespace. Also do not confuse ss -ltnp's absence of a listener with a firewall or SELinux problem; if nothing is listening in the namespace you are inspecting, no firewall rule or SELinux policy anywhere will make the connection succeed, and time spent checking those first is wasted until is-it-listening-and-where is ruled out. SELinux denials in particular are easy to miss because the application logs may show nothing useful at all (the bind or connection attempt is blocked below the application's own visibility), so always check ausearch or the audit log specifically once a correctly-bound, correctly-firewalled listener still is not reachable.

Direct answer
TIME_WAIT is the state the side that sent the FINAL ACK of a connection close sits in for a fixed period (commonly twice the maximum expected segment lifetime, often around 60 seconds on Linux) before the connection's resources are fully released. It exists so a delayed, duplicate packet from an old connection can't be mistaken for part of a brand-new connection reusing the same address/port pair. A half-open connection is one where only one side still believes the connection is alive; the other side has already reset, crashed, or otherwise abandoned it without a clean FIN exchange.

Structured elaboration
TIME_WAIT exists to protect two things: (1) it guarantees the final ACK the closing side sent actually gets through, by giving time to retransmit it if the peer's FIN gets retransmitted (meaning the ACK was lost); (2) it prevents a stray, delayed packet from a previous incarnation of a connection (same 4-tuple: source IP, source port, destination IP, destination port) from being delivered into a brand new connection that happens to reuse the same 4-tuple before the old segments have had time to disappear from the network.

To detect a large number of sockets stuck in TIME_WAIT on Linux, ss -tan state time-wait | wc -l (or the older netstat -ant | grep TIME_WAIT | wc -l) gives a live count; watching this metric over time distinguishes a normal, self-draining backlog from a genuine problem.

Worked example
A server that closes millions of short-lived outbound connections per hour (for instance, a service making one HTTP call per request to an upstream) can exhaust its available ephemeral source ports if TIME_WAIT sockets accumulate faster than they expire, because each TIME_WAIT socket still holds its port reserved. The two standard mitigations are: raise the number of available client-side (ephemeral) ports and/or reuse connections via keepalive/connection pooling so fewer connections churn through TIME_WAIT in the first place; and, on the SERVER side specifically, enabling SO_REUSEADDR and, where safe, tcp_tw_reuse lets a new outgoing connection reuse a TIME_WAIT 4-tuple once TCP timestamps confirm it's safe to do so, rather than waiting out the full timer.

Trade-offs & pitfalls
Disabling or drastically shortening TIME_WAIT globally (rather than tuning port ranges or reuse settings) is the wrong fix: it reintroduces the exact correctness problem TIME_WAIT was designed to prevent, stray old packets landing in a new connection and corrupting it. The safe levers are reducing HOW MANY connections churn through the state (pooling, keepalive) and widening the ephemeral port range, not shrinking the safety window itself.

Situation / Goal
I needed Help Desk to reset passwords and unlock accounts only for users in the Employees OU, with full auditability and least-privilege controls.

Delegation steps (practical)

• Open Active Directory Users and Computers, right-click Employees OU → Delegate Control.
• Add the Help Desk group.
• Use the wizard: choose “Create a custom task to delegate” → “Only the following objects in the folder” → select “User objects.”
• Grant permissions: “Reset user passwords and force password change at next logon.”
• If unlock requires explicit ACL change, use Advanced Security on the OU to grant Write lockoutTime (or use granular ACE via “Delegate Control” to add that right).

If using AD ACLs directly

• Open ADUC → View → Advanced Features → OU Properties → Security → Advanced → Add Help Desk group → select specific Extended Rights: “Reset Password” and allow write lockoutTime.

Auditing configuration

• Enable success auditing for “Account Management” in Group Policy (Computer Configuration → Policies → Windows Settings → Security Settings → Advanced Audit Policy Configuration → Account Management → Audit User Account Management: Success).
• Configure SACL on Employees OU (Advanced Security → Auditing) to audit Success for Reset Password and Write lockoutTime.
• Forward logs to SIEM; alert on Event IDs: 4724 (password reset attempt), 4767 (account unlocked), and correlate with ticket IDs.

Procedural controls for least privilege & accountability

• Require ticket with unique ID and approval before Help Desk action; log ticket ID in AD operation notes.
• Enforce MFA for Help Desk consoles and privileged sessions.
• Monthly access review (recertification) of the Help Desk group memberships.
• Implement Just-In-Time elevation (Privileged Access Management) for sensitive cases.
• Retain audit logs 1 year; run weekly automated reports of resets/unlocks and investigate anomalies.

Result: Help Desk can perform necessary support without expanded rights, all actions are auditable and controlled, preserving least privilege and accountability.

Direct answer: Redundancy is having extra, interchangeable components standing by so one can take over if another fails; replication is actively keeping copies of state (data) synchronized across multiple nodes so the state itself survives a failure, not just the compute that serves it. They're often used together, but they solve different problems: redundancy alone is enough for a stateless service, because any interchangeable instance can serve any request; a stateful service needs replication too, because a fresh redundant instance with no data isn't actually a working replacement.

Structured elaboration

Stateless example: a set of identical app server instances behind a load balancer, handling API requests with no local state. If one instance dies, the load balancer routes around it and an autoscaler replaces it; the new instance needs no data transfer because there was never any instance-local state to lose. This is pure redundancy: extra interchangeable copies of the same stateless computation.

Stateful example: a primary-replica database. The primary accepts writes; replicas continuously receive a copy of the write stream (replication). If the primary fails, a replica is promoted to take over. Unlike the stateless case, simply having an extra database instance running (redundancy alone, no replication) would give you an empty database, not a working replacement, because there's no mechanism copying the actual data into it.

A failure mode redundancy alone doesn't protect against, for the stateful case: data loss or corruption on the primary itself. If the primary's disk corrupts a row, or a bad write silently corrupts application-level data, having a redundant (but not yet caught-up, or synchronously replicating that same bad write) standby doesn't help, because either the standby doesn't have the data yet (async lag) or it faithfully replicated the corruption along with everything else (synchronous replication of a logically bad write). Redundancy protects against a node dying; it does not protect against the data itself being wrong, that requires backups (a separate, point-in-time copy decoupled from live replication) and, for silent corruption specifically, checksums or application-level validation.

How this generalizes: a useful mental checklist for fault tolerance covers five distinct techniques, and redundancy and replication are only two of them: retries (recover from a transient failure by trying again), bulkheads (isolate one failure from spreading to unrelated resources), failover (the mechanism that switches traffic to a healthy replacement), redundancy (having that replacement exist at all), and replication (making sure the replacement actually has the state it needs). A strong answer names which of these a given design decision is actually addressing, since "redundancy" gets used loosely to mean all five in casual conversation.

Trade-offs & pitfalls

• Replication has a cost redundancy alone doesn't: network bandwidth, storage for extra copies, and a consistency model to reason about (synchronous replication costs write latency; asynchronous replication risks data loss on failover, the classic RPO trade-off).
• A common wrong turn: assuming "we have 3 replicas" automatically means "we're protected," without checking replication lag. A replica that's minutes behind at failover time silently loses however much data arrived in that window, unless the promotion logic explicitly accounts for lag and refuses to promote a too-far-behind replica.
• Redundancy for stateless services is comparatively cheap and low-risk to over-provision; replication for stateful services is not, since more replicas means more write-path coordination overhead (for synchronous replication) or more divergence risk (for asynchronous/multi-leader), so it isn't a "just add more" lever in the same way.

Direct answer

That specific message means the SSH daemon's (sshd's) strict permission checks rejected bob's home directory or .ssh directory before it ever evaluated the key itself. By default, sshd refuses to trust a public key if the account's home directory, its .ssh directory, or the key file is writable by anyone other than its owner, because a group- or world-writable path there would let another local user plant their own key and log in as bob. The fix is tightening ownership and permissions on the exact chain of directories sshd checks, then verifying the fix actually took effect.

Structured elaboration

Root causes. Any of these produce the same "bad ownership or modes" message:

1. /home/bob is not owned by bob (or root), often from files restored by a backup tool or copied in by another administrator's account.
2. /home/bob or /home/bob/.ssh has a group- or world-writable bit set (sshd's StrictModes setting, on by default, rejects this).
3. /home/bob/.ssh/authorized_keys itself is writable by group or other, even if the directories above it are fine.

Commands to fix it, applied in the order that matches how sshd walks the path:

# 1. Home directory must be owned by bob, and not writable by group or other.
sudo chown bob:bob /home/bob
sudo chmod go-w /home/bob

# 2. .ssh directory: owner-only access.
sudo mkdir -p /home/bob/.ssh
sudo chown bob:bob /home/bob/.ssh
sudo chmod 700 /home/bob/.ssh

# 3. authorized_keys file: owner-only read/write, no execute needed.
sudo chown bob:bob /home/bob/.ssh/authorized_keys
sudo chmod 600 /home/bob/.ssh/authorized_keys

Verifying the fix. Confirm the actual mode bits, not just that the commands ran without error:

stat -c "%a %U:%G %n" /home/bob /home/bob/.ssh /home/bob/.ssh/authorized_keys

Expect something like 755 bob:bob /home/bob, 700 bob:bob /home/bob/.ssh, and 600 bob:bob /home/bob/.ssh/authorized_keys. Then attempt the actual login with verbose logging so sshd's own reasoning is visible: ssh -vvv bob@host, and separately watch the server side with sudo tail -f /var/log/auth.log (or /var/log/secure on Red Hat-family distributions) for the corresponding "Accepted publickey" line, rather than assuming success from the client side alone.

If permissions look correct and it still fails, check SELinux (Security-Enhanced Linux, a mandatory access control system available on many Linux distributions). Standard Unix permission bits and SELinux are two independent checks; a directory can have perfectly correct ownership and mode and still be denied if its SELinux file context (label) is wrong, most commonly after files were restored from a backup or copied from another location without preserving context:

ls -Z /home/bob/.ssh /home/bob/.ssh/authorized_keys

.ssh and its contents should carry the ssh_home_t context. If it shows something else (commonly a generic context like user_home_t after a restore), restore the correct label rather than trying to set it by hand:

sudo restorecon -Rv /home/bob/.ssh

If that does not resolve it, check for an actual denial being logged rather than guessing further:

sudo ausearch -m avc -ts recent

An access vector cache (AVC) denial entry naming sshd and the .ssh path confirms SELinux, not a stale permission bit, is the actual blocker.

Worked example

A concrete before-and-after for this exact scenario. Suppose /home/bob/.ssh was left group-writable (mode 775) after a provisioning script ran as root and did not lock it down, a common way this exact error gets introduced:

Before: drwxrwxr-x  bob bob  /home/bob/.ssh   (group-writable, sshd rejects this)
        -rw-r--r--  bob bob  /home/bob/.ssh/authorized_keys

After applying the fix commands above:
        drwx------  bob bob  /home/bob/.ssh
        -rw-------  bob bob  /home/bob/.ssh/authorized_keys

Running the fix commands against exactly this before-state (verified directly: chmod go-w on the home directory, chmod 700 on .ssh, chmod 600 on authorized_keys) produces exactly the after-state shown, the group-write bit is gone from .ssh and authorized_keys no longer has any bits set beyond owner read/write. That is the state sshd's strict-mode check requires before it will even look at the key's contents.

Trade-offs and pitfalls

• Fixing only authorized_keys and skipping the parent directories is a common half-fix. sshd checks the whole path, home directory, .ssh, and the key file itself, so a perfectly permissioned key file inside a group-writable .ssh directory still fails.
• Assuming a fix worked because the commands returned no error is a common mistake. Always verify the actual mode bits with stat, and confirm the login itself succeeds with verbose logging, rather than trusting that chmod silently doing nothing wrong means the state is now correct.
• Jumping straight to SELinux before confirming ownership and mode bits wastes time. SELinux denials produce a similar practical symptom (login refused) but a different log signature; check the simpler, more common cause first, and only chase an SELinux label mismatch once the standard permission chain is confirmed correct.
• Manually setting an SELinux context instead of using restorecon risks getting the label subtly wrong. restorecon restores the context the system's own policy defines for that path, which is more reliable than guessing a context string by hand.

Approach

• Create a user "deploy" with sudo and an SSH public key.
• Ensure apt lists are updated and required packages installed.
• Start and enable nginx; install a monitoring agent (example: datadog-agent placeholder).
• Keep the snippet minimal and ready to paste as cloud-init user-data.

cloud-init user-data (YAML)

#cloud-config
users:
  - name: deploy
    gecos: Deploy User
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - "ssh-rsa AAAA...your-public-key... user@example.com"

package_update: true
packages:
  - nginx
  - curl
  - apt-transport-https

runcmd:
  - [ sh, -c, "systemctl enable --now nginx" ]
  - [ sh, -c, "echo 'Installing monitoring agent (example)'; curl -sS https://example.com/install-agent.sh | bash" ]

Notes:

• Replace the SSH key and monitoring-agent install command/URL with your actual public key and vendor instructions (e.g., Datadog, Prometheus node-exporter).
• For production, pin package versions or add apt repositories and GPG keys securely.

Approach (brief)
I’d collect node/cgroup/container, kubelet and app-level metrics, and translate them into Prometheus alerts and dashboards to detect CPU throttling, OOMKills, node pressure, and slow startup.

Metrics & Sources

• CPU throttling
  • metric: container_cpu_cfs_throttled_seconds_total, container_cpu_cfs_throttled_periods_total
  • source: cAdvisor / kubelet metrics exposed on kubelet/cadvisor endpoints
  • alert: high increase in throttled_seconds or throttled_periods per container over 5m
• OOMKills
  • metric: kube_pod_container_status_terminated_reason{reason="OOMKilled"} (kube-state-metrics) and node OOM events from kubelet (container_last_seen/eviction counters)
  • alert: any new OOMKilled termination or >0 OOMs in 1m for a workload
• Node pressure (CPU/memory/disk)
  • metrics: node_exporter / kubelet: node_memory_MemAvailable_bytes, node_filesystem_avail_bytes, node_load1, node_cpu_seconds_total (or containerized equivalents); kube_node_status_condition{condition="MemoryPressure"/"DiskPressure"/"PIDPressure"} (kubelet via node exporter/kube-state-metrics)
  • alert: MemoryPressure/DiskPressure true; available disk < 10% or load >> cores
• Slow application startup
  • metrics: application readiness probe latency or custom startup_duration_seconds histogram (application) and kube_pod_start_time_seconds (kube-state-metrics)
  • alert: pod startup_duration > expected SLA (e.g., >60s) or readiness not true after X seconds

Example Prometheus alerts

- alert: ContainerCPUThrottlingHigh
  expr: rate(container_cpu_cfs_throttled_periods_total[5m]) > 0.1
  for: 5m
  labels: {severity: warning}
  annotations: {summary: "High CPU throttling for {{ $labels.container }}"}

- alert: PodOOMKilled
  expr: increase(kube_pod_container_status_terminated_reason{reason="OOMKilled"}[5m]) > 0
  for: 1m
  labels: {severity: critical}

How to act / runbooks

• Throttling: increase cpu limits/requests, move to node with spare capacity, tune QoS or use burstable QoS
• OOMKills: increase memory requests/limits, fix memory leaks, adjust JVM flags, add liveness probes
• Node pressure: cordon and drain unhealthy nodes, add capacity, clean up disk usage
• Slow startup: optimize init logic, lazy-load, warm caches, adjust readinessProbe/startupProbe

Collect logs (kubelet logs, container stderr) and correlate alerts with Grafana dashboards showing time series and top offenders.