Observability and Monitoring Architecture Questions

Direct answer

Muting is not a fix because it just trades false pages for false confidence. The actual fix is measurement first: classify every page from the last few weeks by whether it required real action, then attack the noisy checks at the source, thresholds, dependencies, flapping detection, rather than the notification layer, and only after that recalibrate what is actually allowed to page a human at all.

Structured elaboration

A concrete triage sequence:

1. Pull the last 2 to 4 weeks of notification history and classify each page: actioned (someone did something), acknowledged but noise (looked at it, nothing to do), or auto-resolved before anyone responded.
2. For the worst offenders, fix the actual cause: enable and tune flap detection for checks oscillating up and down, add multi-attempt confirmation for anything paging on a single sample, and add host and service dependencies so one switch or one host outage does not fan out into dozens of individual pages.
3. Reclassify severity: split checks into "pages a human now," "opens a ticket," and "dashboard only, no notification." Most of a legacy 3,000-check Nagios install has accumulated checks in the first bucket that never belonged there.
4. Re-baseline and repeat the measurement after each round of changes, because fixing the top offenders will surface the next tier that was previously masked by the noise.

Worked example

This kind of classification pass on a legacy install tends to be sharply skewed. A representative breakdown for a 3,000-check, 40-page-a-night estate: if 12 checks (0.4% of the 3,000) turn out to account for 31 of the 40 nightly pages, fixing just those 12, through flap detection, multi-attempt confirmation, or dependency suppression, drops nightly pages from 40 to roughly 9 (40 minus 31) without touching the other 2,988 checks at all. That is the case for measuring before changing anything: a handful of checks usually explain most of the pain.

Trade-offs and pitfalls

• This takes real calendar time, weeks, not a single sprint, if done properly. The temptation to just silence the loudest checks immediately is understandable under pager fatigue, but it directly causes the next real incident to be missed because on-call has learned to distrust pages.
• Adding dependencies and flap detection can itself hide a real problem if configured too aggressively, for example suppressing all child alerts when a parent host looks flaky, right when you actually needed to know a specific child service also failed independently.
• Political friction is common: some noisy checks exist because someone once got blamed for missing something, so removing or downgrading them needs a clear "here is the data showing this has not been actionable in a month" case, not just a unilateral change.

What the interviewer probes next

They will usually press on the politics of this, since downgrading a check someone else configured needs real evidence behind it, and on how you would tell whether the fatigue is genuinely fixed rather than the pager just being quiet this one week.

Direct answer

Collect a time series of used capacity per volume, for example daily df samples or whatever your monitoring system already stores, fit a trend line to it, and project forward to when it crosses your action threshold. You need enough history to smooth out noise (weekly patterns, one-off cleanups) but recent enough to reflect current growth, and the output should be a date, not just a percentage, because a date is what triggers a purchase order.

Structured elaboration

Collect at least daily df-style samples of used GB per volume, not just percent, since percent alone hides how many GB a jump actually represents on a large disk. Fit a trend line (ordinary least squares is enough for this) to get a growth rate in GB per day, then project forward to find the day the fit crosses your capacity ceiling.

What triggers the purchase order: pick a lead time longer than your procurement cycle. If buying and provisioning new storage takes 3 weeks, trigger the order when the trend crosses "90% full in 5 weeks," not when the disk is already at 90%.

Worked example

Executed example using 14 days of sampled disk usage on a 500 GB volume, fitting a least-squares line and projecting forward:

measured samples (GB used): [350.5, 350.4, 355.1, 357.4, 355.1, 358.0, 362.2,
                              363.5, 366.0, 366.7, 371.3, 373.2, 374.1, 377.7]
fitted growth rate: 2.105 GB/day (true underlying rate in this simulation was 2.0 GB/day)
fitted intercept (day 0 est.): 349.3 GB
90% full (450 GB) projected at day 47.9 from day 0
days remaining from today (day 13) to the 90% mark: 34.9 days

The fit (ordinary least squares) recovers the true 2.0 GB/day growth rate closely, 2.105 estimated, even with daily measurement noise. That is the point: a single day's jump or dip should not be read as a trend change, the line across many days is what you act on.

Trade-offs and pitfalls

• A straight-line fit assumes linear growth; a service about to onboard a large new customer or double its retention window will blow past a linear projection, so pair the trend with awareness of planned changes, not just historical data.
• Too little history (a few days) makes the slope noisy and unreliable; too much history (a year) can hide a recent acceleration by averaging it away. Recompute on a rolling window, for example the trailing 30 days, and re-evaluate weekly.
• An aggregate trend across 200 servers hides the one server about to fill up next week; you want both a fleet-wide summary for planning and a per-host projection for the "who pages tonight" question.

What the interviewer probes next

They will usually push on whether the fit is really linear or whether a single unusual day, a big import, a bulk cleanup, is quietly steering it, and on how you would roll a whole fleet of these projections into something a non-technical stakeholder can act on without reading a chart.

Direct answer

Split retention into hot (raw, fast storage, days), warm (compressed, still full resolution, weeks), cold (downsampled and heavily sampled, up to a year), and archive (compliance-only raw extracts, years, cheapest storage). Alerting stays correct because every alert either reads from the hot/warm tiers where it needs full resolution, or is redesigned to work on the statistical properties (percentiles, error counts) that downsampling preserves. Compliance stays correct because the raw bytes a regulator might ask for are pulled into a separate, signed archive path that bypasses the operational downsampling entirely.

Tiering policy

Downsampling rule for metrics: never discard raw points, only aggregate. Roll up 10s scrapes into 1-minute buckets storing count, sum, min, max, and pre-computed percentile buckets (histogram, not just an average), so alerting on p95/p99 still works post-downsample. Averaging away the histogram is the single most common mistake here: an average of averages silently breaks p99-based alerts.

Downsampling rule for logs: keep 100% of error/warn-level events forever within the cold window; sample info/debug logs at a low, deterministic rate (hash on trace ID) so a given request's logs are either fully kept or fully dropped, never split. That preserves the ability to reconstruct a whole request's log trail if it was selected.

Keeping alerting correct across tiers

• Alert rules that fire on absolute thresholds (error count, latency p99) are pinned to run against hot/warm data only, since that is where fidelity is guaranteed.
• Longer-window alerts (weekly seasonality, slow burn SLO budget) run against cold-tier rollups, but only because the rollup step preserved histograms rather than plain averages, so the alert's percentile math still holds.
• Backtest every downsampling change: replay the last 90 days of raw data through the proposed rollup and confirm the alert would have fired at the same times it fired against raw data, before shipping the rollup config.

Keeping compliance correct across tiers

• Compliance retention is a separate write path, not a side effect of the operational tiers. At ingest, a policy tags records matching compliance scope (PII, regulated data classes, specific tenants); those records get a signed, immutable copy written to the archive tier at full fidelity, independent of what the operational hot/warm/cold pipeline does to the rest of the stream.
• The operational tiers are free to downsample or expire on their own schedule because the compliance obligation is already satisfied by the separate archive copy.
• Archive retrieval is on-demand (async restore), so day-to-day cost stays low, but an auditor's request for raw data from 3 years ago is answerable.

Worked example

Assume a service ingests $G = 500$ GB/day of combined logs and metrics, split 30% metrics / 70% logs, and the storage budget is a hot-tier unit price of $X = \$23\text{/TB-month}$ (a concrete stand-in for the given $X). Illustrative relative unit prices for the other tiers, typical of SSD-backed vs. compressed-object vs. archival storage:

Tier volumes (hot: 7 days raw; warm: next 23 days at 3:1 compression; cold: next 335 days, metrics down 10:1 and logs sampled to 1%; archive: a 2%-of-ingest compliance subset, retained 7 years = 2555 days, compressed 4:1):

Monthly cost:

Compare that to the naive policy of keeping everything raw, at hot-tier price, for the full 7-year compliance window:

Tiering plus targeted downsampling cuts the steady-state storage bill by roughly 99.5% relative to keeping everything at hot-tier fidelity for the retention window, while still satisfying a 7-year compliance obligation and keeping alerting on full-fidelity data for the first month.

flowchart LR
  Ingest[Raw Ingest] --> Hot[Hot 0-7d full res]
  Hot --> Warm[Warm 8-30d compressed 3x]
  Warm --> Cold[Cold 31-365d downsampled 10x, logs 1%]
  Cold --> Archive[Archive 365d+ compliance subset, 4x compressed]
  Hot --> AlertEval[Alert Evaluation]
  Warm --> AlertEval
  Cold --> AlertEval
  Archive --> Restore[On-demand Restore]
  Restore --> AlertEval
  Ingest --> Vault[Signed Compliance Vault]
  Vault --> Archive

Trade-offs and pitfalls

• Downsampling metrics to plain averages instead of histograms is the classic mistake: it looks fine until a p99-based SLO alert goes silent because the percentile can no longer be reconstructed from the rollup.
• Deterministic (hash-based) log sampling beats random sampling because it keeps a request's full log trail together instead of showing half a trace.
• Treating compliance retention as "whatever the operational tiers happen to still have" instead of a separate write path is a common design flaw: an operational cost-cutting change (shortening the cold window) can silently create a compliance gap unless the archive copy is decoupled from it.
• Archive restore latency (minutes to hours, depending on backend) is a real trade-off against cost; if legal holds require faster turnaround, that changes the archive tier choice and its price multiplier.
• Budget monitoring has to be proactive: alert on forecasted spend crossing the budget before the bill arrives, not after, since tier transitions are usually async and lag actual ingest growth.

Direct answer

For a 500 host hybrid estate with bare-metal Windows and Linux, network gear, and a handful of cloud VMs, the real decision driver is agent model and protocol reach, not feature checklists. Nagios and Zabbix both natively speak SNMP for network gear and have mature agents for Windows, while Prometheus with node_exporter is pull-based, Linux-native, and has weak native support for Windows and none for SNMP without extra exporters. That single fact usually decides it for a genuinely mixed on-prem estate.

Structured elaboration

• Nagios: agent-based (NRPE or NSClient++) or agentless (SSH, SNMP) checks, a huge plugin ecosystem built over decades, but configuration is mostly flat text files, which gets unwieldy past a few hundred hosts without a config-generation layer on top such as Puppet or Ansible templates.
• Zabbix: has its own lightweight native agent for both Windows and Linux, first-class SNMP polling built in, SNMP trap intake via its bundled trap-receiver script paired with Net-SNMP's snmptrapd, a real web UI with database-backed config that scales more gracefully to hundreds of hosts than flat Nagios files, and built-in trend storage so you do not need a separate time-series database.
• Prometheus with node_exporter: node_exporter only covers Linux/Unix host metrics well; Windows needs a separate windows_exporter, and network gear needs snmp_exporter as a bolt-on translator, since Prometheus itself only speaks its pull-based HTTP scrape protocol, not SNMP. It shines for containerized and cloud-native workloads with dynamic service discovery, a poor fit for static bare-metal inventory that barely changes month to month.

Given this estate, mostly static Windows/Linux bare metal plus network gear plus a few cloud VMs, Zabbix is usually the pragmatic default: one agent model covers both operating systems, SNMP is native rather than bolted on, and the host count is static enough that Prometheus's dynamic discovery strength is not worth much here. If the cloud VM footprint were the majority instead of a handful, that calculus flips toward Prometheus.

Worked example

Put concrete numbers on the 500 hosts: say 380 Linux bare metal, 80 Windows bare metal, 30 network switches and UPS units, and 10 cloud VMs (380 + 80 + 30 + 10 = 500). Zabbix's single native agent covers all 460 Windows and Linux hosts, its native SNMP polling covers the 30 network devices, and the same agent covers the 10 cloud VMs too, one tool for all 500. Prometheus with node_exporter alone only natively reaches the 380 Linux hosts; the 80 Windows hosts need windows_exporter, and the 30 network devices need snmp_exporter. That is still three separate exporter types, node_exporter, windows_exporter, and snmp_exporter, deployed across the estate just to match the coverage Zabbix gets from one agent.

Trade-offs and pitfalls

• Nagios's flat-file config becomes a real operational burden past a few hundred hosts without a templating layer; teams often underestimate this until they are maintaining thousands of object definitions by hand.
• Zabbix's backing database (usually MySQL or PostgreSQL) becomes something you now have to keep highly available and capacity-plan for, a new operational dependency you did not have with flat-file Nagios.
• Running Prometheus here means running and maintaining three separate exporters, node, windows, and snmp, plus Prometheus itself plus Alertmanager, more moving parts than a single Zabbix or Nagios install for the same coverage.
• These are not mutually exclusive in practice; plenty of shops run Zabbix for the bare-metal estate and Prometheus for the Kubernetes workloads side by side, which is often the honest answer rather than picking one tool for everything.

What the interviewer probes next

They will usually dig into whether the cloud VM slice should just use the same agent as everything else or be treated as a separate Prometheus-scraped island, and they will want a concrete cutover sequence rather than just a target-state recommendation.

Direct answer

Load average and CPU percentage measure different things: load average also counts processes waiting on I/O in the run queue, not just processes waiting for CPU. On a 4 core box, a load of 8 means, on average, two processes are queued per core, a real overload signal, but since CPU sits at only 20% they are not queued waiting for CPU time. That combination almost always means something is I/O bound (disk, network, or a lock), not CPU bound. The fix is to find which processes are stuck in that waiting state and what they are waiting on.

Structured elaboration

Walk the toolchain in order:

• uptime: confirms the load trend (1/5/15 min averages) is real and not a one-off spike.
• vmstat 1 5: check the wa (I/O wait) column. High wa with low us/sy confirms the CPU is idle waiting on I/O, not busy computing.
• iostat -x 1 5: look at %util and await per device to find which disk is saturated.
• ps aux filtered on process state, to find processes in uninterruptible sleep (state D), the OS-level signature of a process blocked on I/O that cannot even be killed until the I/O completes.

Worked example

Ran the exact filter against a sample process table to demonstrate the technique, not a live host:

== processes stuck in uninterruptible sleep (D state) ==
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mysql     1122 12.4  8.2 812340 84200 ?        Dl   09:14   4:41 /usr/sbin/mysqld
backup    2200  0.4  0.2  15200  3100 ?        D    09:41   0:00 tar czf /backups/db.tar.gz /data
web        980  3.1  1.5  95200 22100 ?        S    Jul24   1:02 nginx: worker process

Command: ps aux | awk 'NR==1 || $8 ~ /^D/' (in a real ps aux listing, STAT is field 8, right after TTY). Run against the table above it drops the healthy nginx worker (state S) and keeps only the two processes actually blocked on I/O: a MySQL process and a backup tar job, both in state D/Dl. That narrows an 8 load average down to two concrete suspects and explains high load with idle-looking CPU.

Trade-offs and pitfalls

• top's default view hides wa; you have to check the CPU detail line or use vmstat/mpstat to see it.
• A high load average from many short-lived processes (a fork bomb, a cron storm) can look similar to an I/O problem at a glance; vmstat's r (runnable) column versus b (blocked) column tells them apart.
• Network-attached storage (NFS, iSCSI) can cause D state processes while iostat shows the local disk as idle. Check nfsstat or network latency too.

What the interviewer probes next

They will usually push on what you would do once you have identified the blocked process (kill it, or is it un-killable and you need to fix the underlying storage), and whether you would have caught this earlier with a proactive alert instead of a manual investigation.