Observability and Monitoring Architecture Questions
Building visibility into infrastructure and services: metrics, logs, and traces, dashboards and alerting, SLIs/SLOs, and the design of an observability stack. Covers instrumenting systems for actionable signal, reducing alert noise, and diagnosing production issues from telemetry. Infrastructure-wide observability, distinct from network-specific monitoring.
Compare Nagios, Zabbix, and Prometheus with node exporter as the monitoring stack for a 500 host estate that mixes bare metal Windows and Linux servers, network gear, and a few cloud VMs. What would push you toward one over the others?
Sample Answer
Direct answer
For a 500 host hybrid estate with bare-metal Windows and Linux, network gear, and a handful of cloud VMs, the real decision driver is agent model and protocol reach, not feature checklists. Nagios and Zabbix both natively speak SNMP for network gear and have mature agents for Windows, while Prometheus with node_exporter is pull-based, Linux-native, and has weak native support for Windows and none for SNMP without extra exporters. That single fact usually decides it for a genuinely mixed on-prem estate.
Structured elaboration
- Nagios: agent-based (NRPE or NSClient++) or agentless (SSH, SNMP) checks, a huge plugin ecosystem built over decades, but configuration is mostly flat text files, which gets unwieldy past a few hundred hosts without a config-generation layer on top such as Puppet or Ansible templates.
- Zabbix: has its own lightweight native agent for both Windows and Linux, first-class SNMP polling built in, SNMP trap intake via its bundled trap-receiver script paired with Net-SNMP's
snmptrapd, a real web UI with database-backed config that scales more gracefully to hundreds of hosts than flat Nagios files, and built-in trend storage so you do not need a separate time-series database. - Prometheus with node_exporter: node_exporter only covers Linux/Unix host metrics well; Windows needs a separate
windows_exporter, and network gear needssnmp_exporteras a bolt-on translator, since Prometheus itself only speaks its pull-based HTTP scrape protocol, not SNMP. It shines for containerized and cloud-native workloads with dynamic service discovery, a poor fit for static bare-metal inventory that barely changes month to month.
Given this estate, mostly static Windows/Linux bare metal plus network gear plus a few cloud VMs, Zabbix is usually the pragmatic default: one agent model covers both operating systems, SNMP is native rather than bolted on, and the host count is static enough that Prometheus's dynamic discovery strength is not worth much here. If the cloud VM footprint were the majority instead of a handful, that calculus flips toward Prometheus.
Worked example
Put concrete numbers on the 500 hosts: say 380 Linux bare metal, 80 Windows bare metal, 30 network switches and UPS units, and 10 cloud VMs (380 + 80 + 30 + 10 = 500). Zabbix's single native agent covers all 460 Windows and Linux hosts, its native SNMP polling covers the 30 network devices, and the same agent covers the 10 cloud VMs too, one tool for all 500. Prometheus with node_exporter alone only natively reaches the 380 Linux hosts; the 80 Windows hosts need windows_exporter, and the 30 network devices need snmp_exporter. That is still three separate exporter types, node_exporter, windows_exporter, and snmp_exporter, deployed across the estate just to match the coverage Zabbix gets from one agent.
Trade-offs and pitfalls
- Nagios's flat-file config becomes a real operational burden past a few hundred hosts without a templating layer; teams often underestimate this until they are maintaining thousands of object definitions by hand.
- Zabbix's backing database (usually MySQL or PostgreSQL) becomes something you now have to keep highly available and capacity-plan for, a new operational dependency you did not have with flat-file Nagios.
- Running Prometheus here means running and maintaining three separate exporters, node, windows, and snmp, plus Prometheus itself plus Alertmanager, more moving parts than a single Zabbix or Nagios install for the same coverage.
- These are not mutually exclusive in practice; plenty of shops run Zabbix for the bare-metal estate and Prometheus for the Kubernetes workloads side by side, which is often the honest answer rather than picking one tool for everything.
What the interviewer probes next
They will usually dig into whether the cloud VM slice should just use the same agent as everything else or be treated as a separate Prometheus-scraped island, and they will want a concrete cutover sequence rather than just a target-state recommendation.
You need to collect telemetry from a hybrid environment where some hosts sit behind corporate firewalls with no inbound access. Evaluate push, pull, and proxy/relay collection designs and recommend a secure architecture, including service discovery and how you'd authenticate agents (for example mutual TLS) across NAT and firewall boundaries.
Sample Answer
Direct answer
Pull doesn't work here: the collector can't open an inbound connection through a corporate firewall with no inbound access, so this comes down to push versus a local relay. Use agent-initiated push, authenticated with mutual TLS, and add a local relay inside networks where many hosts share one restrictive egress path, so the relay makes the one outbound connection to the central collector and the individual hosts only ever talk to something already inside their own network.
Structured elaboration
| Model | Works behind inbound-blocked firewalls? | Where auth/complexity lives | Best for |
|---|---|---|---|
| Pull (collector initiates) | No, needs inbound access or a VPN hole punched | Collector-side | Networks you fully control, no NAT |
| Push (agent initiates) | Yes, only needs outbound | Agent-side, per-host credentials | Small to medium fleets, direct internet egress allowed |
| Proxy/relay (agent pushes to local relay, relay pushes out) | Yes, and centralizes the one outbound connection | Relay-side | Large fleets behind one restrictive egress point |
Recommended architecture: agents push. Where a corporate network has many hosts behind one firewall, deploy a relay inside that network; agents push to the relay over the local network, and the relay makes the single outbound mTLS connection to the central ingest tier. Where no relay is present, agents push directly to a regional ingest endpoint.
Authentication: mutual TLS with short-lived certificates (issued via an internal PKI or an SPIFFE/SPIRE-style workload identity, an open standard that automatically issues short-lived cryptographic identities to services and hosts), not long-lived static tokens. Each agent proves its identity on every connection; the collector or relay validates the certificate chain and maps the identity to a tenant/authorization scope before accepting data.
Service discovery: agents resolve a priority list, local relay first (via internal DNS SRV records), falling back to a regional ingest load balancer if no relay is configured. Relays themselves register with a central control plane over their own mTLS connection so the platform knows which relays are healthy.
Traversing NAT and firewalls in practice: rely on standard outbound HTTPS (443) so no firewall exception is needed beyond what's already open for normal internet-bound traffic; use HTTP/2 or a persistent WebSocket-style multiplexed stream so one connection carries many events instead of opening a new TLS handshake per batch, which also reduces the load a busy proxy has to inspect.
flowchart LR
H1[Host behind firewall] -- push, mTLS --> R[Local relay]
H2[Host behind firewall] -- push, mTLS --> R
R -- single outbound, mTLS --> LB[Regional ingest LB]
H3[Host, direct egress allowed] -- push, mTLS --> LB
LB --> Q[(Durable ingest queue)]
Worked example
Sizing one relay for a corporate network segment. Assume N=5,000 agents behind one relay, each pushing a batch every 15 seconds, averaging 50KB/batch:
batchRatePerSec=155,000=333.3 batches/sec throughput=333.3×50KB=16,667 KB/sConverting to megabits/sec (×8 bits, ÷1000):
throughputMbps=1,00016,667×8≈133.3 MbpsAgainst an illustrative 1 Gbps relay uplink:
133.31,000=7.5× headroomOne relay comfortably serves 5,000 agents at this batch rate with 7.5x headroom before the uplink itself becomes the bottleneck, which is the concrete answer to "does one relay per corporate segment suffice or do we need to shard relays" for a segment this size.
Certificate rotation load. With 24-hour certificate lifetimes for those 5,000 agents:
renewalsPerDay=5,000×2424=5,000/day≈0.058/sec averageTightening to 1-hour lifetimes for a higher-security posture:
renewalsPerDay=5,000×124=120,000/day≈1.39/sec averageGoing from 24-hour to 1-hour certificate lifetimes is a 24x increase in renewal traffic against the PKI/issuance service, still under 1.4 renewals/sec average for this fleet size, which is the kind of load a normal internal CA handles without needing dedicated scaling, but it's worth stating the number rather than assuming "short-lived certs are free."
Trade-offs & pitfalls
Common wrong turns: defaulting to pull because it feels simpler to reason about ("the collector controls the schedule"), without checking that it's actually infeasible the moment a single host sits behind NAT with no inbound rule, which is the stated constraint here; putting authentication only at the relay and trusting everything on the internal network behind it, which means a compromised host on that network can inject or read telemetry for every other host sharing the relay; and skipping short-lived certificates in favor of long-lived static tokens to avoid the rotation-load question above, which trades a manageable, quantifiable renewal cost for an unbounded blast radius if one token leaks.
Design access control for an observability platform used by 50 engineering teams: an RBAC model, namespace or tenant isolation, per-team dashboards and saved queries, audit trails, and SSO/SAML integration. What's different about the admin role for platform operators, and how does your answer change between a SaaS deployment and an on-prem one?
Sample Answer
Build access control as three layers that compose: identity and provisioning (who is this, what teams are they in), a policy engine that maps identity plus team to permitted actions on specific resources, and an immutable audit trail of every access decision. The 50-team scale is what makes a policy-engine approach (rather than hand-rolled per-resource checks) worth the setup cost.
Design
flowchart LR
A[SSO / SAML IdP] --> B[SCIM Provisioning]
B --> C[Team / Group Mapping]
C --> D[Policy Engine: OPA]
D --> E[Dashboard and Query ACL]
D --> F[Platform Admin Scope]
E --> G[Audit Log: append-only]
F --> G
- Identity and provisioning: SSO via SAML/OIDC for authentication, SCIM (or LDAP sync) for provisioning, so team membership changes (new hire, team transfer, offboarding) propagate automatically instead of relying on manual grants.
- RBAC model: a small set of coarse roles (Viewer, Editor, TeamLead, PlatformAdmin) combined with resource-scoped permissions (
dashboard:read,dashboard:write,query:run,query:save,data:ingest). Evaluate as(subject, action, resource, context)through a policy engine (e.g., OPA/Rego) rather than scatteringif user.role == 'admin'checks through application code, since 50 teams' worth of dashboard-sharing and cross-team-visibility rules will otherwise become unmaintainable ad hoc logic. - Namespace/tenant isolation: each team's dashboards, saved queries, and (if applicable) raw telemetry are tagged with a team/namespace ID; the policy engine denies cross-team access by default, and sharing is an explicit grant, not an implicit consequence of both teams querying the same underlying data.
- Audit trail: append-only log of every access decision (actor, action, resource, timestamp, allow/deny), shipped to a separate system (SIEM or dedicated log store) so a compromised or misconfigured application instance can't retroactively edit its own audit history.
What's different about the platform-operator admin role
PlatformAdmin needs privileges no team role should have (upgrading the platform, changing global retention policy, accessing any team's data for support purposes) but that scope is exactly what makes it the highest-risk role. Treat it differently from every other role: require a separate approval workflow for admin grants, use time-boxed or break-glass access (temporary elevation with mandatory justification and MFA, auto-expiring) rather than standing admin permissions, and audit admin actions with the same rigor as the audit trail above, ideally with additional real-time alerting since a compromised admin credential is a platform-wide risk, not a single-team risk.
SaaS vs. on-prem
| Aspect | SaaS | On-prem |
|---|---|---|
| Multi-tenancy model | Logical: namespace/tenant ID enforced in the data plane across shared infrastructure | Often single-tenant per deployment already, or offer namespace isolation within one customer's own cluster |
| Identity integration | Centralized IdP the platform operator controls (own SSO tenant, own SCIM sync) | Must integrate with the customer's existing SSO/SAML/LDAP, which varies per customer and requires a flexible integration layer |
| Encryption/key management | Centralized KMS, platform operator manages keys (with option for customer-managed keys for higher-security tiers) | Customer typically owns key management; platform needs to support bring-your-own-KMS |
| Audit log destination | Centralized SIEM the platform operator runs | Must export to the customer's own SIEM/log destination; platform can't assume it owns the audit pipeline |
| Isolation guarantee | Logical isolation is normally acceptable, since the platform operator's own infrastructure enforces it | Some customers deploying on-prem specifically want single-tenant network isolation (K8s namespace + network policy, or fully separate deployment) as a hard requirement, not a preference |
A quick sizing check on audit volume
A frequent unstated assumption is that audit logging is "too expensive to keep forever." For 50 teams with roughly 30 engineers each performing about 200 dashboard/query actions/day at 300 bytes/audit record:
teams, engineers_per_team, actions_per_day, record_bytes = 50, 30, 200, 300
total_engineers = teams * engineers_per_team # 1,500
events_per_day = total_engineers * actions_per_day # 300,000
bytes_per_day = events_per_day * record_bytes # 90 MB/day
total_bytes_1yr = bytes_per_day * 365 # ~32.85 GB/year
At roughly 32.85 GB for a full year of audit history across the whole platform, audit-log retention is cheap relative to telemetry storage; there's rarely a cost reason to truncate it, which supports keeping compliance-grade audit history far longer than raw telemetry.
Trade-offs and pitfalls
- Embedding authorization checks directly in each service (instead of a shared policy engine) is the design that looks fine at 5 teams and becomes unmaintainable at 50, because every new sharing rule requires a code change in every service that touches dashboards or queries.
- Standing admin access (a permanent PlatformAdmin role assignment with no expiry) is the most common real-world gap between a documented RBAC design and actual practice; break-glass, time-boxed elevation closes that gap but requires operational discipline to actually use.
- On-prem deployments that assume the SaaS identity/audit architecture transfers unchanged will hit friction the first time a customer's SSO provider or compliance requirement doesn't match the platform operator's assumptions; the abstraction boundary (pluggable IdP, pluggable audit sink) needs to be designed in from the start, not retrofitted.
- Least-privilege between on-call engineers and management is a common refinement: an on-call engineer needs broad read access across teams during an incident but not write/admin access, while a manager might need visibility without either; collapsing both into a single "Editor" role loses that distinction.
Design multi-region telemetry ingestion that supports low-latency local queries in each region as well as global long-term analytics, while respecting data-residency constraints that keep certain telemetry from leaving its region. How would replication, federation, and query routing work, and how would you avoid excessive data duplication and egress cost?
Sample Answer
Direct answer
Keep raw telemetry in the region where it was generated and serve local dashboards directly from that regional store. For global, long-term analytics, replicate only pre-aggregated rollups (never raw points) to a central aggregate store over a star topology, and give the query router two paths: a fast local read for single-region queries, and a scatter-gather federated read that fans a query out to each owning region and merges results when a global view needs residency-locked data that was never centrally copied. This bounds egress to the size of the summaries instead of the raw stream and keeps residency-restricted series from ever leaving their region.
Structured elaboration
Regional tier (per region):
- Ingest agents write to a regional durable log, then into a regional time-series store that serves local dashboards at full resolution.
- A residency tag is attached to every series at ingest time (
region_locked: true/false). Locked series are excluded from every export path, full stop, not just the raw one. - A rollup exporter runs on a fixed schedule (for example every 5 minutes), producing one summary record per series per window (count, sum, min, max, and a mergeable percentile sketch), and ships only unlocked series.
Global tier:
- A single global aggregate store receives rollups from every region over a star topology (each region ships once, to one destination), not a full mesh where every region replicates to every peer.
- The global store answers cross-region and long-retention analytics directly when the requested series are present there.
Federated query router:
- Classifies each incoming query as local (single region, recent window) or global (spans regions, or requests residency-locked series that never left home).
- Local queries route straight to the regional store; the router adds nothing but auth and residency checks.
- Global queries against unlocked series read the global aggregate store.
- Global queries that touch locked series cannot be answered centrally: the router issues parallel subqueries to each owning region, applies the same aggregation the caller asked for at the region, and merges the partial (already-aggregated) results at the router. Raw data still never crosses the region boundary; only the small aggregated answer does.
flowchart LR
subgraph RegionA[Region A]
A1[Ingest + Raw Store]
end
subgraph RegionB[Region B]
B1[Ingest + Raw Store]
end
subgraph RegionC[Region C]
C1[Ingest + Raw Store]
end
A1 -- unlocked rollups --> G[(Global Aggregate Store)]
B1 -- unlocked rollups --> G
C1 -- unlocked rollups --> G
Q[Federated Query Router] -- local query --> A1
Q -- local query --> B1
Q -- local query --> C1
Q -- global query, unlocked --> G
Q -- global query, locked: scatter-gather --> A1
Q -- global query, locked: scatter-gather --> B1
Avoiding duplication: rollups replace raw fan-out, the star topology replaces mesh replication, and locked series are excluded from export entirely rather than exported-then-filtered (filtering after export would already have paid the egress cost).
Worked example
Assume 4 regions, each with S=50,000 active series scraped every 15s (4 samples/series/minute), so each region ingests r=200,000 raw samples/minute:
rawSamplesPerDay=200,000×1440=288,000,000At braw=24 bytes/sample (delta-encoded, compressed), one region's raw volume is:
Vraw=288,000,000×24 bytes=6.912 GB/dayNaive approach (full raw replication, mesh topology, R=4 regions):
Each region ships its full raw stream to the other R−1=3 peers:
This design (5-minute rollups, star topology to one global store):
Each series produces one rollup record every 5 minutes; broll=200 bytes (count, sum, min, max, t-digest fragment):
Reduction:
EgressrollupEgressnaive=11.5282.944=7.2×This 7.2x comes from two independent effects that both matter and are worth naming separately in the room: dropping the mesh fan-out for a star topology accounts for a factor of (R−1)=3, and collapsing 15s raw resolution into 5-minute summaries accounts for the remaining ≈2.4×. Neither alone gets you there; a star topology with raw replication still ships 3x too much data, and rollups alone don't help if every region still fans out to every peer.
Trade-offs & pitfalls
| Choice | Wins | Costs |
|---|---|---|
| Star topology to one global store | Egress scales with R, not R(R−1) | Single logical destination is a availability/scaling focal point; needs its own HA design |
| Rollup-only export | Cuts payload size, hides raw values from cross-region transit | Loses raw-point debugging for incidents that need cross-region correlation past the rollup window |
| Scatter-gather for locked series | True zero-export compliance | Global queries touching locked series pay per-region round-trip latency and can't be served from a warm cache the way unlocked-series queries can |
Common wrong turns: exporting raw data and filtering residency at the query layer instead of at the export layer (the compliance violation already happened by the time it's filtered); treating "aggregate before export" as a privacy control by itself without also tagging and hard-blocking locked series (an aggregate over a locked series is still a residency violation if it lets someone infer region-specific values); and building the mesh topology by default because it's the naive extension of single-region replication, when a star to one global sink is both cheaper and simpler to reason about for compliance audits.
Compare Elasticsearch, ClickHouse, and an object-storage-plus-index approach as the backend for log storage at scale. Discuss search performance for ad-hoc queries, ingestion throughput, cost at long retention, schema flexibility, and operational overhead for each.
Sample Answer
For long-retention log storage, the honest default is a tiered combination: ClickHouse (or an object-storage-plus-index approach) for the bulk of retained volume, with Elasticsearch reserved for the recent window where full-text ad-hoc search actually matters. Picking Elasticsearch as the sole backend for everything gets expensive fast at long retention; picking object-storage-only sacrifices interactive search on recent data.
What each backend is actually optimized for
flowchart LR
L[Log Producers] --> P[Ingestion Pipeline]
P --> ES[Elasticsearch: inverted index + shards]
P --> CH[ClickHouse: columnar table]
P --> OSS[Object Storage: Parquet blocks]
OSS --> IDX[Secondary Index]
ES --> Q[Ad-hoc Query]
CH --> Q
IDX --> Q
| Dimension | Elasticsearch | ClickHouse | Object storage + index |
|---|---|---|---|
| Ad-hoc search performance | Best: inverted index gives sub-second full-text and fuzzy search, native relevance scoring | Good for structured/columnar filters and aggregations, weaker for free-text (needs token tables or a bolted-on search layer) | Weakest without a fresh index; latency depends on how current the index is and whether it covers the field you need |
| Ingestion throughput | Moderate; bulk API helps but refresh interval, replication, and inverted-index build cost CPU per document | High: columnar writes, batched inserts, background merges are cheap per row | Highest for raw storage (append/upload is nearly unbounded); the indexing pipeline, not the storage, is the bottleneck |
| Cost at long retention | Highest: replication plus inverted-index/doc-values overhead multiplies stored bytes well beyond raw | Low: strong columnar compression | Lowest: cheapest byte-for-byte, and you can defer indexing entirely for cold data |
| Schema flexibility | Schema-on-write with dynamic mapping; flexible for JSON but mapping explosions are a real failure mode | Schema-on-write, columnar; wide/nested tables work but schema evolution needs explicit migration | Most flexible: store raw JSON as objects, apply schema at query time |
| Operational overhead | High: shard sizing, heap/GC tuning, index lifecycle management | Moderate: merge/partition tuning, but fewer distinct failure modes than an ES cluster at the same scale | Low for the storage tier itself; complexity shifts to whatever indexing/query-engine pipeline you build on top |
Worked cost comparison at 30-day retention
Take 50,000 log events/sec, averaging 1 KB raw per event, retained 30 days:
raw bytes=50,000×1,000×86,400×30=1.296×1014 bytes=129.6 TBFor Elasticsearch, assume a replication factor of 2 (one primary plus one replica, the normal HA baseline) and an inverted-index/doc-values overhead multiplier of roughly 1.3x over the raw JSON (a labeled assumption, not a measured constant, since it depends on mapping and field count):
ES stored=129.6×2×1.3=336.96 TBFor ClickHouse, assume an 8x columnar compression ratio versus raw JSON (a labeled assumption; typical for structured log fields with LZ4/ZSTD codecs) and the same replication factor of 2 for HA:
CH stored=8129.6×2=32.4 TBFor object storage plus index, assume 10x compression from columnar Parquet with ZSTD (labeled assumption) and no replication multiplier (the object store's own erasure coding already provides durability), plus a 5% secondary-index overhead:
OSS+IDX stored=10129.6×1.05=13.61 TBSo at this volume and retention, Elasticsearch stores roughly 10.4x more data than ClickHouse and about 24.8x more than object-storage-plus-index for the identical raw input, purely from replication and index overhead, before you've even paid for the compute to keep those indices hot.
Recommendation and trade-offs
- Use Elasticsearch for a short hot window (days, not months) where engineers are actively doing free-text incident search; keep index count and field mapping disciplined to avoid mapping explosions.
- Use ClickHouse (or a managed equivalent) as the default long-retention backend when most queries are structured filters plus aggregation (by service, level, status code) rather than free-text search; this is the common case for "how many 5xxs did service X throw last Tuesday."
- Use object-storage-plus-index only when queries are rare, tolerate higher latency, and you want the lowest possible cost floor for compliance-driven retention (e.g., 1-year audit logs nobody expects to search interactively).
- The trap senior candidates catch and juniors miss: this isn't a single-backend decision. A hot/warm/cold tiering strategy (ES or a fast index for the last few days, ClickHouse for weeks to months, object storage for the long tail) captures the strengths of each without paying any one system's worst-case cost or latency for the whole retention window.
- Schema flexibility cuts both ways: ES's dynamic mapping feels convenient until an unbounded field (a stray high-cardinality tag) blows up your mapping and cluster health; object storage defers that cost to query time, which is safer for ingestion but slower for discovery.
Unlock Full Question Bank
Get access to all 41 Observability and Monitoring Architecture interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.