Covers strategies and tooling for observing network health and performance. Topics include active health checks versus passive telemetry, what to measure at interface and flow level, flow based telemetry such as NetFlow and sFlow and export formats such as Internet Protocol Flow Information Export, Simple Network Management Protocol based metrics, metrics hierarchy and granularity, retention and aggregation considerations, alerting strategy to manage signal to noise and avoid alert fatigue, dashboards and status pages, runbook and incident playbooks, topology and capacity planning, and common observability platforms and integrations such as Prometheus the Elastic stack and Splunk or cloud native alternatives. Interviews evaluate ability to design what to monitor how to collect it and how to turn telemetry into reliable operational signals.
HardSystem Design
61 practiced
You must store and index several petabytes per month of NetFlow/IPFIX records for security and capacity planning. Propose a storage and indexing architecture (e.g., columnar DB like ClickHouse, Parquet on S3, or Elasticsearch), including table/index schema, partitioning strategy, compression, ingestion pipeline, pre-aggregation for common queries, and trade-offs between query latency and storage cost.
Sample Answer
**Clarify requirements**- Ingest several PB/month (assume 3–10 PB), retention 90–365 days; query types: fast lookups for incident triage (last 24h), long-term trending/aggregation for capacity planning.**High-level architecture**- Hot store: ClickHouse cluster (columnar, low-latency analytics) for recent 7–30 days.- Cold store: Parquet files on S3 (partitioned) for long-term, cheaper storage; query via Trino or Athena.- Metadata/catalog via Hive metastore; lifecycle managed by Airflow.**Table / index schema (ClickHouse)**- Table: netflow_raw (MergeTree) - Columns: ts DateTime, src_ip UInt32, dst_ip UInt32, src_port UInt16, dst_port UInt16, proto UInt8, bytes UInt64, packets UInt64, flow_id UInt64, exporter_id UInt32, tags LowCardinality(String) - Primary key: (toStartOfMinute(ts), src_ip, dst_ip, src_port, dst_port, proto) - Use ORDER BY (ts, src_ip, dst_ip)**Partitioning & compression**- Partition by toYYYYMMDD(ts) for ClickHouse; in S3 by date/hour and exporter.- Use ClickHouse codecs: LZ4 for speed on hot, ZSTD level 3–9 for colder parts.- Parquet: Snappy or ZSTD, columnar types (INT for IPs), dictionary encoding for LowCardinality fields.**Ingestion pipeline**- Collectors (fast C/C++ or Go) -> Kafka (topic per region/exporter) -> Stream processors (ksql/Flume/Spark Structured Streaming) for enrichment -> load to ClickHouse using batched Insert with buffer tables and async merges; also write raw Parquet to S3 via connector for cold store.**Pre-aggregation**- Materialized views in ClickHouse: - 1-min and 1-hour rollups by (ts_bucket, src_ip, dst_ip, proto) for common queries (top talkers, top ports).- Store rollups longer in ClickHouse or push hourly aggregates to S3 as Parquet.**Query patterns & latency vs cost**- Hot ClickHouse: sub-second to seconds for triage, higher cost (NVMe, memory).- Cold S3+Trino: minutes for ad-hoc historical queries, very low storage cost.- Trade-offs: increase hot retention reduces query latency for older windows but raises storage/IO cost; coarser rollups save cost but lose granularity for forensic work.**Operational considerations**- Ensure shard by exporter/region to avoid hotspots; provisioning for burst (ddos) with autoscaling Kafka/ingest workers.- Backpressure: drop/aggregate at ingest if overwhelmed (sampled mode).- Security: encrypt at rest (S3 SSE), network ACLs, IAM roles, logging.- Monitoring: ingest lag, partition sizes, slow queries, compaction stats.This design balances sub-second triage needs with multi-PB retention cost control while preserving forensic granularity via tiered storage and rollups.
MediumTechnical
71 practiced
Compare SNMP polling with streaming telemetry approaches (gNMI/gRPC and OpenConfig/YANG). For a migration plan affecting 10,000 devices, enumerate the operational benefits, tooling and collector changes required, compatibility risks with vendors, security implications, and how the telemetry data model would improve observability.
Sample Answer
**Overview / short comparison**SNMP polling is pull-based, low-bandwidth but high-latency for many metrics and relies on MIBs; streaming telemetry (gNMI/gRPC + OpenConfig/YANG) is push-based, efficient, supports structured data and near-real-time insight.**Operational benefits (10,000 devices)**- Lower polling overhead and reduced SNMP poll storm risk.- Near real-time alerts and finer fidelity (event-driven, sampled, or periodic streams).- Standardized, vendor-neutral models (OpenConfig) improve cross-vendor correlation and automation.- Better scalability: collectors ingest streams rather than polling each device on schedule.**Tooling & collector changes**- Replace/augment SNMP NMS with gNMI-capable collectors (e.g., Telegraf, Prometheus remote write adapters, OpenConfig-capable collectors like OpenTelemetry agents, gNMI-gRPC proxies).- Implement message bus (Kafka) for scale, buffering, and replay.- ETL/normalization layer to map vendor YANG to canonical OpenConfig schema.- Migration scripts, config management (Ansible/Crosswork) to enable telemetry on devices.**Vendor compatibility risks**- Older devices may lack gNMI or OpenConfig support; require software upgrades, replacements, or SNMP-to-gNMI gateways.- Partial YANG coverage across vendors necessitates per-vendor mapping and testing.- Feature parity gaps (some telemetry counters/configs only available via vendor-specific models).**Security implications**- gRPC with TLS + mutual authentication recommended (mTLS).- Proper cert lifecycle, PKI, and key management at scale.- Secure collectors (RBAC, hardened endpoints), network-level isolation (VPNs or dedicated management fabric).- Monitor for data exfiltration; apply rate limits and encryption in transit and at rest.**Observability / telemetry model improvements**- Strongly-typed, hierarchical data (YANG/OpenConfig) enables consistent dashboards, deterministic alerts, and automated remediation.- Metadata (timestamps, sequence numbers, sensor paths) improves correlation and root-cause analysis.- Ability to subscribe to state changes reduces blind spots versus periodic SNMP polls.**Migration approach (high level)**- Inventory device capabilities → pilot 100 devices across vendors → deploy collectors and Kafka → run gNMI in parallel with SNMP for 3 months → validate parity, automate onboarding → roll out in waves, retire SNMP where appropriate.
EasyTechnical
78 practiced
Define a metrics hierarchy and granularity plan for network observability. For interface counters, flow aggregates, and routing telemetry, specify what to store at raw (high resolution), hourly, and daily granularity. Discuss label design and cardinality trade-offs and provide guidance on which labels to drop when rolling up to coarser granularity.
Sample Answer
**Definition & goals**A metrics hierarchy defines what to retain at each timescale so you can troubleshoot (high-res), analyze trends (hourly), and report (daily) while controlling storage and cardinality.**Raw (high resolution, e.g., 1s–1m)**- Interface counters: per-device, per-interface, per-VLAN, per-queue, per-direction (in/out), per-protocol (if sampled), timestamped.- Flow aggregates: sampled flows with 5-tuple, bytes/packets, start/end, DSCP, application tag if available.- Routing telemetry: per-peer per-prefix adjacency changes, BGP attributes, route install latency.**Hourly rollup**- Interface: device + interface + direction + aggregated bytes/packets/errors/drop counts (no per-queue or VLAN).- Flows: aggregate by src/dst subnet (/24 or tenant), protocol, total bytes/packets.- Routing: per-peer prefix-counts, prefix flaps aggregated.**Daily rollup**- Interface: device + interface totals, utilization percentiles (p50/p95), error totals.- Flows: top-N src/dst subnets and protocols; otherwise subnet-level aggregates.- Routing: daily prefix churn, long-term stability metrics.**Label design & cardinality trade-offs**- Keep high-cardinality labels only at raw: full IPs, queue IDs, flow IDs, application tags.- Hourly: drop queue IDs, host-level process IDs, full IPs → replace with subnet or tenant.- Daily: drop per-direction or protocol where not needed; retain device, interface, and high-level role.- Guideline: if a label multiplies cardinality by >10x and is rarely used for rollup queries, drop it earlier.**Which labels to drop when rolling up**- Raw→Hourly: full-src-ip, full-dst-ip, port numbers, queue-id, VLAN-id, flow-id.- Hourly→Daily: direction (if symmetric), protocol ports (keep protocol), DSCP, tenant-subtags.- Always keep: device, interface, time-bucket, and business-relevant tags (site, environment).This balance preserves forensic capability at high-res while keeping long-term storage and query performance manageable.
HardTechnical
54 practiced
You need to collect per-interface metrics for 200,000 interfaces and keep Prometheus stable and queryable. Propose concrete architecture options (label design, relabeling to reduce cardinality, push/agent-side aggregation, horizontal sharding, remote-write to long-term stores), estimate expected series counts for alternatives, and propose monitoring/alerting for Prometheus resource saturation.
Sample Answer
**Situation & goal**Collect per-interface metrics for 200,000 interfaces while keeping Prometheus stable and queryable. Focus on concrete architectures, series estimates, cardinality-reduction techniques, and monitoring/alerts.**1) Baseline estimate (naïve)**- Suppose 10 metrics per interface (ifIn, ifOut, errors, speed, status, etc.) => 200,000 * 10 = 2,000,000 active series in the Prometheus head. With typical label cardinality (interface, device, region, ifIndex) you may get high churn and memory blowup.**2) Label design & relabeling (reduce cardinality)**- Keep only necessary labels: device (host), interface (short id), region. Drop descriptive labels like long names, VLAN or MAC in head.- Use relabel_config to map long interface names to stable short IDs, and to drop static labels at scrape-time.- Example: drop per-user or per-flow labels; move low-cardinality labels to metric name when appropriate.**3) Agent-side aggregation / push model**- Run lightweight agents (Telegraf / Vector / custom exporter) on collectors to: - Aggregate counters across polling windows (sum/avg) per device. - Emit per-device rollups (device-level totals) to Prometheus, and per-interface deltas at lower frequency.- Example: emit detailed per-interface every 5–15m, and high-frequency rollups every 15–60s to keep dashboards responsive.**4) Horizontal sharding & long-term store**- Option A — Sharded Prometheus fleet: shard by device ranges (e.g., 20 Prometheus instances each handling 10k interfaces). Each handles ~100k series (if 5 metrics/interface) => ~1M series per shard — tune down to 50k interfaces/shard.- Option B — Use Cortex/Thanos/Mimir with ingesters + querier, remote_write all scrapes. Keep a small Prometheus for scraping and remote_write to long-term store; use query frontend/caching.- Recommended: combine sharding + remote_write so head stays small and queries offload to long-term store.**5) Expected series for alternatives (examples)**- Full detail in head: 2M series (unstable for single Prometheus)- Agg + lower freq (every 5m per-interface): 200k interfaces * 2 metrics * (12 per hour) ~ manageable for long-term store; head only stores rollups: ~200k series- Sharded (20 shards) with per-interface metrics at 15s: ~100k series/shard — workable with high memory.**6) Monitoring & alerting for Prometheus saturation**- Metrics to monitor: - prometheus_tsdb_head_series > threshold (e.g., per-shard 150k) -> Alert - prometheus_tsdb_wal_fsync_duration_seconds > 0.5s -> IO saturation - prometheus_engine_query_duration_seconds (p95/p99) increases -> slow queries - process_resident_memory_bytes > 80% of RAM - TSDB WAL size growth rate high -> high churn - compaction_duration_seconds / compaction_chunk_range -> compaction backlog- Concrete alerts: - Critical: head_series > 120k for 5m or memory > 90% -> scale/shard now - Warning: WAL fsync > 0.5s or compaction backlog > 10m -> investigate disk/IO - Query latency p99 > 2s -> enable query frontend/caching or route to long-term store- Add dashboards: head series over time, churn (new series per minute), WAL size, compaction, scrape durations, and per-scrape error rates.**7) Trade-offs & recommendation**- For 200k interfaces, do not run a single Prometheus. Best approach: scrape via agents that aggregate and relabel, shard Prometheus by device group, remote_write to a scalable store (Cortex/Thanos/Mimir) for queries/history. Keep head series intentionally small (rollups + critical per-interface metrics at lower frequency), use long-term store for high-cardinality per-interface detail and ad-hoc analysis.This combination minimizes head memory/CPU, keeps queries responsive, and provides long-term analytics while controlling label cardinality.
MediumSystem Design
108 practiced
Design a scalable flow-telemetry pipeline that ingests and analyzes 100,000 flow records per second (NetFlow/IPFIX/sFlow). Include collectors, buffering/messaging layer (e.g., Kafka), parsing and enrichment (geoip, ASN), hot-store for fast queries, long-term archive, sampling and aggregation strategies, and a rough approach to sizing key components and estimating costs.
Sample Answer
**Clarify requirements & constraints**- Ingest 100,000 flow records/sec (assume avg record 500 B → ~50 MB/s = ~432 Mb/s raw). - Real-time queries (<2s) on recent data; long-term archive for forensics; enrich with GeoIP/ASN; tolerate UDP loss; cost-conscious.**High-level architecture**- Collectors (UDP) → Load balancer → Kafka (ingest buffer) → Stream processors (parsing + enrichment) → Hot-store (fast queries) + Archive (S3) → Dashboard/alerts**Collectors**- Several stateless collector VMs/containers (e.g., 8-16) running nfdump/pmacct or custom C/C++ UDP listeners with batching and backpressure to local senders. Use source-based hashing to distribute flows evenly.- Use VM NIC tuning: large UDP buffers, SO_REUSEPORT, irq balance.**Buffering / Messaging (Kafka)**- Use Kafka/MSK as durable buffer. Target throughput ~60 MB/s to allow overhead.- Partitions: 12–24 partitions (more partitions for parallelism); replication factor 3.- Retention: 24–48 hours for replay.**Parsing & Enrichment**- Stream processors: Flink or Kafka Streams (autoscaled). Steps: - Parse NetFlow/IPFIX templates, normalize. - Enrich: GeoIP (local MaxMind DB cached), ASN lookup (local RIR DB or Redis cache). - Output both raw enriched flows and aggregated metrics.**Hot-store for fast queries**- Use ClickHouse or Elasticsearch/OpenSearch for recent fast queries/OLAP. ClickHouse preferable for high ingest and aggregations.- Retention on hot store: 7–30 days depending on SLA. Partition by date + src_prefix.**Long-term archive**- Store compressed Parquet/AVRO on S3 with partitioning (date/hour) and lifecycle rules to Glacier after 90 days.**Sampling & Aggregation**- Two-tier strategy: - Full ingest with probabilistic sampling at edge (e.g., adaptively sample 1:10–1:100 during high load). - Continuous aggregations in stream processor: 1s/1m/5m rollups keyed by src/dst/prefix/protocol for dashboards.- Keep unsampled raw for critical spans via retention policy.**Sizing (rough)**- Ingest: 100k/sec * 500 B ≈ 50 MB/s → plan for 100 MB/s headroom.- Kafka: 3 brokers x i3.large or r5d.large (NVMe) or MSK with provisioned throughput; 12–24 partitions.- Stream processors: 4–8 Flink task managers (m5.xlarge) depending on CPU for enrichment.- ClickHouse: 3 nodes (r5.4xlarge) with SSDs for hot days; scale horizontally.- S3 for archive: 50 MB/s sustained → ~4.3 TB/day → ~130 TB/month raw → compressed ~40–80 TB.**Cost estimate (order of magnitude, cloud)**- Kafka/MSK cluster: $1k–3k/mo- Collectors (8 x m5.large): ~$800/mo- Stream processors (8 x m5.xlarge): ~$2k–3k/mo- ClickHouse cluster (3 x r5.4xlarge + storage): ~$4k–6k/mo- S3 storage (80 TB) + Glacier lifecycle + egress: ~$1k–2k/mo- Total: ~10k–15k USD/month (depends on instance types, retention)**Trade-offs & reliability**- Kafka provides replay and smoothing; ClickHouse gives fast aggregations; S3 inexpensive archive. - If cost constrained, increase sampling, reduce hot retention, or use managed analytics (Athena) for queries.**Operational notes**- Monitor lag, packet drops, CPU on collectors. Automate schema/template handling. Regularly update GeoIP/ASN DBs. Test failure scenarios (broker down, collector overload).
Unlock Full Question Bank
Get access to hundreds of Network Monitoring and Observability interview questions and detailed answers.
